The modem varies it's transmit levels to compensate for splitters, insertion loss at the tap, dc's, etc. and for cable loss, not to get above the noise floor. If a return amp is not balanced correctly, especially if it's overdriving, it will raise the noise floor with the modem signal.
This is CABL.com posting #188226. Tiny Link: cabl.co/mW74