You should have less that 1 db per 100 feet of attenuation on the RG11 for the return. Is the tap you're coming off of a terminating tap, and what is the value? Is there a line EQ? What are your transmit levels on the modem?
This is CABL.com posting #316038. Tiny Link: cabl.co/mbuny