Confusing way of doing it. Your running 860 gear trying to balance with 500 mhz carrier? How are you adjusting to the correct pad level or eq? If your using 860 eqs are you calculating that in to the pad selection?(An 860 mhz eq will attenuate 500 mhz quite a bit more than 853.25) That is just a problem waiting to happen. I doubt that is what is causing the roll off, but depending how screwed up the amps feeding you are misbalanced could be a issue. You always want a carrier at the top of your bandwidth to balance the system with.
This is CABL.com posting #303387. Tiny Link: cabl.co/mbq5v