I did a simular set up about 12yrs ago with RG11 and it worked. I think I was just under 1Volt P-P.. I can't remember exactly what I had, but I remember pictures were fine.
Just to double check, hook up a color bar generator to a 250' piece of RG11 and see what you have at the end of it with a video volt meter.
Scott