Virgin inherited a lot of old networks over the years and some of it is not capable of supporting modern demands. My house is 30 years old and the cable was installed by Nynex but it became obvious the cabling was degrading underground somewhere as we lost our phone service for days after heavy rain.
Another trick they used to do is over subscribe areas, so speed went down to dial up levels at peak times. This is why they limit upload speeds on the lower tariffs, or at least used to.
When I check now, my address doesn't now have Virgin services so either they've chosen not to upgrade the cabling, or accept there is no more capacity. Not that I'd go back to them.
I'd check the VM forums or digital spy to see if there's any known issues in your area. But like all these companies, they'll bullshit you as long as they can. They even blamed BT for their own phone service when investigating my issues.