I've grown up with fox hunting, certainly in my younger years and now I would agree with you, that there's isn't a strong argument to support it. However, I've no time for the vermin and wouldn't hesitate to shoot one on sight.
That said, I feel there are a lot commenting, not so much against the hunting, but more to do with what they perceive as a 'class' resentment. That these people are able to do what I can't do, they've perhaps had a better education than me, or live in a bigger house, drive a better car, so I have a hate for them.
For many I don't think the fox hunting comes into it, they'd have the same view if it was drag hunting.