Not sure why you let it bother you, we've been getting it for years and I find amusing in a pathetic way.
Portsmouth sang it to us at fratton park a few years back, I just burst out laughing at them. I'd been there plenty of times when we were shit and we used to take the place over....it's all the shite teams have to sling at you along with "we support our local team" if it makes them happy good luck to them.
A few lads did make a big flag which does the rounds at away games, with "where were you when you were shit ?" a picture of a group of 1970's Chelsea skinheads and underneath it says, "In your town, in your ground, in your end"
As I said though it's a pretty desperate chant and most would love to be on the receiving end as it would mean they were a top side.