When you go to an away game, you have two choices. Wear your colours with pride, have a few beers, maybe even have a few with some home fans, talk about the game and so on.
Or, you can act like you own the place, give the locals some shit, and generally ask for trouble.
It's not hard is it? Route one gets you a good day out, slightly pissed, and home safely. The other route may well end with you getting a kicking.
No point arguing about the rights and wrongs of fighting. You act like a dick, there's always another dick just waiting to chin you. That's life, don't act like a dick.