No it hasn't.
Your experience may be this but its total bollocks.
Certain brands, clever marketing, hollywoodesque movies related to hooliganism and so on have significantly influenced people to think this when it couldn't be further from the truth.
Looking at pictures even just 30yrs ago most people in the stadium would just be wearing regular clothing.
If you would've said horse racing and fashion go hand in hand I would've agreed but football? No way.
Companies who make clothing saw a market opportunity years ago and decided to latch onto the sport popularity and throw their designs at the male football fan demographic... This doesn't mean fashion has always been engrossed with football