I personally couldn't care less if I was the only person in the ground watching.
Empty seats are not the problem, but when people can't get in to a sold out game despite there being thousands of empty seats I think its a bit of a problem. If the club were able to sell those seats and get the associated match day spend in club shop, beer, food etc, then the price of my season ticket wouldn't need to go up.
I'll repeat for the 721st time, no one is saying you personally have to attend every game - just give your ticket to a mate or put it on the exchange. Its a lose-lose situation otherwise.