I like Yes fine and, worse, I like Rush, but you're right. As long as someone who likes these bands is keenly aware of the self-indulgence embedded within and doesn't ascribe too much worldly meaning to their noodlings, I say listen to them and enjoy some incredibly talented chaps going nuts for eight-plus minutes a track (especially Chris Squire).
I start to get more annoyed when I'm informed that Yes are somehow a "best" rock and roll band when their music had literally zero social relevance and precious little forward influence on how rock music evolved (other than helping spur the formation of a rash of bands who wanted to do the opposite of what Yes was doing).