I encounter the same issues. I'm too fast for the slow lane and too slow for the fast lane so always head for "medium".
I'm always faced with people with no spatial awareness, in that they are slow, but don't give way at the end of lengths for faster swimmers, or too fast for the lane and they go all "passive aggressive".
And I've had some interesting scenarios at the Aquatics Centre in Manchester where some of the Chinese students don't seem to understand the concept of clockwise and anti-clockwise.
My pet hate though is people using the pool to chat, whether it be those hogging the end of the lane to have inane conversations, or middle aged women swimming in parallel really slowly gibbering away and hogging the whole lane. Just go to a café or a pub if you want to chat about the price of fish!