I used to enjoy going to Anfield as it was a great crack and I never had any bother there.
I went to Roker Park when we got dumped out of the Cup in 73, and that was a good night. Again, everyone was fine with us.
Highbury was also a pleasant place to visit years ago.
Unfortunately, most of the other grounds in those days we visited were a bit menacing, but that was the way it was, not that it deterred us.
You could get involved or you could ignore it.