Not sure a ban should really be a necessity or would be useful, however people who wear a garment that makes you look radically different from the norm (based on the cultures and/or history of a specific country) shouldn't be too surprised if people find this uncomfortable or unusual.
It's probably more ridiculous to expect the majority of people to accept this as quite normal, as there are many things that make us feel uncomfortable around others. We all have different preferences and tolerances, and cultural comfort together with familiarity are particularly strong ones, as these are formed from such an early age. This is primarily why we make the life choices we do.
We are often more comfortable around people who are similar to us, and who have more in common with us (be that personality, clothes style, sense of humour etc.). That's a proven fact.
Yes, we should aim to be tolerant, open-minded and all-encompassing towards all humans, irrespective of cultural differences, however it would be prudent to remember that there are many who feel they are being taken too far out of their comfort zone in accepting these differences. Dismissively tagging a person's discomfort as something racially motivated, is not only lazy, but also ruins any attempt at serious discussion.