A lot of this comes from what has happened historically over the last 40 years. We haven't been at war in Chinese or Indian countries and we haven't been bombed in our cities by an extreme minority of them either.
Muslims are also a growing demographic, demographic estimates currently estimate that potentially 15% of the UK population will be Muslim by 2050, it's currently 6.5%. No other demographic is growing this fast. So some people will regard this as a slow invasion and Islam itself is very different to other cultures.
This isn't to say that certain people aren't racist towards other minorities because of course they are, they just view Muslims as a greater threat.
For myself I have zero problem with Muslims at all but I do have big questions about Islam and its long term compatibility with UK culture and law. How Islam treats women for example cannot be accepted ever in the UK. Obviously it will never be accepted but as the population grows it's understandable to have some concern.