I think he just upset a lot of people who've seen and endured racism in this country when he told a black woman on public TV that bringing up racism was "boring" and then, when the woman said he might not be best placed to recognise racism, given his background, told her that she was the racist.
Or when he got all red in the face and screamed about how outrageous and disgustingly politically correct it was to have a Sikh actor in 1917, because the version of history he got taught didn't allow for non white people in the British army during WW1 (despite the fact Sikh soldiers did serve shoulder to shoulder with white soldiers). At that point some people thought, "that's odd, that bloke who refused to consider racism might exist is bringing up race again".
Or when he demanded everyone boycott Sainsbury's for celebrating black history month.
It's also amazing how despite the fact Fox actually apologised and explained he's too uneducated, stupid and unable to express what he means, he's now decided that it should be his full time job. I'm sure that has nothing to do with the £50m funding someone shoved at him to start a political party.