He's been called a racist because he' parrots all the talking points of racists, just like all the others who have played this routine before him and all those who will do it after him. Dancing around saying or doing something explicitly racist doesn't change that, just like how it doesn't when people try and claim Tommy Robinson isn't a racist, or various others who do this dog whistle routine where they play to racists, nudge and wink at them while trying to stay juuuuust on the right side of the line so they can act offended when people call them racist. They play this game as long as they can, to grift and make money off the other racists who are delighted a new champion has emerged to "say what we're really thinking", until they end up having to push further and further over the line to stay relevant, burning too many bridges so even the shittier TV shows and radio shows won't have them on and end up like Katie Hopkins or that Milo guy, touring eastern europe giving paid talks to the far right dickheads who will still have them.
It's a tired and boring act that I have no idea why people bother tying to defend.