Part of me thinks if he proceeded to state and discuss a holocaust joke with the jewish community as the subject, it would have been less controversial.
Because it would have been far more obvious that there was no intent behind it, if there was any laughter it would have been at the shock of it, and it would have been clearer as a piece of satire.
Would have maybe been at risk of wider criticism, but also much easier to distance himself from.
Note his routine isnt the joke, in isolation, it is a disussion of A joke. But using the gypsy community leaves both the audience and him at question of is there prejudice in it, and are they focusing it on an easier target.