God i'm glad someone has pointed this out. It wasn't even a joke, it's just shit and, in my view, racist.
Racism comes in many shapes and forms and whilst I agree that Alan Sugar probably doesn't actively hate black people, his joke does betray the fact he probably holds some racist views and assumptions about black people. There's certainly some kind of "othering" going on in his head if that's the kind of thing that springs to mind when he sees some black people and he expects other people to be on his wavelength with that too. He should have just apologised straight away.
For somebody to see a group of black athletes, competing at the absolute pinnacle of their profession and their brain somehow associate that with people selling counterfeit goods on a beach thousands of miles away, IS disrespectful and offensive. I don't even know how he made that reach, there's no other elements at play that lead you to make that association. They're not on a beach, there's no items floating around that people would usually sell on a beach and they're not dressed like beach vendors. It's literally only the colour of their skin he's going off there.
If you went to see a doctor and he was black, would you make the joke "oh it's funny seeing you here, think i saw you selling knock off sunglasses on the beach in Marbella last week." Of course you wouldn't.
He fucked up and he should have held his hand up. Those who are trying to suggest the people pulling him up on it are somehow in the wrong need to go for a lie down.