First things first, black people aren't disabled nor do they need treating like they are so it's a false equivalence. Therefore, the need for a 'safe space' doesn't exist in the way that it would for somebody with a disability who may be less able to defend themselves from physical danger or is emotionally predisposed to perceiving normal behaviour as threatening.
That said, even in the context of a disabled persons' safe space, it would be excessive and divisive to demand that no able-bodied person was allowed in there if, for example, an able-bodied person had spent the past 20 years caring for a disabled person and could contribute positively to the space.
My nephew has cerebral palsy and at no point has he ever requested a space comprised solely of other disabled people in order to feel safe. I can tell you right now with 100% certainty he would find the idea absolutely laughable.
Does that mean there shouldn't be meetings or days focused on disabilities or mechanisms for teaching and reporting racial discrimination? No, it doesn't. But there's a big difference between that and the toxic division of 'safe spaces' which preys on the age-old trope that 'the other' is a threat.