It is, but this misses a lot. At the end of the day, skin colour or race is just one of dozens of human genetic characteristics we are born with and can't change about ourselves, so why is it inherently so much more offensive to insult someone based on their skin colour than any of the others like eye colour or hair colour or height or...anything?
The answer is simply because for 500 years, one race was treated as if they were subhuman and were systematically exploited for the others' gain, leading to a very painful, bloody, centuries long struggle that's still not complete for people of all colour to be equals. The entire concept of dividing humans into races came about in the 17th century to justify transatlantic slavery.
So when you remove all that history behind insulting someone over their race, it's still racist, but it lacks the same weight of insult, it's not transporting the person you say it to back to a time when they were literally regarded as not human, because removing black people's humanity was the only way for the Church to square away slavery and christianity in the 15th-19th centuries.