I think someone hitting someone who is weaker and unable to defend themselves is more wrong than someone hitting someone who physically is capable of stopping the abuse if they get to a point where they genuinely feel they have to. So a man hitting a woman, in most circumstances, is worse, same for hitting a child, or a disabled person.
As you say, obviously both are very wrong, but one is, in my opinion, significantly worse, as the fear that comes with that helplessness of being completely at the mercy of the person attacking you is a pretty significant factor.