In a word, yes. I'm going to use yesterdays game as an example for 2 hypothetical situations.
Defenders;
In the second half a Crystal Palace defender blasted the ball off the body of his team mate from about 3 yards away. If that had hit his hand it correctly wouldn't have been a penalty.
Attackers;
Take the same situation but switch it to the other end of the pitch. Cahill lashes the ball in the same manner, it is going wide but hits Zaha on the arm and falls straight to McCarthy who taps it in.
I think that is fair, if you want every handball to given as free regardless then that's different. Just he prepared for 5 or 6 penalties every week.