IMO, because it's not like for like. A lot more is codified rather than opinion of a specific incident.
Rugby is much slower, and rarely moves a long way in a short time - players are always behind the ball so the 50 yard pass doesn't happen, and the officials are usually closer. An obstructing player is really obvious.
Offsides are in line with the ball, not defenders running in the opposite direction. If the ball goes forward, that's enough.
There are less options for a foul challenge that isn't obvious immediately. Most things that would get play stopped are pretty obvious - there isn't the curious concept of the current handball law and whether it was shoulder or arm.
Also, sanctions are more standardised - start at yellow/red and look for why it should be varied. Also, the effect of a yellow card is usually very significant in rugby, and players will avoid them if they can - easier for the ref, and not a lot of playacting to work out.