It's a balance isn't it?: getting more decisions right vs disruption to the game. The more I see of this nonsense, the more I come down on the side of scrap it, let the referees referee, let the game flow and let the fans vent at the referees again every game like we used to.
PS: Re your point about no-contact penalties, there are other ways to punish cheating. Off-line review of potential incidents and increasing retrospective punishment each offence, for example. Now that is a use of VAR I can get behind :)