Your last line is precisely what you’re advocating for with this idea, when a team is out of challenges and the referee makes a big mistake.
Yeah, it may happen, but teams shouldn't challenge unless they're pretty sure there's a mistake. This would mean teams would have to go back to generally accepting the refs decisions, but would also allow errors to be fixed (provided teams don't fuck themselves up by wasting the challenges).
Not saying it would be perfect, but we've seen tons of goals disallowed for offsides and handballs where nobody has even appealed. That to me is wrong and messing up the game. It's got to be better than that.