You are correct, the leave vote is a hugely broad spectrum and this is exactly what makes it appealing to so many. Despite this unholy coalition of Rees-Mogg and his free market zealots, Labour Party Bennites, innocents who believe what they read on busses, daily mail reading Doris from the shires, rabid EDL racists and UKIPs swivel eyed loons, leave did not "smash it" in the referendum, it won by a fag paper.
It is this same unlikely union of leavers that makes Brexit entirely un-deliverable - "leaving" means different things to all the various different leave groups. So much so that they cannot create a deal that they all agree on. As a result we are going to end up with "no deal" which has zero mandate as it was not what leave campaigned for - in fact we were explicitly told on numerous occasions in the referendum campaign that we would get a "great deal", "easiest deal in history", "we hold all the cards", "German car manufacturers need us" etc etc. That, old chum, is not democracy, or at least it is not democracy functioning as it should.
However i digress, as correct as you are to highlight that this is not a left / right issue, you are just as incorrect with your desire to turn this into an entirely class led issue. The "exclusively remain middle classes" is an utter fallacy - 43% of the middle and upper middle classes voted to leave. More than 4 in 10 of voters in those leafy suburbs you describe voted to leave. It is no less diverse than the working class communities where 36% voted to remain. These are the facts of the situation, it is no more a class issue than it is a party issue.