I don't know what Labour can do, the majority of the time their political stance is far more in line with protecting your average person/everybody but they have to take so many stances in a fast changing world that eventually they'll be behind a stance that somebody doesn't agree with.
People are genuinely having a go at them for taking a knee against racism but will happily go and support their football teams with players making the same morale stance.
The Tory party on the other hand are anti-change (although they claim they're for change) unless it benefits them, they would like all the issues we have to stay the same because it benefits them. They get far more leeway in the media for it too, which masks their errors far more easily. 'Oh you've been robbing us for hundreds of millions, doesn't matter because you didn't take the knee'.
Baffles me that anyone could put an X in the Tory box after what they've done this year.
The problem Labour and Labour supporters have is they are living only in the present and are missing what has happened over the last 10 years on some huge questions concerning the EU and immigration. The usual mantra of shooting people down for having these concerns as racists or xenophobes is not working.
As you've said, saying it baffles you about someone putting an X in the Tory box misses the point that you likely didn't support Brexit and so don't care about it, but a massive number do and Labour did not and still do not have their backs. Why would the people of Hartlepool vote for the Tories considering they never have before? They haven't become stupid overnight, what they do have is a frustration with Labour because it doesn't speak for them.
The Tories are not a far-right party or even a centre-right party, they are a populist party. Brexit has been the biggest populist question for a century and Labour have been on the wrong side of the electorate on it for over a decade. It says it all that in 2019 the Tories didn't even really need a manifesto to win.
2015 - Miliband opposed a referendum, Cameron gave people one - Tory majority
2017 - May said deal, Corbyn said deal - hung parliament
2019 - Boris said no deal, Corbyn said deal closer to the EU or another referendum - Tory majority
I don't think that any of this is a rejection of Corbyn or the Corbyn hangover with Starmer as so many claim. It is however a rejection of the current radical 'woke', pro-immigration and pro-Europe elements within Labour that are pulling the strings on policy and in campaigning. They shouldn't be looking at why people rejected stalwarts like John Mcdonnell (who accepted the referendum) but rather why they rejected someone like Emily Thornberry (who didn't).