No it wouldn't be fine and it isn't going to be fine, there's an echo chamber sounding the same bell over and over again and it wont make it so. Control of our borders and the protection of British citizens (That includes jobs) should be paramount in the minds of our government.
ok but thats not what we the brexit vote was about.
It was about leaving the EU.
Some campaigned for a brexit with strict border controls, some campaigned that we could leave but stay in the single market(which requires free movement of people)
People voted on both those issues (in what numbers, who knows, the question was not asked). Therefore unless we have a second vote on the type of brexit we want both are equally valid and legal ways to leave the EU.