To benefit from freedom of movement within the European Union you need to be a member of that Union which requires a lengthy accession process. Alternatively, a country can subscribe to the Single Market and become a rule taker outside the political decision making process.
Either way means, to some extent, converging your economy with the EU and paying into the EU budget etc.
If a non European country, say Algeria, opted to join and all other member states agreed then they would be put on the path to membership and the benefits including the Single Market and freedom of movement. I mention Algeria as the EU is it’s major trading partner and it would not be a huge step to see nations like this become members.
The EU is a legal construct with rules and regulations that you join or associate with to enjoy the benefits, namely the Single Market. The Single Market is the driver of freedom of movement, one of its four pillars. It‘s a package deal, not an à la carte menu. Something we discovered in the Brexit negotiations, although, seemingly, still little understood after four years of talking about the bleeding thing.
The truly dumb thing is you know this because you stated ‘throwing open all borders overnight would be chaos’ and yes, it probably would be, which is why it starts with neighbouring countries with a high degree of economic and cultural convergence and expands slowly outwards.
Unfortunately this natural evolution then leads to emotional nonsense about FoM being ‘racist‘, at which point I lose the will to live over the sheer fucking stupidity of people.