What generally happens is fairly well educated, hard working , productive people with the character and get up and go move countries in an attempt to better themselves through hard work and effort. They move countries and improve productivity in that country and often get jobs over locals as they are prepared to work harder for less. See the intra EU immigration or a huge chunk of Mexican migration into the US though that is obviously often illegal and thus harder to collect and quantify the benefit.
The poorest, the least educated, the lazy and those who contribute least to society rarely move, rarely chase bettering themselves in this way be that due to lack of opportunity, aspiration, expectation - each will have different reasons. So it has never been the "dregs of society" ( or whatever the latest nom du jour is from the right) - they stay where they are.