The free movement of workers was a central plank in the original EU idea. It is now at the base of much of the discontent and antagonism towards the EU manifest in big minority votes for anti EU parties in many parts of the region.
There are three different elements to free movement. The first is the deep unpopularity of free movement for the small minority of crooks, terrorists and criminals that attract considerable attention. It is bad enough for a country to have its own home grown crooks and criminals, but even worse if you have to allow them in from other places without sensible border controls to tackle the issue. Many people in the UK want the UK government to have more control over our borders, and to have our own criminal justice powers to deal with cross border crime issues as well as with home grown.
The second is the widespread opposition to benefit tourists. The original idea was the free movement of workers. Few think someone should be able to travel around the EU to find the parts of the region that offer the highest benefits for being out of work and then settle down to a life of living off the state. Surely the EU countries could come to an agreement that benefit tourism is banned?
There remains the related issue of workers moving to countries with the most generous benefit top ups in employment. That too is against the spirit of the free movement of workers. A member state which has to pay the bills for subsidised housing or top up benefits should be free to make its own rules of whether it wishes migrant workers to qualify or not. If it has a skills and labour shortage it may wish to be generous. If it has a lot of home unemployed it may not wish to offer migrants such generous terms.
The third is the issue of migrant labour under cutting home employees. This has become an issue since a number of countries with much lower wages joined the EU. Free movement of workers is easier to manage and produces less mobility where a group of relatively rich countries with high employment levels form a grouping. As soon as you allow low wage countries in, or countries in the area develop very high unemployment levels, then rates of pay for migrants becomes an issue with established workforces. The EU partially recognised this by allowing countries to delay opening their labour markets to new entrant countries – a freedom the Labour government did not use in the UK. The next development could be to have restricted opening of the wider employment markets, geared to levels of average earnings in their own countries.