This would just exacerbate the existing problem of wage stagnation which is the primary good reason for not having freedom of movement. FoM adversely affects low/unskilled by increasing labour supply. Far more people want to come to the UK and work than want to go the other way and so the people that are negatively impacted are those that have to compete for jobs with people willing to work for less than minimum wage.
There are some specific things that are problematic as pointed out in that article which can be solved with a bit of co-operation. We don't need to rejoin the EU to have specific visas for touring musicians. We don't have to rejoin the EU to go back to having ID lists instead of passports for visiting school groups etc. A bit of co-operation can go a long way to alleviate a lot of the issues. We want to avoid a situation where we allow companies just to import foreign workers because it is cheaper but there is no reason we can't be flexible on specific areas.