This makes sense to me.
I think the reason that the country voted to leave wasn't due to the clearly bigoted "Little Englanders" it was the more numerous "moderate" leavers for whom the main concern was not specifically immigration but rather the idea of sovereignty. At the time of the vote EU law held supremacy over national laws of all member states. This meant that in cases of conflict between EU law and the law of the UK, EU law would prevail. All issues regarding to EU law would be referred through the European Court of Justice which ensured all states applied EU law the same way so any judgements made were fully binding on UK courts leading to a sentiment that the UK lacked any real legal sovereignty. This in turn led many to feel that being part of the EU limited the country's ability to set its own laws. More importantly perhaps was that EU directives automatically became law in the UK without the need for a vote for approval in Parliament.
For a lot of people this was a concern and the leave campaign strongly argued that exiting the EU would return full legislative and judicial control to Britain, in turn meaning British courts and Parliament would have the final say in legal and legislative matters. This is what I personally feel eventually swayed the vote.
I voted remain (as sovereignty didn't bother me) however I do understand the concerns of friends and colleagues who felt that potentially allowing the EU to bypass parliament to enact laws that UK citizens would be bound by was deeply worrying. These people are categorically not racists or anti-European or any of the other insults commonly thrown at leave voters.
Areas such as freedom of movement, VISAs, import/export agreements and so on wouldn't be a concern for those people (they would welcome them) as long as the UK itself had final say on what was, or wasn't legal in the UK (having being ratified by Parliament).
