I think you're making the mistake of thinking that professional referees, who have got where they are by showing a career-long ability to make impartial decisions and remain neutral even under severe provocation, are somehow going around 'hating' particular teams on the basis of geography.
I can assure you, as a former referee even though at a fairly low level myself, that is just not the case.
In any event, in addition to declaring any team affiliation, referees also have to declare any other conflicts of interest they might have. In the unlikely event that a referee has developed some kind of irrational loathing for a team that they feel would affect their performance, they would be duty-bound to declare that also.
If they didn't do so and it came to light later (as the saying goes, "Nothing remains hidden for ever") then their career, at least as a top-level referee, would be over
Although as I say, in my experience that just isn't very likely to be the case.