Defer to the one who wasn't unsighted. Or have a linesman for the 18 yard box and one for the rest, but on the same side - so only one is active at any given point.
Instead of extra linesmen, you could maybe go for a ref in each half like they have in hockey - less pressure on a single ref and...