Exchange 2010 Cas Array load Balancing not balanced

CA-1 and CA-3 are set up in one NLB cluster and Ca-2 and CA-4 are in another.

CA-2 and CA-4 load balancing works fine.

CA-1 and CA-3 load balancing has a problem. When looking at perfmon, CA-4 is handling over 500 connections and Ca-1 is handling 11.

Both servers have been restarted , all services are running. I have no idea where else to look for the cause of this problem.

When databases are failed over to the site where the problem NLB is located, users experience significant mail delay. When databases are failed over to the site where the working NLB is located, users see no delays.

Where do I start? What do I look at to find the cause of this unbalanced load balancing?

I would undo the Windows NLB and recreate it to begin with.
Then start planning to get a proper load balancer. WNLB sucks and isn't recommended by the Exchange product team, and it is just these kinds of stupid things which is why that is so.