It is possible your servers share, perhaps, a common resource that is timing out at certain times, and that your health check requests are being made at the same time (and thus pulling the backend servers out at the same time).

You can try using the HAProxy option spread-checks to randomize health checks.