i have a pubsub application (mostly chat but some other goodies being pub-ed and sub-ed too) running on node & socket.io.

i'm load testing this app by spinning up some other, real large, boxes and running a node app i wrote for this purpose that spawns a ton of processes that connect using the socket.io-client package.

i found that i can get about 1k concurrent connections to a single 1gig rackspace cloud box. we need to support between 10k and 100k concurrent connections (for specific events, not all the time) though so i put a load balancer in front and figured before a big event i'd spin up more machines. but i've put a haproxy box in front and found that with 2 servers and 2k users i'm golden but with 4 servers even 3k users is a struggle!

i noticed that when my load tests start causing lots of disconnects the node servers are experiencing very high cpu usage (in the 90%) which i find odd because when 2 servers and 2k users i end up with a max of 70% which quickly diminishes.

1 Answer
1

Turn on HAProxy HTTP logging if you haven't already and see if you can find the cause of the disconnects in either that or syslog of the server. The HAProxy HTTP log format includes the termination_state for requests which should help point you in the right direction.