Well, finally found a work around for the 1.4GB limit imposed on the V8 heap. This limitation was at first a “hard” limit, and after some tweaks it became a “soft” limit. Now, the combined tweaks listed below have removed the limit entirely! YESSSSSS!!

The first two were already mentioned in my previous few blog articles, and the new ones (#3 and #4) finally open up the possibility of utilizing the real capacity of your server hardware.

1) ulimit -n 999999

This effectively increases the number of sockets Node.js can have open. You won’t get far without it, as the default tends to be around 1024.

2) –nouse-idle-notification

This is a command line parameter you can pass to node. It gets passed along to the V8 engine, and will prevent it from constantly running that darn garbage collector. IMO, you can’t do a real-time server with constant ~4 second latency hiccups every 30 seconds or so.

You might want to also use the flag “–expose-gc”, which will enable the “gc();” function in your server JavaScript code. You can then tie this to an admin mechanism, so you will retain the power to trigger garbage collection at any time you want, without having to restart the server. For the most part, if you don’t leak Objects all over the place, you won’t really need to do this often, or at all. Still, it’s useful to have the capability.

3) –max-old-space-size=8192

This you can tweak to fit your particular server, but the value is in MB. I chose 8GB because my expectation is that 4GB is going to be plenty, and 8GB was just for good measure. You may also consider using “–max-new-space-size=2048” (measured in KB, as opposed to the other). I don’t believe that one is nearly as critical, though.

4)Compile the latest V8 source code, with two modifications

The current node distribution isn’t using the new V8 engine which has significant improvements to garbage collection and performance in general. It’s worth upgrading, if you need more memory and better performance. Luckily, upgrading is really easy.

The second line above is new. Basically, you just need to set that V8_MAX_SEMISPACE definition. It will otherwise default to a much lower value, causing frequent garbage collection to trigger, depending on the memory characteristics of your server JS.

This was done “just in case” because it costs me money each time I run a scaling test. I wanted to be damn sure it was not going to trigger the external allocation limit GC!

That’s it! You will now want to rebuild both V8 and node, I would recommend doing a clean build for both, since replacing the entire V8 source may otherwise fail to take effect.

While writing this, I’ve got a server log open with 2281 MB of V8 heap usage. That’s far beyond the normal 1.4 GB limitation. The garbage collection is behaving, the server remains VERY responsive and low latency. In addition to the 250k concurrent and active connections, each of those nodes is also sending a sprite every 5 seconds.

There is still CPU to spare, the only limitation keeping me from trying 500k concurrent is my Amazon EC2 quota, which I already had to request an upgrade to do 250k.

That change with V8_MAX_SEMISPACE_SIZE=536870912 is very strange. Yes, it will reduce the number of GCs, but the pauses will be much longer when they happen. The semispace GCs are supposed to be frequent, but very fast: Normally around 0-2ms, rarely more than 5ms.

Erik, but the function it’s calling is named “CollectAllGarbage” – it takes as much as 10 seconds!

I started a thread on the V8 dev groups which talks about why it seems my densely connected graph structure is so slow to garbage collect for. My preference has been to keep GC enabled, and somehow improve how it performs with that sort of data structure involved in your program, but the thread dead ended and I found no easy way to hack it up.

I think the ideal solution would be to provide a mechanism to exclude “static” Objects, of the JS devs choosing, from garbage collection. Consider how the “gc();” function exists now. It’s an optional switch that gives you a little more power of garbage collection. The only addition I want is a new function (lets call it “gcExclude”) that can be enabled, and lets you hand it Object handles which will then be moved out of the garbage collector’s path of traversal. You can then later add them back, or they automatically add back in emergency low memory situations or program exit.

Don’t you figure, with node.js’s popularity, there are quite a few devs who are desperate for a way to mitigate the quality of service problems associated with stop-the-world garbage collection? This would seem a sensible solution which wouldn’t compromise V8’s design, would only impact devs who need it, and solves the problem with a simple function call .

CollectAllGarbage called from AdjustAmountOfExternalAllocatedMemory has nothing to do with semispace size. Semispace size controls size of young generation. Young generation is managed by a copying collector — so if you make it too large that would cause very-very long pauses if survival rate is high (imagine copying 100 megabytes back and worth every now and then).

BTW I think we figured out why you were seeing very frequent GCs when approaching 1.4gb limit: apparently you were overflowing int used as counter external allocated memory. Erik committed a change to fix that: r11265.

Theoretically AdjustAmountOfExternalAllocatedMemory should start an incremental GC cycle instead of causing full collection but we are still discussing how to ensure progress of incremental marking if most allocations are happening externally.

Caustic your answer seems to relate to the external memory related GC triggering code, but my comment was related to the semispace change. The semispace GCs are always stop-the-world so if you want to avoid pauses you should actually be making the semispaces smaller. A 1Mbyte limit helps keep semispace-related pauses in the 1ms range.

Erik – the reason for the change I made with V8_MAX_SEMISPACE_SIZE, was to indirectly affect the external allocated memory trigger. There is a bit of code which bases that threshold off the semispace size. I believe they’re meant to be in proportion to each other or something?

Sounds like that brings with it some collateral damage to performance, though it must not have been significant enough of a hit to prevent 250k concurrent. Maybe if I use a cleaner technique to prevent that external alloc triggered GC, performance would be good enough to handle 500k :]

Hello, thanks for you share.
I insepect the v8 source code, find that (Heap.cc)Heap class,
intptr_t max_old_generation_size_; –this is the v8 max_old_space_size
so,the max size is 2GB because the type is int, and testify by like this ” node –max_old_space_size=1919 **.js”
the max size is 1919 and larger then 1919 is failed.
so you set –max-old-space-size=8192 is it success ？？？

The code you mention isn’t there in the version I used for this article (3.10.0).

Also, the command line parameter for old space is measured in MB, so 8GB actually requires only 16 bits (0x2000, 8192). I haven’t checked the latest code, so this could be wrong if it is per-multiplied to bytes before being placed in that variable. Sorry that I don’t have the time to keep this blog up to date with the latest versions of V8.

On a micro instance of EC2 I never get past say 1500 concurrent connections of node.js. I tried spawning 2 instances and having HAProxy in front of them but still it doesnt work beyond 1500. Till 1000 it looks almost fine but socket time out errors start to come on client side after that.
Please suggest what can be the issue

[…] These 3 articles do a great job explaining how (and why) to use command line arguments like –nouse-idle-notifications. Playing with some of these tweaks seemed to help our performance. Scaling Node to 100k connections Scaling Node to 250k connections Escaping the 1.4Gb Heap Limit […]

For anyone interested, one of the best ways to avoid the Heap, is to use Buffers. Buffers exist outside of the v8 heap, and are faster to access. Please watch whenever you convert to and from a buffer, as you are in affect creating an alloc on the heap. For high throughput applications always use streaming conventions, and learn how to work with buffers.