This comment has been minimized.

The benchmark is invalid since code has been added to cache the file explicitly, this breaks the benchmark - in a real web server you would never do this. If files get updated in the file system while the server is running then the updated file should be returned. Remove the explicit caching code and Vert.x greatly outperforms Node.js. This tells you that Node does not take account of OS level caching.

Running a benchmark on a single core cloud VM is plain stupid. No-one who cares about performance will ever do this, they will run with multiple cores. If you do this, then Vert.x outperforms Node.js even further.

This is running Vert.x JS server. If you run the Vert.x Java server, performance will be even higher still.

The benchmark is invalid since code has been added to cache the file explicitly, this breaks the benchmark - in a real web server you would never do this. If files get updated in the file system while the server is running then the updated file should be returned. Remove the explicit caching code and Vert.x greatly outperforms Node.js. This tells you that Node does not take account of OS level caching.

Running a benchmark on a single core cloud VM is plain stupid. No-one who cares about performance will ever do this, they will run with multiple cores. If you do this, then Vert.x outperforms Node.js even further.

This is running Vert.x JS server. If you run the Vert.x Java server, performance will be even higher still.

This comment has been minimized.

The benchmark is valid because both vert.x and node.js are doing the same workload. If both are doing the same workload then it's totally valid. I could have them both calculating prime numbers, both resizing images, or both doing anything. They are both reading a file, explicitly caching it then serving it. Comparing performance for this operation is completely valid. Just because you don't like the results doesn't make it invalid. Sounds like sour grapes.

I ran this with multiple cores here: https://gist.github.com/2657432 and nodejs outperformed vert.x still. So the multiple core argument is invalid. That's even using the cluster functionality which isn't yet ready for prime time. I suspect if I used haproxy to load balance multiple instances I'd get even better numbers with nodejs.

The benchmark is valid because both vert.x and node.js are doing the same workload. If both are doing the same workload then it's totally valid. I could have them both calculating prime numbers, both resizing images, or both doing anything. They are both reading a file, explicitly caching it then serving it. Comparing performance for this operation is completely valid. Just because you don't like the results doesn't make it invalid. Sounds like sour grapes.

I ran this with multiple cores here: https://gist.github.com/2657432 and nodejs outperformed vert.x still. So the multiple core argument is invalid. That's even using the cluster functionality which isn't yet ready for prime time. I suspect if I used haproxy to load balance multiple instances I'd get even better numbers with nodejs.

Possibly. I dont know.

This comment has been minimized.

On 11/05/12 11:41, Courtney Couch wrote:
1. The benchmark is valid because both vert.x and node.js are doing the same workload. If both are doing the same workload then it's totally valid. I could have them both calculating prime numbers, both resizing images, or both doing anything. They are both reading a file, explicitly caching it then serving it.

So you're benchmarking a completely pointless thing.
How about we benchmark some Fibonacci generators? ;)

On 11/05/12 11:41, Courtney Couch wrote:
1. The benchmark is valid because both vert.x and node.js are doing the same workload. If both are doing the same workload then it's totally valid. I could have them both calculating prime numbers, both resizing images, or both doing anything. They are both reading a file, explicitly caching it then serving it.

So you're benchmarking a completely pointless thing.
How about we benchmark some Fibonacci generators? ;)

This comment has been minimized.

Serving the same single file over and over is pretty unrealistic. Lets say there were 10k different files and the requests would serve one randomly. I suspect node's read would load them faster (provided the sizes of the files are small). Then if you did want mmap type access there are node modules to handle that type of IO. IO isn't a once size fits all solution. I suspect using mmap based access using an mmap module would give you identical results to my results above.

Also if someone was really serving the same file over and over, I suspect it woud be kept in memory on the app server... or perhaps memcached... or redis... or perhaps use something that excels at static file delivery.. nginx.. or use a cdn.. In any case really I think we should just agree to disagree on what makes a relevant comparison. I don't see this getting anywhere.

Anyway, I do agree the tests above are trivial. My opinion is that they are no more so than the ones on your blog. In some situations vert.x will be faster I'm sure, and in some node.js will be faster (I suspect nodejs will be faster in most situations but thats opinion since there really are no in depth benchmarks comparing the two). It's a matter of figuring out given your problem space which is the better solution.

Serving the same single file over and over is pretty unrealistic. Lets say there were 10k different files and the requests would serve one randomly. I suspect node's read would load them faster (provided the sizes of the files are small). Then if you did want mmap type access there are node modules to handle that type of IO. IO isn't a once size fits all solution. I suspect using mmap based access using an mmap module would give you identical results to my results above.

Also if someone was really serving the same file over and over, I suspect it woud be kept in memory on the app server... or perhaps memcached... or redis... or perhaps use something that excels at static file delivery.. nginx.. or use a cdn.. In any case really I think we should just agree to disagree on what makes a relevant comparison. I don't see this getting anywhere.

Anyway, I do agree the tests above are trivial. My opinion is that they are no more so than the ones on your blog. In some situations vert.x will be faster I'm sure, and in some node.js will be faster (I suspect nodejs will be faster in most situations but thats opinion since there really are no in depth benchmarks comparing the two). It's a matter of figuring out given your problem space which is the better solution.

should be ignored. It's essentially a comparison between direct file system access vs cached file access. Absolutely not a comparison between vertx and node. It feels like an marketing attempt to get some attention.

This definitely does not mean that vertx is a bad product. It's a fresh breath of air for Java community. Java is a bloated environment where simplicity is absent. Your product brings this new spirit that exists on other languages. I suggest you focus on the features you summarize on your nice manual

But you should definitely stop giving false promises and hidden truths about the product. There is no way you can beat c/v8 in performance with java/rhino. You need to shine on the parts that makes the difference - like language independency. That's really a good one.

should be ignored. It's essentially a comparison between direct file system access vs cached file access. Absolutely not a comparison between vertx and node. It feels like an marketing attempt to get some attention.

This definitely does not mean that vertx is a bad product. It's a fresh breath of air for Java community. Java is a bloated environment where simplicity is absent. Your product brings this new spirit that exists on other languages. I suggest you focus on the features you summarize on your nice manual

But you should definitely stop giving false promises and hidden truths about the product. There is no way you can beat c/v8 in performance with java/rhino. You need to shine on the parts that makes the difference - like language independency. That's really a good one.

This comment has been minimized.

about the bs called "nobody runs on a single core"
when ever you rent some hosted machines, you rent a virtual machine where the host has 12 cpu cores but the vm it self just shows 8000mhz (so the max performance is limited per hosting package - my virtual machine can use 8000mhz per second but without seeing on which core my load is processed). in general the most hosting providers doesn't show the guest os on their machines how many cpus are available and here is why nodejs shines so bright.

about the bs called "nobody runs on a single core"
when ever you rent some hosted machines, you rent a virtual machine where the host has 12 cpu cores but the vm it self just shows 8000mhz (so the max performance is limited per hosting package - my virtual machine can use 8000mhz per second but without seeing on which core my load is processed). in general the most hosting providers doesn't show the guest os on their machines how many cpus are available and here is why nodejs shines so bright.

One thing to note however, if indeed the difference is that nodejs is using read() and vertx is using mmap() (I know for a fact nodejs uses read()) then nodejs will be more performance for one off small file reads. So if you only need to read a file once, or rarely, and the files are not HUGE then nodejs file reading will get more performance. There is a cost to using memory mapped os file reads. So keep that in mind if you switch to using mmap bindings.

One thing to note however, if indeed the difference is that nodejs is using read() and vertx is using mmap() (I know for a fact nodejs uses read()) then nodejs will be more performance for one off small file reads. So if you only need to read a file once, or rarely, and the files are not HUGE then nodejs file reading will get more performance. There is a cost to using memory mapped os file reads. So keep that in mind if you switch to using mmap bindings.

This comment has been minimized.

@fibric you make a good point. Also, node still shines on multicore setups (https://gist.github.com/2657432) as well ;) It's somewhat of a myth that nodejs can't be setup to take advantage of multiple cores. The tests there were even using node cluster which is known to be quite slow. I suspect haproxy + independent node instances would scale linearly with the cores (provided the processes are bound to individual cores).

@fibric you make a good point. Also, node still shines on multicore setups (https://gist.github.com/2657432) as well ;) It's somewhat of a myth that nodejs can't be setup to take advantage of multiple cores. The tests there were even using node cluster which is known to be quite slow. I suspect haproxy + independent node instances would scale linearly with the cores (provided the processes are bound to individual cores).

This comment has been minimized.

Note the mmap is probably only used when readFile() is used. If you use streams I don't believe it uses memory mapping, also (as mentioned before) if you use sendFile(), vert.x will avoid copying through userspace altogether.
This is highly efficient, but you only really see the benefits for larger files (which is why I didn't use it in the benchmark).

Note the mmap is probably only used when readFile() is used. If you use streams I don't believe it uses memory mapping, also (as mentioned before) if you use sendFile(), vert.x will avoid copying through userspace altogether.
This is highly efficient, but you only really see the benefits for larger files (which is why I didn't use it in the benchmark).

This comment has been minimized.

@tipiirai exactly. This is hardly a useful benchmark. Was merely meant to show that there's more to the story than the vert.x benchmarks show which has everyone saying "vert.x is 5x faster than nodejs!" which simply isn't true.

@tipiirai exactly. This is hardly a useful benchmark. Was merely meant to show that there's more to the story than the vert.x benchmarks show which has everyone saying "vert.x is 5x faster than nodejs!" which simply isn't true.

This comment has been minimized.

the moral of the story here is.. There are many different ways to handle disk IO on the OS level. You explicitly pick the bindings you want to use with nodejs via modules. You need to be aware of the ramifications of the pros and cons of the IO you are doing. The flaw in the original benchmarks that inspired me to do t his is that the two benchmarks were using two different IO methods on the OS level and trying to compare them.. It was more of a benchmark of read() vs mmap() on reading a single small file over and over again than it was a benchmark of node and vertx. I don't think @purplefox realized that. I suspect he isn't familiar with how node handles IO in order to realize he was comparing apples and oranges.

There are plenty of benchmarks comparing read() and mmap() and when it's faster to use one or the other. Reading the same small file over and over again is a bad use case for read() and you can see that in the results on the original vert.x benchmarks.

You need to pick the IO bindings based on the type of data access.. Don't always trust that some language will understand what kind of IO you want to do.. using just some file read method on whatever language you are using. Find out what bindings it's really using on the OS level to see if those fit your scenario (that's if you are worried about the IO performance)

the moral of the story here is.. There are many different ways to handle disk IO on the OS level. You explicitly pick the bindings you want to use with nodejs via modules. You need to be aware of the ramifications of the pros and cons of the IO you are doing. The flaw in the original benchmarks that inspired me to do t his is that the two benchmarks were using two different IO methods on the OS level and trying to compare them.. It was more of a benchmark of read() vs mmap() on reading a single small file over and over again than it was a benchmark of node and vertx. I don't think @purplefox realized that. I suspect he isn't familiar with how node handles IO in order to realize he was comparing apples and oranges.

There are plenty of benchmarks comparing read() and mmap() and when it's faster to use one or the other. Reading the same small file over and over again is a bad use case for read() and you can see that in the results on the original vert.x benchmarks.

You need to pick the IO bindings based on the type of data access.. Don't always trust that some language will understand what kind of IO you want to do.. using just some file read method on whatever language you are using. Find out what bindings it's really using on the OS level to see if those fit your scenario (that's if you are worried about the IO performance)

This comment has been minimized.

Yea I didn't really spend any time on this.. just a few minutes changing the IO silliness (oh and the other benchmark showing node on multiple cores). I really should have tried mmap as well but really I should be spending my time doing actual stuff instead of making benchmarks hah. I think I made my point though that vertx isn't simply 5x faster than node as is being propagated.

Yea I didn't really spend any time on this.. just a few minutes changing the IO silliness (oh and the other benchmark showing node on multiple cores). I really should have tried mmap as well but really I should be spending my time doing actual stuff instead of making benchmarks hah. I think I made my point though that vertx isn't simply 5x faster than node as is being propagated.