Load testing fluentd with wrk2 and OpenResty

We’ve written some complicated transform rules for fluentd to add fields to the record, serialize it with Apache Avro and dump it all into kafka. That was fine, until some service owners said “Hey, we don’t want to learn this Avro stuff, just let is write to fluentd on UDP” Management decided we should load test fluentd, so the ticket I got was:

“The test ideally should be UDP events sent to localhost at a rate of 1,500, 3,000 and 10,000 events per second.”

Great, so how do I manage that? A colleague told me about wrk2, which generates HTTP requests at precise rates, is fast, and has a low system impact. The thing is, it ONLY generates HTTP GET requests, and it’s written in C, which I wasn’t about to bother learning for this one-off effort.

Enter OpenResty. If you’re not familiar, OpenResty is nginx with a LUA interpreter built in, so you can write LUA scripts to handle requests. There are a lot of things OpenResty is wrong for, but this was one of the things I thought it could do well. Fast, multi-threaded, efficient. What does that have to do with “UDP events sent to localhost”? Well, I wrote just enough LUA to do that, based on GET requests generated by wrk2. The LUA script is here if you want to have a look. It returns 503 if it can’t send the request. In the code, you’ll see reference to “somefile.json”, that’s the mock payload which is sent to fluentd.

The results were interesting. fluentd is single threaded, so it just ran the core it was on at 100% while it read UDP packets out of the kernel buffer, but, it didn’t drop not one. When I ran it for 60 seconds, it generated 90,000 events and sure enough I got 90,000 Kafka messages out the other end of the pipeline. Way to go Ruby!