The gzip optimizations currently included are for multithreaded environments only (e.g. pigz). In many cases, like basic VPS on DigitalOcean, Vultr, AWS, etc. we will only have one CPU core. pigz and other multithreaded solutions don't help in that case. Could you please add the gzip optimizations made by Cloudflare, Intel, or the zlib-ng project? They're all single-threaded as far as I know, and so they give a nice boost even with one CPU core.

(That was a year ago. It looks like Cloudflare generally does the best, but for the specific case of compressing HTML at level 5, which is similar to what nginx does (usually level 6), Intel and zlib-ng are actually fastest. Their code has changes since then.)

did a quick test of bench.pl from zlib-bench/bench.pl at master · jsnell/zlib-bench · GitHub with updated git repo commit versions but seems cloudflare's zlib fails to compile on CentOS 7 while the others compiled at least. Seems for cloudflare CRC32C support you need Intel proccessor with msse4.2 hardware support. And my Intel Core i7 3635QM processor i believe does support such but still ? Might need to pass a CFLAG for -msse4.2 ?

One thing to be aware of, these replacement zlib libraries are not always "drop in" safe as some software such as nginx pre-allocate the zlib state buffers by hard coding in the original zlib structure sizes (the original zlib makes guarantees about memory usage that technically makes this behavior somewhat OK). Depending on how well the software is designed, the forked zlib versions may result in buffer overflows or other crashing. (nginx is fine - it spams your log file with errors and re-allocates).

did a quick test of bench.pl from zlib-bench/bench.pl at master · jsnell/zlib-bench · GitHub with updated git repo commit versions but seems cloudflare's zlib fails to compile on CentOS 7 while the others compiled at least. Seems for cloudflare CRC32C support you need Intel proccessor with msse4.2 hardware support. And my Intel Core i7 3635QM processor i believe does support such but still ? Might need to pass a CFLAG for -msse4.2 ?

Click to expand...

Your 3635QM definitely supports SSE 4.2. Your CPU is an Ivy Bridge generation, and SSE 4.2 was introduced in Nehalem in 2008. Nehalem was before Sandy Bridge, which was before Ivy Bridge.

At this point, any cloud VPS provider will support SSE 4.2 instructions. I just ran CPUID on one of my Vultr servers and it says AVX2 is available, which surprised me. That means that they're using Haswell or newer CPUs, so they're way past SSE 4.2. Amazon AWS also uses Haswell, and I think everyone's servers are going to at least be Sandy Bridge. Older chips than that are too expensive in that they use much more electricity and they're not able to run as many customers/VPSes on one machine.

The mpclmul instruction is also important – it's carryless multiplication. I think they both use it.

Do you know if the Intel and Cloudflare patches are working in your build, or is there a problem like the guy on Hacker News was saying?

Your 3635QM definitely supports SSE 4.2. Your CPU is an Ivy Bridge generation, and SSE 4.2 was introduced in Nehalem in 2008. Nehalem was before Sandy Bridge, which was before Ivy Bridge.

At this point, any cloud VPS provider will support SSE 4.2 instructions. I just ran CPUID on one of my Vultr servers and it says AVX2 is available, which surprised me. That means that they're using Haswell or newer CPUs, so they're way past SSE 4.2. Amazon AWS also uses Haswell, and I think everyone's servers are going to at least be Sandy Bridge. Older chips than that are too expensive in that they use much more electricity and they're not able to run as many customers/VPSes on one machine.

The mpclmul instruction is also important – it's carryless multiplication. I think they both use it.

Do you know if the Intel and Cloudflare patches are working in your build, or is there a problem like the guy on Hacker News was saying?

Click to expand...

Yeah it was the flags i needed to enable for SSE 4.2 supported cpus with mpclmul instructions. But I think Nginx itself is not liking Cloudflare's zlib fork that's why Nginx failed to compile.

It's probably complicated enough that it's not something I'd have free time to look into more right now given that it's not working with available codes from Cloudflare or Intel. Unless someone finds a better working code and tests it to work.

It's probably complicated enough that it's not something I'd have free time to look into more right now given that it's not working with available codes from Cloudflare or Intel. Unless someone finds a better working code and tests it to work.

Click to expand...

Okay, thanks for the quick reply. I might ask Cloudflare to make their fork usable. It's often hard to use their patches because they just post them without any documentation or obvious way to use them, and don't answer questions on their blog.