but when I run it in Docker I get only 1700reqs/s – CPUs seem to peak at about 55-65%. The same app, the same settings.

I tried increasing ab’s concurrency. The app itself is hosted in Passenger, I tried running it in 20 processes, in 40 processes (Passenger runs the app). Inside Docker it doesn’t seem to be wanting to go higher.

EDIT1 CPU set, CPU shares

EDIT2 bridge/host network

It is also not the docker bridge network. The effect is the same when I use net: host in Docker Compose

EDIT3 Scale

If I run second container with same code with different port exposed – I can get the CPU load up to 77%, but still not 100% like on bare-metal. Note that each of those containers run 20-40 processes load-balanced with Passenger inside.

EDIT4 Ubuntu’s problem?

Ok, it seems to have something to do with Ubuntu.
The same container ran on CoreOS – I’m able to saturate all cores.

But I still don’t understand the limitation.

EDIT5 DigitalOcean testing

To be completely fair I took 2 identical 16GB 8CPU instances on DigitalOcean, both in Frankfurt datacenter.
I installed app on most recent Ubuntu and most recent CoreOS alpha.

I’m not sure how to get exactly the same builds – it seems that CoreOS has Docker builtin and read-only FS and with Ubuntu – I have no idea how to get build exactly e21da33. But the general version is the same 1.10.0

I run ab from external machine on DigitalOcean also in Frankfurt datacenter to ensure that ab is not a variation. I hit the external IP in both cases. The parameters for ab are the same (ab -n 40000 -c 1000 -k), the code is the same.

Please don’t get excited (or flame me) – this is not the answer – I just need more space than a comment will allow! I am not a linux or Docker expert but I really like this sort of problem and have done some research over the weekend and have a few avenues to explore that may help. I don’t have a test rig so have reached an impasse.

Theories so far “For Debian and Ubuntu…”:

Docker is putting container and sub-processes into a cgroup that is
being throttled in some way.

The scheduler for the OS and the scheduler within the Docker
container (systemd?) are in some way ‘fighting’ for the CPU and
constantly displacing each other.

The OS scheduler is treating (a) the Docker Container and (b) the
app inside as separate competing resource requests and therefore
giving each about 50%

It seems to me that the RedHat flavours of linux have in some way
‘integrated’ docker (read “looked at what it does and tweaked their
OS setup or Docker setup to be compatible”). What have they changed
to do this? – it may be the thing that makes the difference.

There is a strong push for not using Docker under RHEL 6 but instead
to use RHEL 7+ – What did they change in RH between these versions
wrt. the CPU scheduling that makes them so keen on using 7+?

What I’d look at next:

cgroup setup whilst running.

Contents of any limits.conf files

Differences in Docker config files between the version on RH and
Ubuntu flavours.

(If I had time) see if Docker on RHEL 6 had the problem (as RHEL 7
doesn’t)