Today, Ubuntu made a Spectre bug fix on Ubuntu available, shipped
in kernel 4.4.0-112. As with the Meltdown fix, we measured the
effect of this update. Unfortunately, we observed a major
performance penalty on MySQL workloads with this new kernel.

In this blog post, we examine the recent
revelations about CPU security vulnerabilities.

The beginning of the new year also brings to light fresh and new
CPU security vulnerabilities. Today’s big offenders originate on
the hardware side – more specifically, the CPU. The reported
hardware kernel bugs allow for direct access to data held in the
computer/server’s memory, which in turn might leak sensitive
data. Some of the most popular CPUs affected by these bugs
are Intel, AMD and ARM.

The most important thing to know is that this vulnerability is
not exploitable remotely, and requires that someone
execute the malicious code locally. However, take extra
precaution when running in virtualized environments (see
below for more information).

The behavior of vm.swappiness was always a bit confusing for
novice linux users, as setting vm.swappiness to 0 would not
completely disable swapping in the system during a memory crunch.
vm.swappiness would only affect the agressiveness of swapping.

Following upstream commit tried to give more control to
parameter. This commit tried to avoid swapping completely when
vm.swappiness is set to 0.

With above commit, setting vm.swappiness to “0” instructs the
kernel not to initiate swapping until the amount of free and
file-backed pages is less than the high water mark in a memory
zone. In other words, it tries to reclaim as much memory that can
be reclaimed, before swapping starts.
This greatly reduced the chances of swapping.

Just under a year ago on my old blog I discussed and even demoed the new Linux live kernel patching
solutions. I was reviewing these technologies out of my own
curiosity as well as HP's Advanced Technology Group having an
interest. I think these technologies are great, I am personally
more of a fan of the user experience of RedHat's kpatch solution
but any solution is a great technical achievement.

Having said this I believe that the use case for this technology
is quite narrow. Last time I looked into these technologies only
patches that affected the code of functions could be modified.
Changing structs and data definitely didn't work and I suspect
that changing function declarations was also dangerous. There is
also a performance …

Building on the original kSplice idea and combining
the efforts of the work done at Red Hat and SuSE,
common infrastructure is now ready to be put into the Linux 3.20
mainline kernel – Red Hat and SuSE have already committed to
using this.

I still reckon it’s freaky trickery, but heck – it works, and
it’s great for server environments that have no redundancy (I
prefer to fix that issue!) and can’t afford any downtime.

I have recently been involved in diagnosing the reasons behind
OOM invocation that would kill the MySQL server process. Of
course these servers were primarily running MySQL. As such the
MySQL server process was the one with the largest amount of
memory allocated.

But the strange thing was that in all the cases, there was no
swapping activity seen and there were enough pages in the page
cache. Ironically all of these servers were CentOS 6.4 running
kernel version 2.6.32-358. Another commonality was the fact that
vm.swappiness was set to 0. This is a pretty much standard
practice and one that is applied on nearly every server that runs
MySQL.

Looking into this further I realized that there was a change
introduced in kernel 3.5-rc1 that altered the swapping
behavior when “vm.swappiness=0″.

Below is the description of the commit that changed
“vm.swappiness=0″ behavior, together with the diff:

There is much more to write about all the work we do at Facebook
with memory management efficiency on our systems, but there was
this one detour investigation in the middle of 2012 that I had to
revisit recently courtesy of Wikipedia.

There are lots of factors that make machines page out memory
segments into disk, thus slowing everything down and locking
software up – from file system cache pressure to runaway memory
leaks to kernel drivers being greedy. But certain swap-out
scenarios are confusing – systems seem to have lots of memory
available, with proper settings file system cache should not
cause swapping, and obviously in production environment all the
memory leaks are ironed out.

And yet in mid-2012 we noticed that our new kernel machines were
swapping out for no obvious reason. When it comes to swapping,
MySQL community will always point to Jeremy’s post on …

dm-cache is (albeit still classified “experimental”) is in the
just released Linux 3.9 kernel. It deals with generic block
devices and uses the device mapper framework. While there have
been a few other similar tools flying around, since this one has
been adopted into the kernel it looks like this will be the one
that you’ll be seeing the most in to the future. It saves
sysadmins the hassle of compiling extra stuff for a system.

A typical use is for an SSD to cache a HDD. Similar to a battery
backed RAID controller, the objective is to insulate the
application from latency caused by the mechanical device, the
most laggy part of which is seek time (measured in milliseconds).
Giventhe relatively high storage capacity of an SSD (in the
hundreds of GBs), this allows you to mostly disregard the
mechanical latency for writes and that’s very useful for database
systems such as MariaDB.

Disk I/O is still a common source of performance issues, despite
modern cloud environments, modern file systems and huge amounts
of main memory serving as file system cache. Understanding how
well that cache is working is a key task while investigating disk
I/O issues. In this post, I’ll show the activity of the ZFS file
system Adaptive Replacement Cache (ARC).

There are often more statistics available than you realize (or
have been documented), which may certainly be true with the ARC.
Apart from showing these statistics, I’ll also show how to extend
observability using dynamic tracing (DTrace). These tracing
techniques are also applicable to any kernel subsystem. This is
an advanced topic, where I’ll sometimes dip into kernel code.

Content reproduced on this site is the property of the respective copyright holders.
It is not reviewed in advance by Oracle and does not necessarily represent the opinion
of Oracle or any other party.