Stable Kernel Releases & Updates

The Linux kernel stable release model started in 2005, when it was determined
that the existing kernel development model (a new release every 2-3 months) was
not meeting the needs of most users. Users wanted bugfixes made during those 2-3
months, and Linux distributions found it difficult to keep kernels up to date
without feedback from the kernel community. In general, attempts to keep
individual kernels secure and with the latest bugfixes was a large and confusing
effort by lots of different individuals.

Stable kernel releases are based directly on Linus Torvalds' releases, and are
released every week or so, depending on various external factors (time of year,
available patches, maintainer workload, etc.). The numbering of the stable
releases starts with the number of the kernel release, and an additional number
is added to the end of it. For example, the 4.4 kernel is released by Linus, and
then the stable kernel releases based on this kernel are numbered 4.4.1, 4.4.2,
4.4.3, and so on. This sequence is usually shortened with the number 4.4.y when
referring to a stable kernel release tree. Each stable kernel release tree is
maintained by a single kernel developer, who is responsible for picking the
needed patches for the release and managing the review/release process.

Stable kernels are maintained for the length of the current development cycle.
After Linus releases a new kernel, the previous stable kernel release tree is
stopped and users must move to the newer released kernel.

Long-term stable kernels

After a year of this new stable release process, it was determined that many
different users of Linux wanted a kernel to be supported for longer than just a
few months. In response, the Long Term Supported (LTS) kernel release was
created, with the first LTS kernel (2.6.16) released in 2006. Since then, a new
LTS kernel has been selected once a year and kernel community maintains that
kernel for a minimum of 2 years.

At the time of this writing, the LTS kernels are the 4.4.y and 4.9.y
releases, and a new kernel is released weekly. Due to the needs of some users
and distributions, a few additional older kernels are maintained by kernel
developers at a slower release cycle. Information about all long-term stable
kernels, who is in charge of them, and how long they will be maintained, can be
found on the
kernel.org
releases page.

LTS kernel releases average 6-8 patches accepted per day, while the normal
stable kernel releases contain 10-15 patches per day. The number of patches
fluctuates per release given the current time of the corresponding development
kernel release, and other external variables. The older a LTS kernel is, the
less patches are applicable to it as many recent bugfixes are not relevant to
older kernels. However, the older a kernel is, the harder it is to backport the
changes that are needed to be applied, due to the changes in the codebase. So
while there might be a lower number of overall patches being applied, the effort
involved in maintaining a LTS kernel is greater than maintaining the normal
stable kernel.

Stable kernel patch rules

The rules for what can be added to a stable kernel release have remained
almost identical since its introduction and are summarized below:

Must be obviously correct and tested.

Must not be bigger than 100 lines.

Must fix only one thing.

Must fix something that has been reported to be an issue.

Can be a new device id or quirk for hardware, but not add major new
functionality.

The last rule, "Must already be merged into Linus Torvalds' tree", prevents
the kernel community from losing fixes. The community never wants a fix to go
into a stable kernel release that is not already in Linus Torvalds' tree, so
that anyone who upgrades should never see a regression. This prevents many
problems that other projects who maintain a stable and development branch can
have.

Kernel updates

The Linux kernel community has promised its userbase that no upgrade will
ever break anything that is currently working in a previous release. That
promise still holds true today. Regressions do happen, but those are the highest
priority bugs and are either quickly fixed, or the change that caused the
regression is quickly reverted from the Linux kernel tree.

This promise holds true for both the incremental stable kernel updates, as
well as the larger major updates that happen every three months. However, the
kernel community can only make this promise for the code that is merged into the
Linux kernel tree. Any code that is merged into a device's kernel that is not in
the kernel.org releases is unknown and
interactions with it can never be planned for, or even considered.

Devices based on Linux that have large patch sets can have major issues when
updating to newer kernels, because of the large number of changes between each
release (10-14 thousand changes per release). SoC patchsets are especially known
to have issues with updating to newer kernels due to their large size and heavy
modification of architecture specific, and sometimes core, kernel code. As a
result, most SoC vendors are starting to standardize on using the LTS releases
for their devices, enabling those devices to receive bug and security updates
directly from the Linux kernel community.

Security

When doing kernel releases, the Linux kernel community almost never declares
specific changes as security fixes. This is due to the basic problem of
the difficulty in determining if a bugfix is a security fix or not at the time
of creation. Also, many bugfixes are only determined to be security related
after much time has passed, so the kernel community strongly recommends always
taking all bugfixes that are released.

Note: For details on Linus Torvalds'
statement on security fixes, refer to the relevant
email
thread.

When security problems are reported to the kernel community, they are fixed as
soon as possible and pushed out publically to the development tree and the
stable releases. As described above, the changes are almost never described as
a "security fix", but rather look like any other bugfix for the kernel. This is
done to allow affected parties the ability to update their systems before the
reporter of the problem announces it.

For details on reporting security bugs to the kernel community to get
them resolved and fixed as soon as possible, refer to
Security
bugs in The Linux kernel user's and administrator's guide at
www.kernel.org.

Because security bugs are not announced to the public by the kernel team, CVE
numbers for Linux kernel-related issues are usually released weeks, months, and
sometimes years after the fix was merged into the stable and development
branches.

Keeping a secure system

When deploying a device that uses Linux, it is strongly recommended that all
LTS kernel updates be taken by the manufacturer and pushed out to their users
after proper testing shows the update works well. This has several advantages:

Releases have been reviewed by the kernel developers as a whole, not in
individual parts.

It is hard, if not impossible, to determine which patches fix "security"
issues and which do not. Almost every LTS release contains at least one known
security fix, and many yet "unknown".

If testing shows a problem, the kernel developer community will react
quickly to resolve the issue.

Attempts to filter out only the changes you run will result in a kernel
tree that is impossible to merge correctly with future upstream releases.