Migrating to Amazon Linux 2

I run all my EC2 workloads on Amazon Linux. It comes with a superb AWS integration, a secure default configuration, regular security updates, and I can open AWS Support tickets if I run into any problems.

In late December 2017, AWS announced the successor of Amazon Linux: Amazon Linux 2.

AWS also announced that Amazon Linux 2018.03 is the last release for the current generation of Amazon Linux and will be supported until June 30, 2020. Therefore, you have to come up with a migration plan.

Amazon Linux 2 comes with the same benefits as Amazon Linux, but it adds some new capabilities:

long-term support: Amazon Linux 2 supports each LTS release for five years

on-premises support: virtual machine images for on-premises development and testing are available

Long-term support

The Amazon Linux delivers a continuous flow of updates that allow you to roll from one version of the Amazon Linux AMI to the most recent. A yum update always moves your system to the latest Amazon Linux version. There were no versions of Amazon Linux available, only snapshots.

Amazon Linux 2 changes this. You will have Amazon Linux 2 versions that are supplied with updates for five years. Once a new Amazon Linux 2 LTS release becomes available, no breaking changes will be introduced by AWS for this release.

systemd

Amazon Linux uses SysVinit to bootstrap the Linux user space and to manage system processes after booting. This procedure is usually called init. One of the major drawbacks of SysVinit is that it starts tasks serially, waiting for each to finish loading before moving on to the next. This can result in long delays during boot.

Amazon Linux 2 uses systemd as the init system. systemd executes elements of its startup sequence in parallel, which is faster than the traditional serial approach from SysVinit. systemdcan also ensure that a service is running (e.g., it restarts a service if it crashed).

systemd is not just the name of the init system daemon but also refers to the entire software bundle around it, which includes:

journald: responsible for event logging (replaces syslog)

udevd: device manager for the Linux kernel, which handles the /dev directory and all user space actions when adding/removing devices

logind: manages user logins and seats in various ways.

I will not cover udevd and logind in this post. You should not get in touch with them as a normal user like me. Keep in mind that networking configuration is not controlled by networkd (also part of systemd software bundle). Instead, networking configuration is controlled by cloud-init which is triggered by systemdseveral times during boot. cloud-init handles early initialization of an EC2 instance (also works with other vendors).

Reading logs from journald

To read all system logs (journal in journald terminology), starting with the oldest entry, run journalctl. The output is paged through less by default. Which means you can scroll down / up an entry with the DOWN / UP arrow keys, or scroll a full page down/up with the SPACE / b keys. Press the q key to quit. To reverse the order, run journalctl -r.

To show only the most recent journal entries, and continuously print new entries, run journalctl -f (like a tail -f).

There are many ways to filter the output. Based on priority, run journalctl -p err to get levels alert, crit, and err (using syslog log levels). Based on the unit, run journalctl -u sshd to get all entries for sshd. Check the further reading links for more information.

Keep in mind that some applications still write logs to /var/log. Journald also forwards logs to rsyslog which is configured (/etc/rsyslog.conf) to write some of them to files in /var/log.

Extras Library

The Extras Library (aka Amazon Linux Extras Repository or Extras mechanism), provides a way to install up-to-date software bundles (topics in Amazon Linux 2 terminology) without impacting the stability of the rest of the operating system.

The awslogs agent does not support journald

awslogs agent cannot read logs directly from the journal. journald fowards all logs to rsyslogwhich is configured (/etc/rsyslog.conf) to write some of the logs to files in /var/log from where the awslogs agent can pick them up.

Where are the log files?

/var/log does not contain all system logs anymore.

If in doubt, you can access all system logs with journalctl.

Ruby is missing

Ruby is no longer installed by default. This breaks cfn-init if you want to install RubyGems.

You can install Ruby 2.0 with yum install ruby or Ruby 2.4 with amazon-linux-extras install ruby2.4.

netcat is missing

netcat (or nc) is no longer installed by default.

You can install ncat with yum install nmap-ncat, but this will install nmap based ncat which behaves differently (e.g., no -z flag anymore). Learn more

Nginx package not available by default

To install nginx, use the new Amazon Linux Extras Repository amazon-linux-extras install nginx1.12.

EPEL repository is missing

The EPEL repository (Extra Packages for Enterprise Linux) is no longer installed by default or available to install. The Extras Library replaces the EPEL repository but contains only a fraction of the packages which may causes troubles during your migration.

NAT and ECS optimized AMIs are missing

NAT and ECS optimized AMI are not available. You can replace your NAT instances with NAT Gateways to get around this problem. But for ECS workloads there is no easy workaround. I advise waiting for news from AWS regarding the ECS optimized AMI.

cfn-init is not integrated with the Extras Library

You can not install packages from the Extras Library with the package mechanism in cfn-initeasily. cfn-init is the way how you can install software onto EC2 instances managed by CloudFormation.

There can either run amazon-linux-extras enable <topic> before running cfn-init which than can install the package by using the package mechanism. Or you can use two config sets. The first config sets uses the command mechanism to enable the topic. The second config set uses the package mechanism to install the enabled package. You have to use two config sets because commands run after package installation. Here is an example:

Summary

Amazon Linux 2 is the new default for running Linux workloads on AWS. Amazon Linux 2 benefits from systemd, LTS, and a new extras library. There are a few pain points when migrating, most notably the missing EPEL repository. Besides that, you should spend some time to understand how systemd works, because that’s central in modern Linux operating systems.

Easy Cloud Company

Established in 2010, Easy Cloud began with the goal of helping small to medium businesses migrate to G Suite (formerly Google Apps) for email and operations, WordPress for websites, and Untangle for security.