Running AWS Fargate with virtual-kubelet

AWS Fargate is a new compute engine that allows you to run containers without having to provision, manage, or scale servers.

Today, Fargate is natively integrated with Amazon Elastic Container Service (ECS). Since we announced Fargate at re:Invent 2017, many customers have told us that they were very interested in using Fargate to run Kubernetes. To help make this happen, we’ve started exploring using virtual-kubelet to connect Fargate and Kubernetes.

This new plugin is an exploratory step towards extending AWS Fargate’s managed data plane to Kubernetes, and we have a lot to do before it works seamlessly for all use cases. For now, here are few simple steps to try it for yourself:

Step 1: Set up a single-master Kubernetes cluster using kops

We will set up a simple test environment to showcase Kubernetes workloads backed by AWS Fargate.

The first step is to set up a kops cluster. In our kops set-up we are using a hybrid environment with two EC2 worker nodes and one virtual-kubelet instance. Any Kubernetes workloads that are not supported on Virtual-Kubelet and Fargate, can easily be scheduled as usual on the two worker nodes. Follow the instructions below using this repository: github.com/kubernetes/kops/blob/master/docs/aws.md. Our blog post on managing Kubernetes clusters on AWS using kops may also be helpful.

$ export KUBERNETES_VERSION=https://storage.googleapis.com/kubernetes-release/release/v1.9.0/
$ export AWS_AVAILABILITY_ZONES=us-east-1
$ export CLUSTER_NAME=dnishi-kops1.k8s.local
$ kops create cluster —name $CLUSTER_NAME —zones $AWS_AVAILABILITY_ZONES —kubernetes-version $KUBERNETES_VERSION --yes
##Tested with Kubernetes v1.8.0+
##Kops 1.6.2+ can be easily created as a gossip-based cluster and the cluster name has to end with k8s.local

Once the kops install finishes, verify your nodes are running with kubectl:

All dependencies are vendored-in to the repository and will be built together with virtual-kubelet. The binary is placed in virtual-kubelet/bin directory.

Step 3: Create your Fargate task

If you haven’t used AWS Fargate before, the easiest way to get started is with Fargate’s first run experience. This will set up Fargate in your AWS account with the default settings. It will create a default Fargate cluster*, IAM roles, and a default VPC with an internet gateway and a default security group. After the first run, it’s easy to fine-tune individual settings. Once you have your first application on Fargate running, visit the AWS ECS console to take a closer look at your Fargate resources. Take a note of the sample-app’s subnets and security groups.

*because Fargate abstracts away servers, the cluster is used only as a logical construct to group running containers.

Step 4: Create an EC2 instance to run virtual-kubelet with the Fargate provider plugin

Create an IAM Role for the EC2 instance. Many AWS services require that you use roles to control what a service can access. When an IAM role serves a specialized purpose for a service, it is categorized as a service role for EC2 instances (for example), or a service-linked role. The EC2 instance running virtual-kubelet needs full access to the ECS APIs. Use the following steps to create the right IAM role:

Deploy nginx with three replicas by using a text editor to create a YAML file with these contents and running it. To assign pods on Fargate (virtual-kubelet), we use nodeSelector constraint in the manifest, which is a field of PodSpec. It specifies a map of key-value pairs. You can also use Kubernetes taints to simulate this behavior.

Notice the nginx pod metadata in bold is reflected in the Fargate Dashboard below as a task definition.

Let’s update the nginx-deployments running on Fargate from v1.7.9 to v1.9.1:

$ kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1
deployment.apps "nginx-deployment" image updated
$ kubectl rollout status deployment/nginx-deployment
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for rollout to finish: 1 old replicas are pending termination...
Waiting for rollout to finish: 1 old replicas are pending termination...
deployment "nginx-deployment" successfully rolled out

Fargate tasks also get updated one task at a time. If you click on a task, you can see the image version update from v1.7.9 to v1.9.1. Within seconds, each task updates to v1.9.1 and this process repeats for each task one at a time.

Deploy 20 pods with a single command. Use the nginx-deployment-new.yaml. As reflected in the Fargate dashboard, the tasks are created in seconds:

Closing Thoughts

Today, the virtual-kubelet provider plugin does not support every Kubernetes use case and application pattern. This is our first step to give Kubernetes users a new way to think about compute, and we are excited to engage with the community on how best to implement nodeless design concepts for Kubernetes. We hope you will join us on this journey, suggest ideas, contribute code, and help test this plugin for your applications. Here’s how you can participate:

Attend the virtual-kubelet working group meeting, held every Thursday at 3:00 pm PST via Zoom.

AWS will be at KubeCon EU in Copenhagen, Denmark on May 2nd to 4th 2018 , demonstrating this integration as well as our upcoming managed Kubernetes service, Amazon Elastic Container Service for Kubernetes (Amazon EKS).

NOTE: This post is written specifically for intermediate-level users. A grasp of Linux, Go, Docker, and K8s is expected. Also, virtual-kubelet is not yet a stable code base. Don’t be surprised if virtual-kubelet crashes. We need help to make it stable!

Nishi Davidson

Nishi Davidson has been in the cloud infrastructure and software application space for 15 years working across engineering, product management and strategy in South East Asia and the US markets. Currently she is responsible for AWS’s open source engineering efforts in the Kubernetes community. In the past, she ran SAP’s private cloud, Kubernetes managed service engineering for internal BUs. Nishi has led product/field engineering teams and introduced multiple cloud products/solutions to the market while working at HP, TCS, Juniper Networks, NetApp and DSSD (EMC). Nishi holds an MBA from Massachusetts Institute of Technology, Sloan and a bachelor’s in Electrical and Electronics Engineering from CEG, Guindy, Anna University.

Onur Filiz

Onur Filiz started coding during the 8-bit home computer era and never stopped since then. He is currently a Principal Software Engineer in AWS, working on Elastic Container Service (ECS), Elastic Container Service for Kubernetes (EKS) and AWS Fargate. His past work areas include operating systems, networking, containers and the cloud. Onur holds an M.Sc. in Telecommunications and Computer Networks from GWU and a B.Sc. in Computer Science from Ege University. You can get in touch with him on Twitter @onurfz.