Hagbarddenstore

A collection of differences between non-China AWS and China AWS and how I’ve
solved them.

Recently I got my hands on a project which is to be deployed across multiple
regions in the world, one of them being China. Naive as I where, I thought
my CloudFormation stacks would just work in China, since they worked so
perfectly in both Europe (eu-west-1) and the US (us-east-1). But… I was
wrong. Very wrong.

After quite a lot of trial and error, I finally managed to get my modified
stack running in China, success!

So, to unburden the pain from you, I’ve compiled a list of all the caveats
I’ve found so far.

This blog post assumes you’re using some like CloudFormation, Terraform or
running commands via aws-cli. The experience will be different in the
AWS Console since it guides you to do the correct choices.

Instance types

This one is quite obvious, but still a bit of a pain.

China is lagging behind a bit when it comes to instance types, so please take
an extra look into your scripts before running them to ensure that the instance
types you’ve selected are available.

AMIs

This really shouldn’t be a surprise at all, since it’s different in all regions.
What caught my attention, was that some images (Ubuntu 16.04) aren’t up to date
to the current version, it even differs across the two regions in China.

Example: In cn-north-1 Ubuntu 16.04 20180109 is available, but in cn-northwest-1
it’s not. This is a bit weird, since 20180109 is available everywhere, but
cn-northwest-1.

I’ve written a small Python script to find all AMIs for Ubuntu 16.04 called
image-map.py and it contains this:

Replace ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180109 with
a string that matches your wanted AMI and then run the script twice, one with
a AWS “the world” account and one with a AWS China account. Example:

IAM trust entities

Another thing that surprised me is that IAM trust entities sometimes are named
differently. The surprise stems from that I assumed the name of the entities to
be something that’s internal to AWS, in which case they are free to choose
whatever name they want, but this wasn’t the case.

The entities I use are monitoring.rds.amazonaws.com, ecs.amazonaws.com
and ec2.amazonaws.com. Both rds and ecs are named the same in China as they
are elsewhere, but ec2 is not. It’s called ec2.amazonaws.com.cn in China.

What I did in my CloudFormation template was to create a condition and use
Fn::If to select which entity name to use. Example:

This snippet uses the same condition to select the correct policy ARN, since
they too are different in China.

There can be other entities that follow the same scheme, but I only came
across these since I mainly use RDS, ECS and EC2, but be aware of Lambda
and other services that heavily rely on IAM roles/profiles.

ARNs

As I just mentioned, ARNs are different in China as well, but they are quite
easy to manage, simple replace arn:aws: with arn:aws-cn: and you’re done!

Given we use the condition China which I defined in the snippet above, we
can select the correct ARN doing this:

Cross-region VPC peering connections

No big surprise here, given that cross-region capability is quite new and
not introduced to that many regions yet, but it still made me a bit sad.

Non-mirrored package repositories

This caused the most pain of all issues. Package repositories aren’t mirrored
in China. The Ubuntu repositories are mirrored, so if you only install packages
from those repositories, you’re fine. But if you like me, run Docker or Gitlab,
you’re gonna have intermittent difficulties reaching the official mirrors.

There is a solution though, since companies inside China offers mirrors that
works.

The GPG key can be fetched from http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg.

Same procedure is needed for Gitlab, replace the official
deb https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/ xenial main with
deb https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/ubuntu xenial main
provided by https://mirror.tuna.tsinghua.edu.cn/ (I guess this is some
kind of school). The official key can be used for this repository.

I haven’t found any need for other mirrors yet, but I’m sure there is a
need depending on which software you use.

No Route53

Again, no surprise, Route53 isn’t available in China. But, you can configure
Route53 with an AWS “world” account and use Route53 in China, since there are
edge nodes in China. You just can’t access the API in China via amazonaws.cn.

IAM isn’t global

IAM is supposed to be global, which it almost is, except China. China has their
own installation of IAM, so you need to duplicate your users from AWS into AWS
China. Not a big issue, but something to be aware of.

Final words

I think it’s amazing that we can use AWS in China, even with the limitations
and caveats. I think there’s still a lot for me to learn about China, so I’ll
probably revisit this post at a later time and update it with new problems and
solutions.

I want to start by saying that I’m quite happy with my current employer, this
post isn’t written to speak ill of my employer, it’s just my view on working as
a remote employee at a company where the culture isn’t exactly remote friendly.

This post is going to be about the problems you can encounter as a remote
employee working for a company where most or all other employees are working at
the same office. I’ll try to suggest solutions to these problems, but not all
are tested, so implement with causion.

Meetings

While chatting and emailing are great tools, sometimes it’s better to be able to
talk to eachother and see eachother. That’s when you need to hold a meeting.
The best way to do this is to meet face to face, but this is quite impractical
if people are scattered across the globe.

Here’s my list of things to do before, during and after having a meeting:

Write an agenda. Decide how much time to allocate for the meeting. Meetings
without an agenda and an end time are wasted meetings.

Book a time when everyone needed for the meeting are able to attend. Put it
into a shared with an alert the day before and an hour before.

Decide the conference tool to use and preferably test it before starting
the meeting. This can be tested between different attendees or a few minutes
before the meeting with all attendees. I recommend Zoom
for remote conferences.

Notify at least 10-15 minutes before the meeting if you’re unable to attend
or will be late. Don’t postpone the meeting for a single person, because that
ruins the schedule for all the other attendees.

Appoint someone to record the meeting and transcribe it and store it
somewhere where everyone can read it at a later time. This is good to remember
what was decided and also for others to catch up on what was said and decided
on the meeting.

If you’re gonna treat attendees with some fruit / cookies / cake / etc, make
sure that the remote attendees also gets treated something. (Call the nearest
bakery or tell them to go buy something at the expense of the company.)

Follow up on what was decided on the meeting after a previously decided time
to implement.

Planning

Planning, planning and more planning! Everyone loves (or hates) planning. It’s
even harder to do properly if you don’t use tools that suite your model of
working. But it needs to be done.

So, tips on making planning easier for everyone, even your remote workers!

Maintain a shared 1 year plan, 6 month plan, 3 month plan and a sprint plan.
The plans should contain features, not tasks.

Keep a shared backlog with all tasks, even tasks that are not connected to a
feature. (Maintenance tasks, bug fixes, etc)

Find the tools that allows you to follow the above structure.

As an employee, maintain your own weekly Kanban board. My Kanban board
consists of three columns: TODO, DOING, DONE. This works well for me. Whenever
a task ends up in DONE, update it in the shared tool as well.

Find a method, try it out at least 6 months, then adjust if needed. Switching
methods of planning often is not fun.

Don’t apply my tips as law. Try them, adjust them and scrap them if needed. It’s
what works for me.

Communication

Written communication is better than oral communication.

With that said, there are different kinds of written and oral documentation.

Chat

Chat is great! Chat sucks!

Chatting is a great tool if the message isn’t super important, it’s a nice place
to vent of some steam. But, if important discussions are happening in the
chatting tool, please summarize what was decided and put it in a blog post or
wiki article.

Email

I love emails. I love email even more when the sender can write a proper email.

So what do I define as a proper email? Well, it needs a subject, preferably
a well written one so I know at glance if I want to read this email now or if
it can be read at a later time. Then it needs content. If you think the whole
thing fits into the subject, don’t send an email, send a message in the chat
application.

So, what is email good for? I find it useful for meeting summaries,
announcements and different kinds of reports.

Requests and tasks should not be sent via email, it should be put into the
planning tool and assigned to me.

SMS

SMS is great! It’s fast, works most of the time and usually gets the attention
of the recipient. But! Use SMS with extreme caution. It should only be used in
life or death situations (Servers are on fire, major parts of the product is
down, etc). SMS should be reserved for alert systems like
PagerDuty or similar.

Conference call (Skype, Zoom, etc)

Use this for meetings when there are remote attendees.

Phone call

As with SMS, use with extreme caution. Is quite nice to use for person to person
communication if the call has been scheduled.

Blog

I love blog posts. They’re straightforward, usually structured and I can read
them whenever it suits me. I wish more people used blogs when communicating on
how to do stuff, announcements, reports and such things.

Wiki

A wiki is great for persistent documentation, stuff that you want to search and
find easily when there’s a problem. It shouldn’t be filled with articles on how
to setup Docker or how to brew coffee. That sort of stuff is better suited for
a blog. What I do expect to find in a wiki are panic lists, documentation on
how stuff is set up, where to find what info and more. That’s what I expect to
find in a wiki. I want to visit the wiki when something is burning and be able
to simply click my way around to find solutions.

Work hours

There’s so much to be said about work hours, but it pretty much boils down to
this: Set a schedule, put it in a shared calendar and communicate whenever
the reality differs from the schedule.

Off-work activities

Activities with your collegues can be great fun, but it requires planning,
especially when there are remote workers. But in general, try to plan things
to be done whenever the remote employees are present. It’s not fun to be the
only one who always misses out on friday beers, go-karting and what not.

I think that’s all I want to write at the moment. Feel free to contact me on
Twitter if you have any comments.

A light introduction to Ansible, promoting best practices and install cowsay
on Ubuntu 14.04.

What the hell is Ansible you ask? Well, I’m here to help!

Ansible is a tool to configure Linux (And Windows) servers via SSH and Python
scripts. It allows you to write scripts in
YAML and Python, which are executed
against and on remote servers.

Why should you use Ansible? You should use Ansible if you want to avoid
tedious and error prone manual work. Sure, it’s fine to run a few commands
on your server to install a few applications, change some configuration
files and so on, but fast forward a year, do you still remember what you did?
Can you quickly run those commands again if you need a second server or need
to replace the existing server?

If you answered yes on both both questions, Ansible isn’t for you. However, if
you didn’t answer yes on both questions, tag along on my journey to teach you
what Ansible is, how to use it for a single server and in large deployments.

Install Ansible

First things first, we need to install Ansible onto your computer, this is the
control computer, the one that executes the commands on the remote targets. The
target computers doesn’t need to have Ansible installed (But they do need
Python installed!).

The easiest and recommended way of installing Ansible is via Pythons
pip tool.

Run the following command to install Ansible via pip:

pip install --user ansible

This installs Ansible into my local Python library, which on my Mac OS X
computer is located at /Users/myname/Library/Python/2.7/bin/ansible. Be
sure to add /Users/myname/Library/Python/2.7/bin to your $PATH variable
to be able to run Ansible properly.

Run ansible --version to verify you have a working installation.

Your first command

Let’s start off by pinging your computer:

ansible -m ping localhost

This will print something similar to this:

localhost | SUCCESS => {
"changed": false,
"ping": "pong"
}

If you get a warning about a missing host file (/etc/ansible/hosts), just
ignore it, we won’t use that file anyway.

The command we ran is Ansibles equivalent of running ping localhost, but
it verifies that it can properly connect to the host. In the case of
localhost, it should always work.

Ad-hoc commands

Remember I said earlier that Ansible is a bunch of scripts you can run
against your targets? Well, Ansible can run stand-alone commands against
your targets as well.

To print the current time according to your computer, run this:

ansible -a "date" localhost

The expected output looks something like this:

localhost | SUCCESS | rc=0 >>
Tue Jun 28 22:15:10 CEST 2016

You now know how to run ad-hoc commands against your computer.

Recap

Let’s recap what we’ve learnt so far.

How to install Ansible

How to make sure the connection to a target works

How to display the current date and time according to a target

But let’s dig a bit deeper and try to understand what the parameters to
the ansible command means.

-m = The module to run. A module in Ansible is a Python script to be
executed on the target. The default module if none is specified is
the shell module. It executes it’s arguments as a standard shell command.

-a = The module arguments. A module can accept zero or more arguments to
decide what to do. In the case of figuring out the targets date, we used
-a with a value of date but didn’t specify a module to run. This
forwards date to the shell module, which runs the command.

localhost = The host pattern to match against. We used the full name of the
host, but you can specify a regex as well, like this: db[0-9], which
will try to connect to all hosts matching the regex. This however, requires
an inventory file. More on that later.

Inventories

So, let’s talk about inventories. An inventory is an ini-like file which
contains all your targets. It can look like this:

127.0.0.1

Altough that will work, it’s not very helpful. Let’s add a name to the host.

my-computer ansible_host=127.0.0.1

This gives is a nicer output, but it requires the remote target to have a
user named the same as your local user. Let’s add that:

my-computer ansible_host=127.0.0.1 ansible_user=myname

Save the file somewhere, call it whatever. I usually create a directory for
my project and create a folder called inventories inside that folder, then
save my inventory file inside that directory. So I end up with something
like this:

.
└── inventories
└── development

My inventory file is called development.

Now we have a complete inventory. To add more targets, simply add a new
line with the information needed.

Playbooks

The collection of scripts to be applied to a target are called a playbook in
Ansible. Let’s make one!

And that’s your first playbook! Save it as playbook.yml in your project directory.

Let’s explain the parts of it.

- hosts: my-computer defines which hosts to apply the tasks to. This can
contain the name of a host or a regex to match hosts. It can also be a group
or the special group all which matches all hosts in an inventory.

tasks: defines a list of tasks to be executed from top to botton on the
target.

- name: install cowsay is your first task. The name isn’t required, but
highly recommended to have. You can name it whatever you want.

apt: > let’s Ansible know that we want to execute the apt module.

name=cowsay is the first argument to the apt module. It’s the name of
the package we’d like to install. Different modules
have different arguments.

update_cache=yes lets the apt know we want to run apt-get update before
installing the package.

become: yes lets Ansible know that we want to run this module with sudo.
So become: yes is equivalent to sudo my-module.

Now that we understand the playbook, let’s run it!

ansible-playbook -i inventory/development playbook.yml

We’re using a new command, ansible-playbook, which is what’s used to
execute a playbook against targets.

The -i inventory/development tells Ansible to use our inventory file
to create a collection of targets to execute the playbook against.

When you run this command, you should end up with something like this:

–name = Name of the instance running etcd, this must be a within the
cluster unique identifier. It’s used by etcd to separate nodes apart. The
hostname or machine id are good candidates.

–initial-advertise-url = URL to advertise to other etcd nodes to allow
internal etcd communication. The hostname or any IP address which other etcd
nodes can reach are good values for this parameter.

–listen-peer-urls = URL on which etcd listens for internal etcd
communication. This should be the same value as --initial-advertise-url in
most cases.

–listen-client-urls = URLs on which etcd listens for client
communication. This is the address you use to communicate with etcd.
Preferably you want to add both 127.0.0.1 and a public IP, to allow both
localhost communication and external clients.

–advertise-client-urls = URLs which etcd advertises to the cluster. This
could be the same as --listen-client-urls minus the localhost address.

–discovery = URL to a discovery service, used by nodes to discover the
cluster when no previous contact has been made. This value should be the same
on all nodes you wish to include in the same cluster. You can get a new URL
by running curl https://discovery.etcd.io/new?size=3 where 3 is the
minimum amount of nodes in the cluster. You need to have atleast 3 nodes in
your cluster to make a highly available cluster. A size between 5 and 9 is
recommended if you’re running a cluster with high uptime requirements.

Now that we’ve got parameters covered, let’s run the command on atleast 3
servers and you should have a functional etcd cluster up and running. You can
verify by running curl http://localhost:2379/version on one of the machines.

Next step is to setup nginx!

Nginx

We’re not gonna do any custom configuration on nginx, so a simple
# apt-get install nginx is sufficient if you’re running a Debian-based OS.

With nginx running (Verify by running curl http://localhost/), let’s move
on to the next step, which is installing and configuration confd!

Confd

So, finally at the step which does all of the magic!

First things first, we need to install confd. Head over to
Github and get the latest
release (At the time of writing, latest is v0.12.0-alpha3), put it on your
nginx machines and unzip. Move the confd binary into /usr/bin/confd.

Next step is to create a configuration file for confd, a template and
optionally an init startup script.

/etc/confd/conf.d/nginx.toml

This is the confd nginx configuration file, it tells confd where to find the
template file, where to place the result, which command to run on change and
what keys to watch.

dest = Name of the file where the output of the template should be
placed.

owner = File owner of the dest file.

mode = File mode of the dest file.

keys = Etcd keys to watch for change. You can watch /, but to ignore
keys you’re not interested in, you should specify which keys you’re
interested in. You don’t need to specify the full keys (That would defeat
the point of this post!), but the static part in the beginning of the key, in
this case /service.

reload_cmd = Command to run after the template has run.

Put the above content in /etc/confd/conf.d/nginx.toml, then continue with
the next file.

/etc/confd/templates/nginx.conf.tmpl

Ah, the template file!

I’m not gonna explain the content of this file, it’s a mix between Go’s
text/template markup and nginx’s configuration file.

Copy and paste the above content into /etc/confd/templates/nginx.conf.tmpl.

/etc/init/confd.conf

This step is optional and requires Upstart (Present on Ubuntu). Feel free to
adapt the script to other init systems. The main part is the last line, which
starts the confd daemon. Replace the etcd-address with the address to one
of the machines running etcd.

If you wish to this as a daemon, without Upstart, simply run
/usr/bin/confd -backend etcd -watch -node http://etcd-address:2379/ &.

Replace $service_name$ with the name of the service you wish to load balance,
replace $hostname$ with hostname or machine id of the instance running the
service, replace $public_ip$ with the IP address on which the nginx
machine(s) can reach the service and lastly replace $port$ with the port on
which the service is listening for incoming HTTP traffic.

Do note the -d ttl=60 parameter, this tells etcd that it should delete the
value in 60 seconds, so you need to continuously execute the curl command to
keep the value in etcd. By doing this, you allow etcd to clean up services that
is no longer available. Tweak the number to suit your use cases. My
recommendation is to have a ttl of 60 and updating every 45 seconds. This
allows for some downtime if a service crashes, but it shouldn’t affect things
that much, but as said, adjust to your use case.

When you stop your service, you need to delete the key (And stop the updating
script/routine!), this is done by running
curl http://any-etcd-node:2379/v2/keys/services/$service_name$/servers/$hostname$ -X DELETE
with the same values as the create script above. This tells etcd and confd that
the service is no longer avilable and should be removed ASAP. My recommendation
is to run this before you stop the service, as to hinder nginx from sending
requests to a service which no longer exists.

Last words

This is it! I hope you enjoyed the post and that you’ll find it useful to setup
your own highly available web applications.

Confd is by no means restricted to just configuring nginx, it can configure
pretty much anything, as long as the configuration is file based and there’s a
reload/restart command available.

I like using nginx since I know it quite well and it’s fast, mature and highly
reliable.

Got any questions? Ping me on Twitter or
send me a message on IRC where I go by the
name Kim^J.

How to setup continuous deployment of a Hugo website hosted on Github to AWS S3
by using Travis CI as the build/deployment service.

I finally did it. I setup something that builds my website and pushes it to AWS
S3.

To be able to follow along, you need a Github
account, an AWS account and you need to register on
Travis-CI.

You should also have the Travis CI CLI installed to be able to encrypt values
in the .travis.yml file.

I’m assuming that you have prior knowledge with AWS, Hugo and Git.

Creating S3 bucket

So, first you need an S3 bucket where you can host your content. Go ahead and
create one and give it a unique name. My preference is to use the same name
for the bucket as you would use for the domain which will point to the bucket.

Example would be naming the bucket example.com if your website URL is
http://example.com.

So, with that done, onto the next task, making the bucket available to the world.

Set bucket policy

So, in the S3 console, navigate to your bucket, click on Properties, expand
Permissions and click on Edit bucket policy.

Click Save and that’s it! Your bucket is now available to the world. Onto the
next task, which is allowing Travis CI to push data to your bucket.

Create IAM user

Head over to the IAM Console. In the left menu, click on Users, then click
on Create New Users. Enter a username of your choice, I recommend travis-ci.
Then click Create and then Download Credentials. Remember where you save
the downloaded file, since you’ll need the values in that file later on.

Next up, allowing the newly created user access to the previously created S3
bucket.

Create bucket policy

Head over to Policies, click Create Policy, click Select next to
Create Your Own Policy. Give the policy a name, like example.com-access or
something nicer. Then write a description if you like. Then, copy the JSON below
and paste it into the Policy Document textbox and replace example.com with
the name of your bucket. Click Create Policy and you’re done!

When running a playbook with sudo: yes, it also runs the facts module
with sudo, so ansible_user_dir will have the value of the root user, rather
than the expected home directory of the ansible_ssh_user.

So, today I where pushing some generated SSH keys to servers to be able to pull
changes from git without having to add each servers respective SSH key to the
git server. Since I’m using Ansible to automate
configuration of servers, I setup the task like this:

Nothing wrong with the above code, I verified with ansible -m setup host that
ansible_user_dir had the correct value. But, when you add sudo: yes to the
playbook, like this:

- hosts: host
sudo: yes
roles:
- { role: deploy_git_keys }

Then Ansible runs the facts module with sudo, which
makes the ansible_user_dir contain the home directory of the user sudo was
ran as, which by default is root. So, it had the value of /root rather
than my expected /home/ubuntu. So the keys where pushed to /root/.ssh/
rather than my expected /home/ubuntu/.ssh/.

So, the solution is to not have sudo: yes in your playbook, but have them on
the various tasks that really need it or to not rely on ansible_user_dir, but
I prefer the former.