Cloud-init NoCloud with URL for meta-data and user-data

Posted on Thursday, April 14, 2016

The basic idea was
install cloud-init on a non-cloud install of Ubuntu. Then to create a meta-data and user-data file
at /var/lib/cloud/seed/nocloud-net then reboot. The user-data file is the cloud-config file
and the meta-data file is some presetting that I got to work partially.

Why would I do all of
this? I am used to using the AWS cloud
or OpenStack, both of which support cloud-init. I am planning on building an OpenStack box
at home… someday… but I keep getting delayed with life.

In the meantime I do
have an ESXi box built out and running at home.
It works great. But boy do I wish
I had a simple nice interface or command line tool that would allow me to tell
it to build a box of X size and run this cloud-init file.

Running cloud-init
locally I can kind of get that working well enough.

But I think I can one up
it. From what I have been reading I
think I can set cloud-init to grab the meta-data and user-data files from a
URL. If I can get that working I can
put all my meta-data and user-data files in a git repo, for tracking, then put
the repo files on an internal nginx server I can link to. Then I just have to edit a cloud-init config
file to point in the right direction and I am good to go.

First a simple example (No URL)

First I want to show how
to get a local simple cloud-init working on a fresh Ubuntu 14.04 install.

Install cloud-init

>
sudo apt-get install cloud-init

Create the /var/lib/cloud/seed/nocloud-net/ folder

> sudo mkdir -p /var/lib/cloud/seed/nocloud-net

Make a very simple meta-data file

>
sudo vi /var/lib/cloud/seed/nocloud-net/meta-data

local-hostname: my-servername

(This will change the
servers name to my-servername)

Use write_files to create a new /etc/network/interfaces
file.

>
sudo vi /var/lib/cloud/seed/nocloud-net/user-data

#cloud-config

#

####################

##################################################

#

# Install packages

#

##################################################

packages:

- htop

- tree

##################################################

#

# Reboot the machine when done.

# This is done to make sure the mount script is

# working

#

##################################################

power_state:

delay: "now"

mode: reboot

message: First Reboot!

timeout: 30

##################################################

#

# Create groups

#

##################################################

groups:

- patman

##################################################

#

# Create users.

# * set their prefered
shell

# * Set their group
membership

# * give them admin permissions

# * Set their password
(copied from ecnrypted version @ /etc/shadow)

# * Input Public SSH
keys to allow access

#

##################################################

users:

- default

- name: patman

shell: /bin/bash

sudo: [ "ALL=(ALL)
ALL" ]

primary-group: patman

passwd: $6$C5Hpbc-NOTREALKEY

ssh-authorized-keys:

- ssh-rsa AAAAB3NzaC-NOTREALKEY

##################################################

#

# runcmd:

# List of commands to
execute

# If a command has a
verbose setting, it's a

# good idea to use
it. Just in case you

# Need to debug later.

# Output is logged to

#
/var/log/cloud-init-output.log

#

##################################################

runcmd:

##################################################

#

# passwd <username> -u

# When a user is created
and their password

# set... their password is locked in /etc/shadow

# (it has a '!' in front of their encrypted

# password, hard to see and a pain)

# This command unlocks it

#

##################################################

- 'passwd patman -u'

##################################################

#

# This fixes the hostname in /etc/hosts

# which removes annoying messages when you

# are trying to run commands using sudo

#

##################################################

- 'echo 127.0.0.1
`hostname` >> /etc/hosts'

##################################################

#

# write_files:

# Files to write out.

#

##################################################

write_files:

##################################################

#

# Set static IP address (would rather do this in meta-data but
it is no

# working at the moment)

# @
/etc/network/interfaces

#

##################################################

- path:
/etc/network/interfaces

permissions: '0644'

owner: root:root

content: |

auto lo

iface lo inet
loopback

#

#

auto eth0

iface eth0 inet
static

address
192.168.0.75

netmask
255.255.255.0

broadcast
192.168.1.255

gateway
192.168.0.1

dns-nameservers 8.8.8.8

What will this cloud-init file do? Let me go piece by piece

packages

##################################################

#

# Install packages

#

##################################################

packages:

- htop

- tree

This section tells it to install htop and tree

power_state (reboot)

##################################################

#

# Reboot the machine when done.

# This is done to make sure the mount script is

# working

#

##################################################

power_state:

delay: "now"

mode: reboot

message: First Reboot!

timeout: 30

This section tells it to reboot after cloud-init is done.

groups

##################################################

#

# Create groups

#

##################################################

groups:

- patman

This section is creates groups. I am creating a group for my user that has
the same name as my user.

users

##################################################

#

# Create users.

# * set their prefered
shell

# * Set their group
membership

# * give them admin permissions

# * Set their password
(copied from ecnrypted version @ /etc/shadow)

# * Input Public SSH
keys to allow access

#

##################################################

users:

- default

- name: patman

shell: /bin/bash

sudo: [ "ALL=(ALL)
ALL" ]

primary-group: patman

passwd: $6$C5Hpbc-NOTREALKEY

ssh-authorized-keys:

- ssh-rsa AAAAB3NzaC-NOTREALKEY

In this section you can add your user and get them set up
the way you want. I've set up the
default shell, my primary group, my password (as taken from /etc/shadow), and
my ssh public keys.

runcmd

##################################################

#

# runcmd:

# List of commands to
execute

# If a command has a
verbose setting, it's a

# good idea to use
it. Just in case you

# Need to debug later.

# Output is logged to

#
/var/log/cloud-init-output.log

#

##################################################

runcmd:

##################################################

#

# passwd <username> -u

# When a user is created
and their password

# set... their password is locked in /etc/shadow

# (it has a '!' in front of their encrypted

# password, hard to see and a pain)

# This command unlocks it

#

##################################################

- 'passwd patman -u'

##################################################

#

# This fixes the hostname in /etc/hosts

# which removes annoying messages when you

# are trying to run commands using sudo

#

##################################################

- 'echo 127.0.0.1 `hostname`
>> /etc/hosts'

Runcmd basically defines a bash script to run. This is useful to do some setup that
otherwise can't be done in other cloud-init modules.

This particular runcmd would result in this bash file being
created and run.

#!/bin/sh

passwd patman -u

echo 127.0.0.1 `hostname` >> /etc/hosts

These command unlock the user patman (who is created in a
locked state). It also updates the
/etc/hosts file to add the new hostname to it so you don't get that annoying
warning when running sudo commands in Ubuntu.

write_files

##################################################

#

# write_files:

# Files to write out.

#

##################################################

write_files:

##################################################

#

# Set static IP address (would rather do this in meta-data but
it is no

# working at the moment)

# @
/etc/network/interfaces

#

##################################################

- path:
/etc/network/interfaces

permissions: '0644'

owner: root:root

content: |

auto lo

iface lo inet
loopback

#

#

auto eth0

iface eth0 inet
static

address
192.168.0.75

netmask
255.255.255.0

broadcast
192.168.1.255

gateway
192.168.0.1

dns-nameservers 8.8.8.8

This section creates a new /etc/network/interfaces
file. In my case it will set a static IP
address to 192.168.0.75 on a reboot.

Then reboot it

> sudo reboot now

Now, I should also be able to login as patman at the static
IP address.

> ssh patman@192.168.0.75

And tree and htop should be installed.

OK that is all working… Now to figure out how to move it to
a URL and get cloud-init to use that URL.

Create a URL

You can come up with your own idea on where to store stuff
but here is my method.

I happen to have an nginx server running on one of my local
machines. I am going to make a directory
there and set It up to listen on port 8888 (which in my case is not available outside my
network).

> sudo vi
/etc/nginx/nginx.conf

I added this section to it.

server {

listen 8888;

location / {

root
/nginx/cloud-init/;

}

}

Let me make the folder

> sudo mkdir
/nginx/cloud-init

> sudo chown
www-data:www-data /nginx/cloud-init

And restart nginx

> sudo service nginx
restart

Within /nginx/cloud-init
I think I will eventually create some folder structure that makes sense for me
and my servers. But for now I will just
make a test folder there.

> cd
/nginx/cloud-init

> sudo mkdir test

Meta-data file/url

Create the meta-data
file

> sudo vi
test/meta-data

And place the following in it.

local-hostname: my-servername

User-data file/url

Create the user-data
file

> sudo vi
test/user-data

And place the following in it.

#cloud-config

#

####################

##################################################

#

# Install packages

#

##################################################

packages:

- htop

- tree

##################################################

#

# Reboot the machine when done.

# This is done to make sure the mount script is

# working

#

##################################################

power_state:

delay: "now"

mode: reboot

message: First Reboot!

timeout: 30

##################################################

#

# Create groups

#

##################################################

groups:

- patman

##################################################

#

# Create users.

# * set their prefered
shell

# * Set their group
membership

# * give them admin
permissions

# * Set their password
(copied from ecnrypted version @ /etc/shadow)

# * Input Public SSH
keys to allow access

#

##################################################

users:

- default

- name: patman

shell: /bin/bash

sudo: [ "ALL=(ALL)
ALL" ]

primary-group: patman

passwd: $6$C5Hpbc-NOTREALKEY

ssh-authorized-keys:

- ssh-rsa AAAAB3NzaC-NOTREALKEY

##################################################

#

# runcmd:

# List of commands to
execute

# If a command has a
verbose setting, it's a

# good idea to use
it. Just in case you

# Need to debug later.

# Output is logged to

#
/var/log/cloud-init-output.log

#

##################################################

runcmd:

##################################################

#

# passwd <username> -u

# When a user is created
and their password

# set... their password is locked in /etc/shadow

# (it has a '!' in front of their encrypted

# password, hard to see and a pain)

# This command unlocks it

#

##################################################

- 'passwd patman -u'

##################################################

#

# This fixes the hostname in /etc/hosts

# which removes annoying messages when you

# are trying to run commands using sudo

#

##################################################

- 'echo 127.0.0.1
`hostname` >> /etc/hosts'

##################################################

#

# write_files:

# Files to write out.

#

##################################################

write_files:

##################################################

#

# Set static IP address (would rather do this in meta-data but
it is no

# working at the moment)

# @ /etc/network/interfaces

#

##################################################

- path:
/etc/network/interfaces

permissions: '0644'

owner: root:root

content: |

auto lo

iface lo inet
loopback

#

#

auto eth0

iface eth0 inet
static

address 192.168.0.75

netmask
255.255.255.0

broadcast
192.168.1.255

gateway
192.168.0.1

dns-nameservers 8.8.8.8

Now, in my case I have a url for meta-data at

http://192.168.0.80:8888/test/meta-data

and a url for user-data at

http://192.168.0.80:8888/test/user-data

I think it should be ready now.

First Attempt (Failed miserably)

Now back to a very fresh install of Ubuntu 14.04.

First install cloud-init.

>
sudo apt-get install cloud-init

Now test and make sure you can reach the urls from this
server using wget. (use your own URLS
but here is my example)