The Aim

Longer Term

As this series progresses we are aiming to create a running “development level quality” environment with the click of a button – or less! – representing some kind of interpretation of the fabled, and often elusive Infrastructure As Code dreamscape.

For example we should, from a standing start, be able to create an environment consisting of:-

1 VM hosting a 1 Kafka Node Cluster (if that is not an oxymoron).

A 3-5 node Kubernetes cluster where the services are deployed

1 VM hosting a DB of sorts … maybe Cassandra or MongoDB .. we’ll see

Shorter Term – i.e. This Article

Okay, so lets say we’re in the land of amorphous micro-services. That means we’re going to be using some kind of messaging broker, yeah … and for us this will be Apache Kafka.

Today we’re going to create a virtual machine which has Apache Kafka installed and configured, including the creation of our desired topics for inter service communication.

Tools and Demo Environment

The required toys which you’ll need to install on your local machine are :-

Note: Please install the latest packages above so they play nicely. For example, versions of Vagrant and VirtualBox are usually behind if using say a Linux Distro’s repository. Certainly true for me, your mileage may vary.

Accompanying Code

Lets Go!

Vagrant Ascend

Working with Vagrant initially, as opposed to jumping straight to creating images tailored for your favourite cloud provider, ensures our VM works locally in the first instance. This will, or at least should, tease out most of the config style issues you’re likely to hit.

For more specific details on the Vagrantfile config choices see the documentation but basically we’re saying:-

Pull down the basic vm image “ubuntu/trusty64” from HashiCorp’s Atlas repository – assuming of course you don’t have a similarly named locally installed box which would then take precedence.

Forward traffic from your local machine to the VM when it is spun up on 9092 – Kafka’s default listening port. This isn’t strictly needed in this article but if you wanted local prcoesses (on your host machine) to interact on the usual kafka ports then it would be.

Use virtual box to provision the VM, and give it 2GB memory. Any less and Kafka may fail to start up.

At this stage we can build a vanilla vm box. If you are in the directory with Vagrantfile do …

Okay so that’s great. Lets get Kafka installed on this.
Firstly log off your VM and then destroy it.

vagrant destroy -f

I prefer to rebuild from scratch each time rather than vagrant’s provision to update any changes as it is more true to what you will build at the packer stage, next in the series.

Enter Ansible

Obviously this article is short in theory – if you want a solid introduction into Ansible I’d recommend Michael Heap’s book Ansible, From Beginner to Pro.

We want to install Zookeeper and Kafka onto our VM automatically when we do vagrant up and as such there is a plethora of configuration tools to choose from; Chef, Puppet and SaltStack are a few good options. They are all fine and dandy, but in our case we will be using Ansible.

Point Vagrant to Ansible for Provisioning

Add ansible provisioning lines to your Vagrant file so that it looks like …

“hosts: all” – just means apply the following playbook to all the hosts Ansible knows about. In our case it will just be our VirtualBox vm, but you could specify hostnames, groupnames etc for Ansible to apply to.

“roles” – the roles to apply to our vm. We will be covering these next.

Now if you run vagrant up at this point, unless you have installed the zookeeper role, it should fail with a message like …

ERROR! the role 'AnsibleShipyard.ansible-zookeeper' was not found in /home/andy/blogs/myKafkaVm/provisioning/roles:/etc/ansible/roles:/home/andy/blogs/myKafkaVm/provisioning

… which obviously refers to the zookeeper role in our playbook. In the above configuration we are saying to use a locally installed Ansible role called AnsibleShipyard.ansible-zookeeper. There isn’t one, so lets install it.

We could of course write our own Ansible roles to do precisely what we want, but if a problem has already been solved then let us not waste the world’s time eh!? We are going to install one from Ansible Galaxy which is a collection of community roles, somewhat like Ruby gems, Python’s PyPI. Anyway, the “role” that we are interested in using here is ansible-zookeeper contributed by AnsibleShipyard.

sudo ansible-galaxy install AnsibleShipyard.ansible-zookeeper

This basically stores the role under /etc/ansible/roles/

and now vagrant up should work again.

And now we’re getting somewhere. You can ssh onto the box and check zookeeper is installed at /opt/zookeeper-{version}. Or ps -ef | grep java and it should be running.

Install Kafka using a downloaded Ansible Role

To install Kafka on the vm, we are going to use another role, but not one from Ansible Galaxy. In this case we are going to download it from git and use it more directly.

Add the “ansible-kafka” role to the playbook.yml so that it looks like the following …