Category: programming

I’ve been creating a fun microservice tool that provides a single API frontend and merges data from multiple backends. Since the app itself relies entirely on external data, I was wondering how in the world I would write unit tests for it. It’s written in python using the amazing apistar framework. All of my external data so far is gathered using the requests library. The answer for this, turns out to requests-mock. Requests-mock will allow you create mock responses to requests.

The documentation is pretty straightforward, but I was having some trouble wrapping my head around how I would use it to test the code in my app. To start simple, I decided to mock consul, which is one of my datasources.

Get a value into consul

First off, let’s go ahead and setup. Go to https://www.consul.io/ and download the consul binary for your OS.

Start consul in dev mode/foreground: consul agent -dev

Insert a key: consul kv put foo bar

Let’s request that key with curl. Be verbose, because there are some headers you’ll want later.

This is a problem. My code is still perfectly fine, but because the live data has changed, my test fails. That is what we hope to solve.

Let’s mock consul.

As I alluded at the top of my post, I hope to solve this with requests-mock. There’s some fancy things I see like registering URIs, but to start, I am just going use the Mocker example they have. It’s a good thing I did a curl request earlier to see what the actual response will be.

This is great. I can develop locally, store working examples in my test code, and test against that.

Requests is fun, but what about python-consul?

The truth is, I don’t talk to consul using requests and base 64 decoding. For some reason, I thought it would
be easier for you to follow along if I did straight requests. But in reality, most people are going to use python-consul. In fact, here is my getkey.py file doing just that.

Oh what disaster! My fancy getkey code is failing tests!. What is x-consul-index anyways? Well, it looks
to be response.headers['X-Consul-Index'], which is a header we saw in the curl request. Fortunately,
mock allows you to provide headers as well.

What is the point?

There isn’t much pointing to having a block of code produce a static value and then check to see if it is that value. However, when we start taking actions based on values (live, maintenance, true, false, -1), we can definitely check to see if our code behaves an expected way based on a collection of sample data we store. I can also check for how I handle incomplete data. A big part of my microservice correlates devices with network interfaces, ip addresses, and vlans. Not every interface has an ip, not every ip has a vlan. Not every network has a default gateway. I have to determine which ip is “primary”. So as I collect examples of devices with different configurations, I should be able to register urls and responses for each device. If my code is expecting a vlan to be a number, but instead I receive a “None” – will I handle that or will I throw an exception error?

Looking forward, I can envision having sample json data stored with functions to provide the desired response and headers needed.

hstore is an option key=>value column type that’s been around in postgresql for a long time. I was looking at it for a project where I want to compare “new data” to old, so I can approve it. There is a hstore-hstore option that compares two hstore collections and shows the differences.

In reality, an hstore column looks like text. It’s just in a format that postgresql understands.

This information could be displayed on a confirmation page. Ideally, a proposed dataset would be placed somewhere, and a page could be rendered on the fly showing any changes an approval would create within the database.

Build the box:

Run codebox in docker

This will run the codebox.io environment on port 8000 and you’ll be editing files that are stored in a directory
called workspace1. ~/workspace1 on the host gets mounted into /workspace1 in the container.

Now, I can access my server on port 8000 (ie, http://192.168.1.32:8000/). This instance is unprotected, it just asks
for an email to get started. But the part I skipped over is that I am actually using nginx to proxy and password
protect this instance.

I can hit Ctrl+C on the terminal to cancel my instance and all my edited files are safely stored in ~/workspace1.

Interesting bits:
I create a new file in the web browser and save it, I see this bit of json in the console output:

I got this same issue going through my nginx proxy and connecting directly on port 8000. A bit of text flashes
quickly on the “terminal” in the web browser, but closes too quickly for me to catch it. Perhaps it’s attempting to
run something that is not installed in the Docker instance. If I can fix that, then I just need to come up with a cool
way to launch workspaces and tie them into my nginx setup (or switch over to hipache
as my front-end webserver).

Anyways, just wanted to record these steps here and show how easy it could be to get your own self hosted IDE.

This weekend I decided to takle both learning Ruby and working with AWS
via the Ruby API. Having only played with both of these in the past,
this presents two learning challenges at once. However, from past
projects, this is how I learn best. I am somewhat familiar with AWS
terms and once made a script in Python to fire up an instance. This was
before Amazon came out with their management console, so I imagine
things have come a long way since then (hopefully easier). I also played
with Ruby for a while, but didn’t have a decent project for it. Having a
project with goals will hopefully keep me on track and give me a way to
measure my progress.

My goals for this project are as follows:

Utilize a web based interface. Using rails seems to be the popular
way to do this, and I’d like to base my template interface off ofboilerstrap5, a combination of twitter-bootstrap and
html5boilerplate. This will probably have the most trial and error
to get it right.

Connect to the AWS api and pull some basic information such as my
account name.

Fetch details about an AMI image. Maybe I’ll be able to parse a list
of public images, or maybe I can just punch in an image ID and pull
up the details.

Start an instance from an AMI image. This might require some steps
like setting up a an S3 bucket — we’ll see.

List my running instances.

Control a running instance – ie, power cycle it.

Destroy an instance.

BONUS: Do something similar with S3 buckets – create, list, destroy.

First off, I need to setup a ruby development environment. Since I have
used PyCharm in the past, I will try JetBrain’s RubyMine for my
editor environment. After installing this, the first thing I learned is
that rails is not installed. I could install using apt-get, but
Jetbrains recommends using RVM. It looks like a nice way to manage
different versions of Ruby, rails, and gems. I know when I have
installed Ruby applications requiring gems, gem versions was always a
source of concern. It is very easy to get mismatched gem versions in the
wild.

RVM install locally to ~/.rvm on linux, which is nice – you don’t mess
up any system wide ruby installations and keep everything local to your
development environment. After installation, I had to figure out a
couple bits with rvm.

rvm install 1.9.2 # installs ruby 1.9.2

rvm list # lists versions of ruby installed

rvm use 1.8.7 # use ruby 1.8.7

First, your terminal has to be setup as a login shell. This tripped me
up for a while until I changed the settings in my terminal emulator.terminator has this as checkmark option.

Finally, once you get ruby and rails working, you can create your rails
project. I’m starting with a rails project because it’s “all the rage”
and gives you a decent running start. Later, I’ll work on switching the
supplied templates with boilerplate + bootstrap based ones.

This gets me started. Next up, I’ll actually create the project from
within RubyMine and just work on basic web functionality.