Saturday, January 26, 2013

Building a Raspberry Pi Cluster

Once upon a time...

So this is how this story began...

The Raspberry Pi has fascinated us geeks from the time it was first announced. We were thrilled to hear that Eben will be presenting at WSO2Con, and even more thrilled when Sanjiva suggested that we should host the WSO2Con Conference App backend on a Raspberry Pi cluster. Say what? Sanjiva always comes up with crazy ideas. The initial briefing was on the 23rd of December, 2012, one day before the team went off for the end of year holiday break. At that time, we were not even sure whether it would be possible to run enterprise grade middleware on the Raspberry Pi. But exploring the unexplored is the norm at WSO2, and achieving the impossible is something we do on a regular basis. So a bunch of daredevil WSO2 geeks took up the challenge of building this system. The project officially got off the ground on the 2nd of January this year.

Running Java & WSO2 middleware on Pi?? Are you nuts!

One of the alpha geeks at WSO2, Lakmal, had managed to get Java running on the Raspberry Pi & had even managed to get WSO2 AppServer running. However, a major worry was that it took up to 6 minutes just to start the AppServer on the Pi. This was unacceptable to us & we decided that we had to be able to optimize our middleware to be able to run on resource constrained environments. The Raspberry Pi model B has only 512MB of RAM & a 700 MHz ARM processor. Two Carbon geeks, Sameera & Pradeep took up the challenge of optimizing the Carbon kernel to run on such constrained environments, and were able to introduce the concept of optimized mode to the Carbon kernel in the process of doing so. They were able to get the start up time down to close to 1 minute!

A nasty surprise!

One of the major surprises came in the form of the memory configuration of the Raspberry Pi. Even though we had the model B which has 512MB RAM, raspi-config & free only showed 256MB! We thought that we had hit a brick wall! After all, we had just 3 weeks left for the conference & there was no way that we could get another shipment quickly. But the "never say die" WSO2 team searched for answers on the web, and found that there was a lot of misinformation. Finally we figured out that a full apt-get update followed by an apt-get upgrade & raspi-config update solves the issue. Now all 512MB was available, and using the raspi-config memory split functionality, we reduced the memory allocated to the GPU to 16MB because the GPU was going to be of little use for our application.

I have the power!

Another major challenge we faced was supplying power to the Raspberry Pi cluster. Individual power supplies was going to be too costly & impractical. So we decided to build our own power supply. We built USB power distribution boards which were powered by regulated 5V power supplies. The Raspberry Pi model B draws between 700-1200mA & the voltage between the two test points on the board, TP1 & TP2 has to be maintained between 4.7-5.2V. Initially we ran into a few hiccups where the voltage dropped well below the 4.7V threshold. After analyzing the circuitry, we discovered that the 1-2 Ohm resistance in the wires was causing this voltage drop. We were able to solve that problem by using thicker low resistance wiring, and build the board so that the voltage drop across the power board was less than 0.5V, which is less than the voltage drop across a standard diode. Electronics geek Sumedha is currently building a couple of regulated power supplies. Here are some photos of the board & power supply in action.

What a mess!

We were planning to have 20+ Raspberry Pis in our cluster in order to run a number of WSO2 products. So, managing the space on the work area, and keeping the wires from getting entangled was another challenge. We had plans to build a rack but Sumedha suggested that we should build a prototype rack, and try out a few configurations before finalizing the rack design. Sumedha who hails from the city of carpenters, Moratuwa, happens to be good at carpentry as well. So over the last weekend, he quickly put together a rack. Shown below is the rack with 2 layers containing 8 Pis per layer & a network switch. Now we have expanded it to 3 layers. See the video at the end of the blog.

We plan to build the final rack using clear plastic sheets.

Pis Pis everywhere!

With the Pis in place, keeping track of them soon became the next obstacle. Finding out the IP addresses to create an ssh session, restarting the Pis & WSO2 software became difficult to manage. Connecting an HDMI cable & keyboard to each & every Pi in the cluster was not going to be a practical option. This is when the idea of a Raspberry Pi Control Center came to our mind. We quickly built a simple webapp & hosted it on StratosLive. Cron jobs calling into Perl & Bash scripts written by Lakmal & Deep, on the Raspberry Pis would periodically publish their information to the control center, and also check whether there are any pending commands from the control center to be executed on the Pis. Some of those commands include blink, reboot, update load balancer IP address & power on/off AppServer cluster. Now, using just a few clicks on this interface, we can control an entire cluster of Raspberry Pis. Locating the Pis on the rack can be a challenge when you have a large enough cluster. So, we used the standard LED blink python script which uses the GPIO library to send signals to the Raspberry Pi GPIO pins to make LEDs connected to the Pis blink. Using this functionality, we were able to locate the Pis, and then we labelled them using the Rack number & Pi number. We are currently working on connecting a switch to the GPIO pins so that when it is pressed, the corresponding entry in the control center application gets highlighted.

Main Control Center Interface

WSO2 Cluster Controls

Load me up baby!

Once the cluster was up & running, we ran a series of JMeter load tests. Preliminary testing was conducted using an 8 Pi cluster. At a concurrency level of 750, the cluster was able to sustain a throughput of 350 TPS at an average latency of close to 1 second. We are working on improving these figures.

Woohoo.... Control Center in action!

The following video shows the team which was trying out the control center with the Raspberry Pi cluster. The background chitchat is in Sinhalese, the main native language of Sri Lanka.

More good stuff to come!

We will be having a demo station at WSO2Con, London, and the giveaways to the conference attendees will include a model B Raspberry Pi, SD card with Raspbian image, and a Raspberry Pi case. Drop in at our demo station and we will let you plug in your Pi into the cluster, and immediately see how your Pi joins the cluster & starts serving requests to the conference app. We will also conduct a session on how we built this setup & application.

So don't miss this golden opportunity and register now for the WSO2 conference & come meet the geeks!