At a Glance

Introduce Yourself

I’m an ‘old dog’ — not only in experience but technique. I’m a JAPH with a love for Agile development.

What’s your role at ShuttleCloud?

I manage our data infrastructure, hosted with Amazon AWS.

How have you designed the platform?

I tightened our security by implementing Amazon VPC. We now have security measures similar to a data center. The setup involves multiple boxes (instances) that can talk to each other but only in certain ways. We have 10-15 different boxes in the architecture that communicate to accomplish tasks, pending the required configuration. Limiting what a box can do on a granular level is a best practice that’s great for security.

Deployment of new boxes is partially automated. While our development team previously used technology like Fabric to quickly deploy a piece of software, we are now moving to Ansible to deploy our services from scratch, including both the configuration of boxes and the deployment of our software.

All developers now use this method as part of the development lifecycle; we consider it a best practice in secure software deployment.

What bottlenecks have you encountered?

We started with Selenium which is a scraping platform for email and contacts, especially contacts. Scraping is simply simulating a web browser and clicking buttons to download and gather data the way a user would. Since Selenium is already a complex piece of software, it was difficult to integrate with our new security requirements. To fix this, we managed to deploy scraping mechanisms within our AWS instances (boxes) and apply the same techniques of internal box communication with Selenium. There is now a cluster of 6 boxes that need to talk to each other with secure connections to prevent penetration or data leakage.

Describe a project you’re proud of.

When we access email services we use POP or IMAP protocols. Recently some of our Ruby scripts failed while accessing the IMAP service of a particular email provider. This script uses the IMAP library provided by Ruby. In debugging we discovered the service was served by two different classes of servers, behind a well-known URL. One class of servers was speaking plain standard IMAP, while the other type had essentially invented a new dialect. This egregious “IMAP+” added single extra ‘space’ characters between parenthesis, which broke the dialog (handled by the Ruby IMAP library) and thus our script, causing it to fail.

Our workaround was to modify the Ruby library in question to relax the system. The fix took a single evening, and we’re really proud of it. We’re sharing this with Ruby’s people soon and hope they commit it as an official change to the library.