I do stuff, and write about it

Being a data scientist (yup, that what we are called nowadays, apparently we are all the rage) and looking for my first bike to buy made me think (a lot) about what ride wanted.

Hundreds of pages on various forums talking about which bike is good, which one is better, reliability, position, torque, sound, etc etc.
Yet in the end, once you have set your mind on one or two models, you still have dozens (if not hundreds) of classifieds for these models, and it is sometime hard to know whether the price is fair, a good deal, or an overpriced deal.

So, having lots of time on my hands at the moment, I started taking a quantitative approach.

Step 1 : Get all the data (or as much as I can).

Step 2 : Analysis, model

Step 3 : Profit (now that I have a model, I can know whether any ad is a good deal or not).

Now that we have a server running our database, we want to make it also able to run our rails app.
As I said in the previous post, I could use the full application cookbook, but I’d have to create my
own cookbook and recipe to use that properly. For now, I’ll just rely on rackbox (and its dependency appbox).

Managing with appbox

The appbox cookbook does pretty much the stuff that we configured earlier (setting up users etc). Since it’s a dependency of rackbox,
we are pretty much forced to use it (rackbox does call appbox default recipe).

Having added appbox to our Berksfile, I had to modify the roles/base.rb :

Let app box know that my admin user name is wam.

Generate a ssh key for to login as the deploy user.

Make sure the app box recipe is run before I run my sudo and users::sysadmins recipe.

We’ll start by creating the deploy key :

1

$ ssh-keygen -t rsa -C "wam@scube" -f ~/.ssh/scube_deploy_rsa

roles/base.rb

123456789101112131415161718

name"base"description"Base role applied to all nodes."run_list("recipe[apt]","recipe[build-essential]","recipe[git]","appbox","recipe[users::sysadmins]",# Necessary to run after appbox to add our stuff"recipe[sudo]"# Same)override_attributes("appbox"=>{"deploy_keys"=>["ssh-rsa [...]"# content of ~/.ssh/scube_deploy_rsa.pub],"admin_user"=>"wam",})

Adding a rack server

I’ve chosen to use passenger + nginx, which is a popular choice among the rails community. I was tempted for a moment by puma on jruby, but I want
my app online faster and will bother changing this kind of thing later (chef makes it easy to test new nodes with new recipes..)

Let’s create a roles/app_server.rb :

roles/app_server.rb

123456789101112131415161718

name"app_server"description"Serving http requests for the app. Main app server"run_list("rackbox")override_attributes("rackbox"=>{"ruby"=>{"versions"=>["1.9.3-p385"],"global"=>"1.9.3-p385"},"apps"=>{"passenger"=>[{"appname"=>"scube","hostname"=>"my.hostname.com"}]}})

Now to make it work with vagrant, two changes are necessary in our Vagrantfile :

We need to make sure that the chef version we are using is 11 or more. By default my vagrant was using chef 10, and the rackbox
(more specifically the runit it uses) was throwing an error (NameError: Cannot find a resource for load_new_resource_state on ubuntu version 12.04).

We add a port mapping to access the http port of our server on localhost:8888

The :shell provision makes sure that vagrant updates the chef gem before actually running our chef-solo provision.
Now it’s on to vagrant provision, stuff should appear in green and http://localhost:8888 should show a 404 error.
Yes of course, we haven’t deployed our app yet. That’ll be next.

Now that we have our vagrant box working with chef, let’s use chef to configure our services and our app.

Chef roles

Assuming that you have read some basics about chef, you’ll know that the cookbooks we have downloaded provide recipes for installing various software.
We could ask vagrant to install a few recipes, but it’s probably better to assemble them in roles.
We’ll then assign the roles to one or several nodes, or use all of them on our box for testing.

For now, we probably want to have one base role (to install common software on all our nodes) and two roles to serve our application :

database\_master : a simple install of postgres should be fine here.

app\_server : this one will serve our RoR app.

One could think of other roles (workers, redis etc), but for my purpose and for now these two (and the base role) should be fine.

Base role

We want our base role to include the following :

apt, git, sudo and build-essential should be installed. We’ll use the default cookbooks/recipes for each of these.

users setup : Should create the users (with their ssh key), give them sudo rights. We’ll use the users cookbook.

We start by editing our Berksfile to make sure all the cookbooks are included (sudo, apt, git, build-essential, users).
Then let’s create a role file.

roles/base.rb

123456789

name"base"description"Base role applied to all nodes."run_list("recipe[apt]","recipe[build-essential]","recipe[git]","recipe[users::sysadmins]","recipe[sudo]")

The order matters here !apt should appear first (it’s used to handle packages), build-essential is used by pretty much
everything, and especially by ruby-shadow which is a gem dependency of users.

Reading the documentation of the users cookbook, we see that we should define the users in a data bag
(a way of telling chef about some data, list, including potentially encrypted password and ssh keys).

Chef solo doesn’t work very well with data bags (or the CLI doesn’t work very well), so we’ll just create the file manually.
Also, we see in the users cookbook that it requires chef-solo-search to run with chef-solo.

Adding cookbook 'chef-solo-search', git: "https://github.com/edelight/chef-solo-search.git" to our Berksfile should be good enough.

Now we need to modify our Vagrantfile to use this role (and not the dummy git recipe we were using). An extra bit of precaution is needed here :
the sudo cookbook/recipe will install sudo qnd configure it by default for the sysadmin group (lucky us, our user is a member).
It will override vagrant’s sudo config, breaking vagrant provision using chef-solo. To avoid that, we use vagrant’s chef.json config
to override the sudo configuration attributes for vagrant :

Then it’s on to vagrant provision, and ssh to whatever port was forwarded to 22 (for me it was ssh localhost -p 2222) to see that you log in
using your ssh key.

If you hit a json parsing exception when chef reads your user json file, make sure you don’t have trailing commas.
You can check your JSON easily in irb using require 'json'; JSON.parse(File.read('data_bags/users/wam.json')).

Creating a custom cookbook …

There’s a big choice to do here. We could either create a whole separate cookbook just for our app, configured with many
default recipes, or for now just use an already created one.

It is very likely that I’ll have to create a cookbook at some point, because it’s the only way to have your own recipes
and reach a high enough level of customization.

.. or use and existing one

I originally had a look at the database cookbook but finally decided
to go the fast way by using two very neat cookbooks, rackbox and databox.
It will probably make sense to use database and application cookbooks, but they seem to be easier to work with when you are using a proper chef server
and your own cookbook/recipes.

rackbox includes appbox by default, which creates its own users for deployment/app running.
I have found that these cookbooks are a bit limited for my taste (for example, they don’t use data_bags, which are a proper way of encrypting
password instead of storing them in your chef repository… Well, next time.

Setting up our roles

Let’s start by adding the cookbooks to our Berksfile and run berks install

and create our roles/database_master.rb. We are using non encrypted passwords here, which isn’t very secure.
We should actually use encrypted data bags, but they don’t play very nicely with roles (they are supposed to be used with recipes, which
would mean custom cookbook), nor do they play nicely with knife solo (although a plugin exist, but it didn’t work very well
in my tests). Let’s start this way, we’ll see later to move to a more robust non solo chef.

roles/database_master.rb

123456789101112131415161718

name"database_master"description"Master postgresql node"run_list("databox::postgresql"# Or "databox" to include mysql as well)default_attributes(:databox=>{:db_root_password=>"PASSWORD_HERE",:postgresql=>[{"database_name"=>"myapp_production","username"=>"myapp","password"=>"ANOTHER_PASSWORD_HERE"}]})

Now running vagrant provision (or vagrant up or vagrant reload depending on whether your current vagrant box is up or not) should run this recipe, adding
the myapp database. We can test that in vagrant ssh

What’s next

Next post will be about configuring a proper rails box using rackbox, setting up capistrano to deploy … then deploy to a vps and get closer to production.
I’m still not entirely happy with this deployment today. I should move to a proper cookbook, as I said, to get more customization options.
For now, I want my app out, and will probably work a bit more later depending on how successful it is. The beauty of chef, after all, is that it makes
it easy to set up new nodes and new deployments.

Mocking inherited_resources helpers in views specs.

So, my tests where failing because I was using the resource, collection and resource_class helpers from some views I was using. So first my tests are failing because resource_class isn’t available in my views. I would have thought that the controller helpers were available in the views, but they aren’t.
The solution is easy. Let’s add the following to our spec/support directory :

spec/support/view_resource_macros.rb

12345678910111213141516171819202122232425

moduleViewResourceMacrosdefhas_resource(name,&block)beforedo# Creates the resource@resource||=yield# Assign to the symbol we wanted, so it's available in the viewassign(name,@resource)# Assigns to @name so that we can use that in our assertionsinstance_variable_set("@#{name}",@resource)# If we pass an array, it's for stubing a collection, if not it's for stubbing a single objectif@resource.is_a?(Array)view.stub(:collection){@resource}view.stub(:resource_class){@resource.first.class}elseview.stub(:resource){@resource}view.stub(:resource_class){@resource.class}endendendendRSpec.configuredo|config|config.extendViewResourceMacros,:type=>:viewend

Then I just created a base/new.html.haml and a base/edit.html.haml views, to use the views inheritance.

base/new.html.haml

12

%h1 Create #{resource_class.model_name.human}=render:partial=>"form"

base/edit.html.haml

12

%h1 Edit #{resource}=render:partial=>"form"

And I have two _form.html.haml partials, one for each controller.
Now the next issue is that our edit and new views are shared, but we still want to test the _form.html.haml partial.

spec/views/cars/_form.html.haml_spec.rb

1234567891011121314151617

require'spec_helperdescribe"cars/_form.html.haml"do{new:->{Car.new}edit:->{Factory.create(:car)}}.eachdo|name,block|context"when called in ##{name}"dohas_resource(name,block)it"renders the form"dorenderrendered.shouldhave_selector("form")endendendend

Shared partial

Finally, when testing for example cars/index.html.haml which uses a partial toolbar.html.haml that actually exists in base views, the following lines are necessary :

123

beforedoviews.lookup_context.prefixes<<"base"end

This was raised as an issue to the rails team, but they commented (rightly I think) that the inheritance
is related to the controller, not the views, so the test case shouldn’t know about it and you’ll have to declare it manually using the lines above.

So, let’s be honest, I’ve been quite lazy here. My latest rails application was deployed with Heroku. Lots of fun, a very pleasant experience.
But now I have this new shinny application that I don’t want to deploy using Heroku :

Because I am worried that if I start with Heroku, I’ll be too lazy to switch later.

Because I know that at some point, my application is going to need plugins and binaries that I can’t get on Heroku.

Because I’d rather have a portable application that I can deploy easily on any type of server.

Just a quick post, having finally figured out how to install dep_selector on mac os x.
The issue is that having xcode installed which configures clang as the default compiler, some native gems break.

It took me some time, but at long least I can now build a native gem using gcc instead of clang on mac os x.
And you’ll see it’s not very easy to have ruby change its compiler for gems native extensions compilation.

I haven’t posted anything for a while now, and after hours of trying to find a solution to my problem, I thought I should share. So here we go.

The problem

Your nice and shiny iOS app is supposed to have two data components : User data and Seed data.
For example, you want to have some (seeded) list of postcodes. The size of data is too big to be shipped with your app,
and we assume that the model is too complex to be just filled by your application at runtime from a downloaded csv/txt file.

So, you start thinking that hey, you’ll generate a sqlite database (persistent data store as they say), put it on a server and have your app download and use it.
You can either duplicate the whole stack (NSManagedObjectContext, NSPersistentStoreCoordinator and NSManagedObjectModel) or, according to apple :

You typically use configurations if you want to store different entities in
different stores. A persistent store coordinator can only have one managed
object model, so by default each store associated with a given coordinator
must contain the same entities. To work around this restriction, you can
create a model that contains the union of all the entities you want to use.
You then create configurations in the model for each of the subsets of
entities that you want to use. You can then use this model when you create a
coordinator. When you add stores, you specify the different store attributes
by configuration. When you are creating your configurations, though, remember
that you cannot create cross-store relationships.

Well, that’s pretty much all the doc you’ll get from apple.
There are a few mentions of this problem thereor there
but not in a clear enough form for me. So, here’s how it works…

So it all started when I asked a friend to test this new app I’m building :
“On your main page, garantee. You should really check your spelling, people will know you are French !”.

So here I went, naively googling for “website spell checker”, with either pricey reports or simple websites
where I’d have to copy-paste my text from all my pages. Then, I found AfterTheDeadline.
It looks great, open-source, recently bought by wordpress.com, and they even provide an API.

So, spent my evening doing stuff, and here we are with a new gem : Merimee (github).