Joseph Kain

It’s been a few weeks since I last wrote about Domain Scrapper. I’ve covered fetching data from services like Reddit and Twitter and for a few posts I’ve been promising to start looking at aggregation of the collected data. In this post I want to start on that task.

Ecto

As I’m sure you’ve heard already, Ecto is the database library to use with Elixir. I’m going to set it up within Domain Scrapper.

I’ve used Ecto in a number of Phoenix based projects before and Phoenix graciously sets up Ecto as part of its initial project. But, recently I wrote a simple script to fetch Elixir github repos as part of my Idiomatic Elixir research. This was the first time I had to set up Ecto by hand and it was a learning experience.

In this post I’m going to repeat the process with Domain Scrapper.

Create a New Aggregator Application Under the Umbrella

Domain Scrapper is an umbrella application and aggregation will be
handled in a new application. So first we’ll create the aggregator:

You will need to use a valid username / password for your postgres installation.

Write the Repo Module

Next, we have to create a Repo module which describes the OTP
application in use. I believe this is used to find our configuration
above:

defmoduleAggregator.RepodouseEcto.Repo,otp_app::aggregatorend

We can now run

mix ecto.create -r Aggregator.Repo

from the root of the project. This will will create the database. The “-r” option was necessary to specify which Repo to create. From an umbrella app we could easily end up with multiple repos across a number of applications. Though for Domain Scrapper this is our first.

I verified the database was created using PG Commander to see the newly created database.

Create a Model and a Migration

Next, we need to create a model but I haven’t fully thought through what’s going into the aggregator yet. I know at the very least that we have a URL to save. So, let’s start with that much by creating apps/aggregator/lib/aggregator/domain.ex:

But, Ecto is kind enough to tell us that we have to start the Repo and suggests we do this in our supervision tree. This should be easily done except that when I went to do it I realized that I had not created the aggregator application as supervised using mix new --sup.

Supervise an Elixir Application from Scratch

At first I thought to start over with a supervised application and go through all the steps again. But then I thought this would be a good learning experience to understand how to setup a supervised application by hand.

To figure out what I needed to do I created a dummy supervised app in my umbrella project:

defmoduleFoodouseApplication# See http://elixir-lang.org/docs/stable/elixir/Application.html# for more information on OTP Applicationsdefstart(_type,_args)doimportSupervisor.Spec,warn:falsechildren=[# Define workers and child supervisors to be supervised# worker(Foo.Worker, [arg1, arg2, arg3]),]# See http://elixir-lang.org/docs/stable/elixir/Supervisor.html# for other strategies and supported optionsopts=[strategy::one_for_one,name:Foo.Supervisor]Supervisor.start_link(children,opts)endend

contrast this with aggregator/lib/aggregator.ex:

defmoduleAggregatordoend

So, we need to use Application to pull in the Application boilerplate and then write a start/2 function to start up the aggregator’s supervision tree. Something like this should work:

Our start/2 function defines a single child worker for Aggregator.Repo since this is the module we wanted to start up in the first place. Then it starts up a Supervisor with the same :one_for_one strategy that the mix would generate by default. Given that we have only one child the choice of strategy shouldn’t make much of a difference.

Continuing to follow the dummy app’s lead, we need to add the Aggregator module to the application definition in app/aggregator/mix.exs like this:

Next steps

We have our basic setup for Ecto finished. Next we need to make it a little easier to work with and then start doing real work with the model.

In the next post I hope to handle making things easier to work with. This means making those timestamps automatic, and setting up the right aliases and imports to make working Ecto a little less verbose.

Setting up Ecto in an Elixir Application was published on December 15, 2015.

Joseph Kain is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to amazon.com.