How to set up your own PerlMongers web service in 10 minutes or less

I've been working with ElasticSearch over the past months as part of the MetaCPAN project. Using ElasticSearch as our back end has worked out really well so far. The reason is that, out of the box, it provides a REST API. So, in our case, we've been able to concentrate on writing code and not on designing an API, defining its behaviour, arguing over URL schemes etc.

To be clear, ES is not written in Perl, but there is a handy Perl module you can use to get yourself up and running in *minutes*.

As my example of how to run our own web service, I've chosen to create a service which hosts info on all of the PerlMongers groups in the comprehensive XML file found here. We're going to grab it, parse it and stuff it into ElasticSearch. Once you have the logic of that part down (and there's not a lot to it), you're basically done.

There's nothing fancy going on over here. First, I've created an index called perlmongers. Then I've just grabbed what I thought were the interesting parts of the XML data and stuffed them into a hash. I've used this hash to populate a type called group. The id I've used for these groups is the internal id already listed in the XML file. I've made the (hopefully safe) assumption that these ids will not change in future.

To see what is now in the index, run a query on your local ES server via your web browser:

http://localhost:9200/perlmongers/group/_search?q=*&size=25

This will return the first 25 entries. You'll see that they look something like this:

The ES server returns your data in JSON. Nice! Now, if you want to see just one result, you can search on a name:

http://localhost:9200/perlmongers/group/_search?q=name:toronto.pm

You can also use an id if you happen to know it:

http://localhost:9200/perlmongers/group/103

Maybe you want to search on state:

http://localhost:9200/perlmongers/group/_search?q=state:new york

Anyway, I think you can see where I'm going with this. How much work did you really have to do here? You had to munge a bit of data, but that's basically it. Along with that, you get all the goodness of a rich API which supports some basic queries (which I've shown you here) along with much more complex queries which you can dig into in the ES docs.

If you want to see ES in action, check out some of the sample URLs which are posted on the MetaCPAN wiki. If you have any questions, feel free to join us on #metacpan at irc.freenode.net. If you feel like contributing some code, please hit us up on IRC as well.

Tagged as:

4 Comments

One question: why do you encode to UTF8? That should be handled by ElasticSearch.pm. Did you see any issues with encoding? I tried deleting that block and rerunning the script, and the encoding seems fine.