All posts for the month December, 2011

I come from a very Spring-heavy background in Java. Spring is a bit of a loose, baggy monster, containing probably hundreds of features which aren’t all terribly well-connected to one another, but one thing that became second nature to me in Spring was the dependency-injection and IoC features. Structuring software into composable components makes sense to me for scalability and ease of testing.

There are a few logical bits in random-album-cover that are clear candidates for components. In particular, the three distinct bits of information we’re getting from the web should probably each have a testing implementation and a live one (one which actually connects to the web and scrapes a page). My Java instincts would probably structure the code something like the following:

Not shown: some Spring configuration that would wire up one of the two quote factory implementations into a CoverFactoryImpl instance.

None of this is rocket science. Translating it to idiomatic Scala is a bit trickier than it seemed at first. There are a fewlibraries which exist to do dependency injection in Scala, but there is also a method which relies on native library features, the Cake Pattern. Adam Warski’s post from 2010 is a good summary of the idea, with code examples.

This had the salutary effect of driving me back to my primary sources (the Odersky et al “staircase book” and to a lesser extent, the O’Reilly book) for a refresher on self type annotations. I’m still not entirely sure I grasp what the ultimate purpose of self types are; I can see the pragmatic effect of requiring that a concrete class which mixes in one trait must also mix in the trait that is referred to as a self type, but I don’t exactly see how to generalize this to more than a one-to-one trait relationship. In some of the larger Spring projects I have worked on, literally hundreds of services (Java singleton instances) were composed and wired together — in the end there was an object graph. If traits can only declare one self type, though, I don’t quite see how you would use this mechanism for anything other than an object list. Clearly I’m missing something, but the staircase book is uncharacteristically terse on this point.

In any event, all was not lost, because chapter 29 of the staircase book essentially sets out to solve the exact problem I’d been working on: how best to structure and compose component-based software in Scala. There seem to be a few ways of going about it, but for now I should have some simple examples to use as the basis for assembling my tiny services. (It also opens up the possibility of comparing some of the more lightweight Scala DI frameworks down the road, which is something I’d probably enjoy.)

So having come up with a decent if not outstanding scaffolding for random-album-cover, I’m now faced with actually making it do what it’s supposed to. (Actually, there are several additional items on the immediate TODO list, including figuring out unit tests.) This basically boils down to making three HTTP calls per new album cover, all of which are slightly different.

The call to get a quotation is a straightforward HTTP request; we’ll retrieve the response body, scrape it for the third-from-last quote, and store the quote (I’m thinking the entire thing, even though we’ll only be using the last several words for the album title).

The call to flickr should probably use their REST APIs; it looks as though flickr.interestingness.getList() is probably what we want here. One added wrinkle is that this will introduce the first deployment artifact into the application (to wit, my flickr API key – at least I think it will).

The call to get a random Wikipedia page will result in a 302 redirect to a new page. I imagine that just parsing the resulting target URL for the random page name will suffice for now, rather than loading the new page up and scraping it for the title.

None of this is terribly complex. So how do we do it in Scala? The Scala books I’ve got don’t really cover any network programming in detail; looking around the web, it seems like there are basically two well-supported approaches: using Java’s Apache HttpClient, or using the native, idiomatic Scala library Dispatch. (Caveat: a few other libraries exist, but as far as I can tell they are generally either small personal projects or else bits of larger frameworks.)

My first impulse was to use Dispatch, but I ran into a bit of a wall in figuring out exactly how to bend the API to my will. Dispatch definitely seems like it’s pretty neat, and there is some documentation about how it works, but there seem to be woefully few examples of how to, say, send a request and check the HTTP response code. It may be that my knowledge of the underlying implicit stuff that drives request / executor / handler generation is just hampered by my sub-par understanding of Scala implicts in general, but I’ve basically been unable to find documentation that is more comprehensive than this page plus this page, and as a newcomer the emoticon-esque operators give me some pause.

That’s not to say that I’m incapable of just messing around until I figure it out, of course, and one luxury I have in this project is that I don’t have a tight deadline for a deliverable. Right now I’m sort of wavering on the edge of either spending some time tinkering with Dispatch in order to get it working, or else starting from a very imperative-style “java + HttpClient + syntax sugar” first draft, and then coming back to swap out Dispatch for HttpClient and taking the whole project in a more functional direction from there. I’ve had similar thoughts about potentially moving the MVC stack from Scalatra to Unfiltered down the road some once it’s stable.

I find that having a hands-on project is very helpful for me when I’m trying to learn to things in programming, and I’ve needed a fairly simple project to help me learn Scala for some time now. Some of my earliest ideas for toy projects were a little too ambitious.

I was trying at one time to write a small Android app in Scala, but Android itself is a fairly complex new system that I was also learning simultaneously, and having poked around at it for a bit I’ve decided that it will probably be best to learn Android in its native Java before spilling chocolatey Scala into its rich peanut butter APIs (in addition to that, I don’t relish the prospect of adding a ProGuard step to my compilation cycle).

My second idea was to produce a framework for solving some classic sets of computer sciencey problems, to wit the 99 Prolog Problems and possibly (though my math skills are sadly lacking) the Project Euler problems. This is still something I’m interested in doing (for my own reference, the Python wiki also has some promising problem sets). I quickly ran into some trouble with this, as well. It might be possible to come up with a set of specs / ScalaTest / etc tests which could then just be run as a suite, with the user filling in the implementations for particular problems, but I came to realize that what I really wanted was a sort of unit-test like framework that would detect test implementations and automatically run them over given sets of verified input and output data.

This is a worthwhile goal, and I still would like to do it (FWIW, I’ve also dabbled in writing such a framework for clojure, though it’s been collecting a bit of dust recently). The problem is that it’s a fairly advanced problem; doing it the way I envision it involves a level of metaprogramming and reflection that isn’t really suited for a first project. Compounding that, in Scala figuring out the proper type definitions of the individual tests might well be an interesting part of finding solutions to the exercises; if I define it myself in unit test definitions the problem sets may well lose a bit of their didactic panache.

While I was ruminating about these matters I chanced upon a sort of low-level internet meme which had never quite broken through to the big leagues: the random album cover algorithm (here’s a partial history). Basically, you combine a random wikipedia page, a random quotation, and a random flickr image to produce a random album cover. The original flickr pool seems to be dead now, but examplesabound.

As a meme, this frankly does not appear to have the legs of, say, Nyan Cat or Rick Astley. As a project for learning Scala, though, it’s well-nigh perfect:

It can be structured as a classic three-tiered web application.

It involves using publicly available HTTP APIs, plus some HTML scraping and file downloading.

The project is simple and small enough that it should be easy to quickly try out different back-end stores as well as different web frameworks and toolkits; this will hopefully also get me to learn the Cake Pattern and other DI equivalents.

The application itself is easy to scale up in complexity, with the first version just getting the three data items, the second combining them to generate a boilerplate cover, and then adding UI and image processing to allow the user to style the result somewhat.

The image processing bit could involve Scala / Java interoperability (assuming there are more Java image-processing libraries than Scala ones, which frankly I have no idea about).

Eventually the thing could get ported and deployed to a GAE-like cloud environment.

It could potentially be ported over to an Android app as well.

So. Here is the github repository. What I have got so far is a very basic Scalatra web application which keeps an in-memory album cover list. It doesn’t currently actually do any HTTP calls right now, about which more later, but it’s a decent starting point, and I feel that I’m well on my way to actually doing something somewhat fun in Scala.

I’m starting this blog mostly as a place to write about technical issues, largely so I can take some notes and participate a little more in subjects that I’m interested in. Currently those are mostly programming-related; I’m teaching myself scala for one thing, and that’s what I’ll probably focus on for starters.