All posts in category Scala

I’m still thinking about the dependency-injection and Akka stuff and will have at least one more longish post on the subject, but I’ve recently become distracted by refactoring some of my old JavaScript code into ClojureScript and figuring out Clojure’s new core.async library (ultimately I hope to form all of these distractions into a ring instead of a straight line, at which time I will be the acme of productivity, but that’s another story). But before I get too deep into Clojure land I wanted to plug a few Scala-related things I’ve come across recently.

Firstly, Derek Wyatt’s book Akka Concurrency: Building reliable software in a multi-core world is excellent, covering a great many Akka topics in an enjoyable style. In particular, his chapter on testing seems profoundly relevant to the mock injection topics I’ve been thinking about, but I haven’t quite absorbed it yet. The whole thing is refreshingly up to date, too (which makes sense since it was only published a month or two ago), and it doesn’t pad out its length with a lot of “learn Scala in 30 days” remedial material. Anyways, I recommend it thoroughly.

Secondly, John Sullivan’s long post about the cake pattern is by far the best treatment I’ve seen of it online, and is required reading for anyone interested in dependency injection in Scala. (It’s been up for several months now but somehow I missed it until recently.) John has written a dependency-injection framework, congeal, which makes instant intuitive sense for me as someone coming from a Java / Spring background; unfortunately it depends on some macro stuff which won’t make it into Scala’s mainline, so it isn’t ready for prime time and will need to be rewritten down the road once Scala’s macros reach their next stable state. There’s a video from ScalaDays 2013 describing the framework.

And finally, following up on the subject of the cake pattern, Daniel Spiewak’s keynote from NEScala, “The Bakery from the Black Lagoon,” is an excellent talk which made me think about the cake pattern in a new way (as more of a compiler-enforced module system than as a form of dependency injection). His implementation of the cake pattern is also interestingly different from most examples I’ve seen online – in particular, he mostly eschews self-types, with the exception of needing one for a virtual class.

Just as a quick follow-up to my previous post, I thought I’d note that the official Akka blog has published a post regarding Akka and dependency injection (kind of weirdly expressed as a mini white paper, as though the internet at large had issued an RFP for an actor-based concurrency system with dependency injection).

While, as I mentioned before, I should emphasize that I’m by no means an expert in Akka or actor best practices, I’m not convinced that this document addresses my particular concerns with the intersection of Akka and dependency injection. I’m still thinking about a larger post breaking down the approaches to this topic that I’ve seen online, but this one sort of falls into the “mock stuff outside of Akka” camp.

The document has two main points: firstly, if you have an existing dependency-injected service, you can pass along a factory which knows where to find it to the Props constructor of an actor, and there’s a way to attach a DI application context to an ActorSystem to support this, in what seems like a pretty convenient way. Secondly, if you need to expose an interface from actors to an existing system based on a DI framework, you can include an ActorSystem singleton in your DI object graph, and then expose a sort of regular-object facade over it which finds specific actors and returns either ActorRefs or futures resulting from sending ask messages to them.

That’s all well and good, but it seems more like a way to integrate between Akka and an existing synchronous DI-based system than anything that makes dependency injection useful or usable inside a purely Akka-based system. (In particular, the document’s unfortunate final section seems to be aimed squarely at recalcitrant middle managers who need to be convinced that a move to Akka will not result in a whole bunch of now-legacy code needing to be tossed out.) While I’m not incredibly interested in this topic myself, I thought Akka already had a talking point for this integration problem in the form of “typed actors“.

The bit that I still haven’t seen addressed is that if Akka likes actors to explicitly manage the lifecycles of other actors they supervise, there doesn’t seem to be any room for the inversion of control that is the hallmark of dependency injection frameworks in the first place. To put it in more concrete terms, if I’m running a partial integration test of my simple notification service and I want it to have a real database actor and mock REST web service actors, how can I tell the actor that supervises all the HTTP worker actors to create mock actors instead of live ones?

I have a half-formed idea of how this could work that involves having a sort of service locator / factory actor which is responsible for actor instantiation, but the idea in my head doesn’t particularly jibe with Akka’s supervision hierarchy, which as far as I can tell is coupled very tightly to actor instantiation.

I’ve been digging into what the expected way is to test my small Akka system, as described in my previous post on the subject. I think my problem partially arises from being unclear as to the proper mode of dependency injection in Akka. In other words, I don’t know what the proper way is for my Root actor to obtain a reference to its Database and HTTP sub-actors. Does it create them itself? Look them up from a service locator? And what if I need to inject mock actors into the system in some parts in order to test it?

Various bits of Akka documentation suggest different approaches to wiring actors together; for instance, this page in the official docs suggests either passing dependent actors as constructor arguments, creating them in a preStart() method, or passing them to their referring actors in a message and using become() to switch between initialization states. This example from the testkit docs takes the latter approach, but I can’t say I like the result:

This does seem to work, but it seems to me that it pollutes the actor with a bunch of test-related code that probably doesn’t belong in production (by which I mean the receive pattern which takes the two dest parameters).

I have found an interesting take on this question in this presentation by Roland Kuhn, introducing akka-testkit from Scala Days 2012—the entire presentation is worth watching, but the part I’m interested in starts at around 22:05 or so. After a not terribly helpful note about how if you have difficulty injecting mocks into your code, then there is probably something wrong with your design (there may be something to that, but it’s not all that helpful to hear when you’re looking for a solution for injection), Mr. Kuhn mentions a third option for users of the (then-new) Akka 2.0: actors can use actor path naming to look up their dependent actors; the test ActorSystem can then supplant the real implementations with mocks at the same locations.

Of course, all of this sort of assumes that you have a way of separating out actor creation and lifecycle control from dependency injection itself. A lot of the other Akka literature I’ve read seems to posit the integrated lifecycle management bits of Akka as a feature, right down to the “Let It Crash” maxim on the public Akka blog, and all of these features seem to be in direct opposition to the inversion of control notions that most dependency injection systems are founded on. In the last part of Mr. Kuhn’s talk above, he suggests breaking up actor models into somewhat discrete trees, which then use service locators or similar things to find one another; this might be something I can look into.

There was also a talk at this year’s Scala Days about integrating Spring and Akka, which might have some merit for this purpose, and I recently ran across this promising post which describes an approach to autowiring actors with Spring and Akka 2.2 (in Java). Overall, though, this doesn’t seem to be a problem with a clear solution.

My current technical interest, mercurial as ever, is in Akka. My present employer is mostly a java shop, but they are open-minded and I have a notion to prototype out a rewrite of a simple system there into Akka and Scala. The system is probably one of the simpler ones we have, known as the “notification service.” It periodically checks for new rows in a particular database table. If it finds any, it fires off a JSON-formatted request to a REST web service, the “delivery service”; if it gets a successful response from this service it will mark the message as delivered in the database. There are a few wrinkles related to locking, and there are actually a few different web services involved, but that’s pretty much the basics. Something possessed me to make a diagram of the existing flow:

The purpose of this system is to deliver notifications to particular users, with the idea being that any subsystem which needs to send a notification to someone can put the right data in the database, where this system will pick up up and hand it off to an existing REST service which winds up doing most of the heavy lifting. The existing service is implemented in Java and Spring, using Quartz as a cron job to kick off a polling method once every 30 minutes or so (we don’t need this service to run particularly swiftly).

It’s not really hard to see how this would translate into a message-based actor model in Scala. You’d probably have one root actor coordinating things. You’d have an actor talking to the database, maybe with a supervisor to restart it as needed, and you’d have another actor to handle the HTTP client calls. Most likely the client actor would spawn off a new actor per individual row of data, and have each of these worker actors make a single HTTP request. On a success, the worker would send a message back to the database actor to update the database row as “completed”; on a failure the worker might just log an error and die.

A rough sketch of that might look like this (pardon my sub-par OmniGraffle skills):

Note that the single-line arrows here represent the actor supervision hierarchy, not message-passing. I’m also not positive that the “DB Worker” actor needs to exist, versus just having the “Database” actor do the work, but it simplifies things to do it this way and I suppose there might be more than one of those (more on this later).

I’m been struggling a bit to come up with a good way to represent message passing in a diagram, but I think I’ve got the gist of the design in this one:

Everything is started by Tick messages which are sent to the root object every 5 seconds via Akka’s scheduler interface (this would be more like 15-30 minutes in production). This causes the root actor to pass a PollDatabase message to the Database actor; the message includes a reference to the HTTP Client actor. For each notification row the Database actor finds in the database, it sends a Notify message to the HttpClient actor. This actor composes a MakeRequest message to one of a pool of Worker actors, including the data from the database and a reference to the Database actor. The Worker performs the HTTP request; if it is successful it sends a RequestSucceeded message to the Database actor, which will ask a DB Worker thread to update the database to mark the relevant row as successfully delivered. If the Worker gets an error, it sends a RequestFailed message to the HTTP Client actor, which at this point will just log the error and continue on.

I will have more to say about this, but this post is already decently long, so maybe I’ll leave this here so I can refer back to it later. There are a few things I’m struggling with:

Despite having read a lot of articles and blog posts on the subject, it’s not obvious to me what the correct way to instantiate and connect these actors is (constructor arguments, preStart() methods, dependency injection, etc).

Related to the above, it’s not clear to me how to test this system without mixing up test code and business logic. In particular I’d like to replace the nodes in yellow above with mock objects and verify that the system still works properly.

I would like to have a reasonable interface to Oracle, without needing to include Spring or something in the project. The Typesafe, Inc solution is to use Slick, but I don’t have a burning desire to sell my co-workers on closed-source, commercial software in addition to a new language and framework.

And now for something completely different. I got a little distracted again by an idea for a simple social networking mashup I’ve had and decided that the best place to deploy it would probably on Google App Engine. App Engine supports Scala and I’ve seen one or two positive reports about using Scalatra on it, so I thought it wouldn’t be too hard to spin something up quickly using my existing code.

It turned out not to be as easy as that, unfortunately; I may try to elaborate on the reasons in a later post, but it seems to come down to Scalate always checking the filesystem for its template routines. In any event, I found that I was spending a lot of time trying to glue weird bits of infrastructure to one another rather than working on my idea.

With that in mind, I decided to start over in Python, and of course as my first task I spent a long time glueing random bits of infrastructure together. In this case I’ve been successful, though, so here’s a brief writeup on getting Jade templates to work with Google App Engine and Google’s webapp2 in python.

I’ve been using a layout similar to the one mentioned in this Stack Overflow answer. It’s puzzling that Google doesn’t really mention anything about a standard way to get libraries and stuff from the wider Python ecosystem installed in a development environment, but essentially I set up a virtualenv, ran pip install pyjade, and then symlinked the pyjade directory from the virtualenv lib directory into the project’s src directory.

3. Add a custom Jinja2 factory.

This is where the magic happens. We write a factory function, jinja2_factory, whose purpose is to add pyjade’s included Jinja2 extension into the Jinja2 instance’s Environment.

Caveat

I’m by no means an expert in Google App Engine, and I’m a little worried that there’s something badly inefficient or similarly wrong about this approach. In particular, I’m hoping the returned templates are cached, but I’m not certain, and I’ve seen reports that Jinja2 Environment creation is an expensive operation on GAE. More to the point, pyjade seems to work as a preprocessor for Jinja2, so it would obviously be better not to need to run it for every request. In any event, this does at least find jade templates and render them properly as HTML.

(I’m running out of snappy cake-related puns here, but I’m saving “the cake is a lie” for a special occasion.)

So after thinking about my interface / implementation dilemma for a while (recap: I’m aiming to separate out the implementations of hexmap from their interface), I decided to approach it from a Java vantage point and see how I’d implement the same thing in Java. The immediately obvious thing that jumped out at me was that the specific problem I was dealing with was sort of a classic application of plain old vanilla inheritance from an abstract class. Having drunk deeply from the well of Spring over my last several years of Java experience, plain old inheritance had begun to seem in some ways like a quaint relic of a bygone era (having been replaced in many cases by service composition).

At any rate, a quick bit of refactoring to get rid of traits altogether got me back to a fairly sane codebase, where I’ve got this:

I will come back and revisit this later; I’m certain there’s a way traits could be useful in this code, but for now I’m leaving them aside and pressing on with more actual program functionality.

As far as that is concerned, my next worry is persisting the hexmap the user sees in their Scalatra session, so that when the user asks for a random path from the server, he or she gets one within the actual bounds of the actual map he or she is seeing. The Scalatra session docs make this look fairly straightforward; we’ll see.

I’ve made some progress on both the client and server sides of the hexmap project. On the client side, there’s now some code to highlight tiles (though it’s still imperfect). I’ve also updated the map representation some, so it looks like this:

The idea is that the data object will expand to have more game or application-specific data. I’m not 100% sure that moving to a sparse representation (versus a 2d matrix) is a good idea, but I don’t think it will hurt me all that much, and it shouldn’t be too difficult to change back if I need to.

On the Scala side of the fence, I’m using the lift-json library to perform serialization to JSON. I’m still not sure I’ve structured this in a way that really makes sense in the long-term. One thing I’m hoping to achieve with this project is to wind up with a fairly abstract, generic hexmap library which can be augmented by application-specific code to implement a particular game or the like. So I’m trying to partition off some of the specific stuff, such as in this case tile colors, into its own conceptual space, but I’m not sure I’m doing it idiomatically.

In particular, I think I’m hitting a slight impedance mismatch between Java and Scala revolving around traits, self-type declarations, and the Cake Pattern versus interfaces and Spring-style dependency injection. I’d like to separate out the interface of a hexmap from the implementation used to actually implement the hexmap, so what I have is this:

My idea for this was that there might be one implementation which uses a MongoDB store, one which uses an in-memory map, etc. The fundamental operation, though, is to pass in a grid address and retrieve the data for that cell. The actual implementation I’ve got now is this:

I think where I’m hitting problems most is trying to figure out how exactly the actual instantiation of “a HexMap with a RandomHexMapStore mixed in” should take place, or in other words where the definition of the specific component assembly I’m creating should appear. In Spring this would be an object graph defined either in a (probably big, crazy, and unwieldy) XML file or via annotations on various class declarations.

What I’ve been trying for starters is just to define a factory method which returns a new random map:

This is working OK for now, and it makes sense to me that different implementations of HexMapStore might have different constructor parameters, so I’m not too bothered by factory methods in general. I’m a little worried by how this approach might or might not scale to much larger class / object graphs, though.

Moreover, so far I’ve been glossing over the most serious immediate problem I have with this: the implementation of the RandomHexMapStore.createRandomMap() method. This method is meant to construct a map composed of randomly selected color elements. The problem is that, being a trait which is mixed in to HexMap, is has no access to HexMap’s height and width class parameters. What I have at the moment is this:

I’m a little baffled by how to get access to class parameters from a trait which is mixed in to that class. When I think about it, I can’t really see a good reason for Scala to allow it, since, after all, RandomHexMapStore could be mixed into any class, not just a descendant of HexMap. This suggests to me that I am probably not using the proper language mechanism for what I’m trying to do. I will have to do some thinking on this, and possibly call on Stack Overflow for help.

Bolting Scalatra on to the existing hexmap project turned out to be fairly easy, though I had the advantage of having an existing minimal project already (the nascent random-album-cover project). My idea for this project is to have nearly all the server transactions handled through Ajax rather than form posts and server-side templating, but while I was getting the directory structure up and running I took the opportunity to translate the existing HTML that I developed for the hexmap into Jade.

My initial impulse, actually, was to use Mustache, and avoid rewriting the HTML altogether. However, I found the layout documentation for scalate Mustache to be basically incomprehensible, and if I’m understanding it correctly, it requires you to insert extra stuff into your layout file and still not remove anything from your content file, which doesn’t exactly seem like it’s heading in the right direction as far as templating system design. Besides, using Jade lets me pretend that I live in a universe where SGML-derived syntaxes never became something that almost every working programmer would have to deal with almost every day.

The downside of this is that I’m no longer able to monkey with the client-side code without a running jetty instance, but I think that should be OK for the near future. The upside is that the code for index.html has shrank down to a svelte 36 lines.

At any rate, I’ve got the client code retrieving and painting a map from the server now. My server-side map database currently looks like this:

It’s a start. I’ve also pulled in the Lift-JSON libraries, which look to be useful. The next step is to implement a few different map stores, hook them up to the client, and test the output. After that, I need to figure out a way to populate actually interesting map data in a persistent store (presumably Mongo). The idea which keeps nagging at the back of my mind is that the existing code could probably be used as the start of a dynamic map editor, but I’m not positive I want to head down that route.

After spending a while uselessly flailing around trying to get the Gimp to do what I wanted it to do, I wised up and checked out some directions for tile creation on the Wesnoth site, which pointed me in the direction of the open-source SVG editor Inkscape. Inkscape, among other nice features, has a polygon tool which is able to make regular hexagons and the like. It will also export selected objects as PNG files, so with minimal effort I’ve been able to replace the tiles I swiped from Wesnoth with new, boring ones of my own devising. Among other improvements, I’ve got lines around the hexes now, and the hexagons themselves are regular. Encouragingly, I didn’t need to change the javascript at all to deal with the new sizes, since it works by inspecting the img sizes from their attributes directly and basing its math on that.

Along those same lines, I’ve been pretty pleased with how easy it is to separate presentation logic from presentation markup in jQuery. Apart from switching image sizes, I can see that it would be fairly easy to set up the function which generates random tiles, randomTile(), to dynamically query the DOM tree, find all of the possible tiles, and then pick one at random (currently it uses a static list of possible tile names). Together with attaching arbitrary custom data to each possible tile, this seems like a promising approach to some sort of game, with a classic game engine / game content separation. However, Barbarian Prince is not that game, being based on a static game map, and part of the point of this exercise has been to learn Scala, so it’s back to the server side after this.

After finding Inkscape, I spent a while messing around with different sizes for the hexes in the hexmap. For a while I was considering using either SVG or HTML5 canvas elements or some such to represent the map, with the idea that it would allow a more easy way to zoom in and out from the map. I feel like both of those areas represent rather large and interesting rabbit holes, though, and I’ve only just emerged from the jQuery rabbit hole I was in. Moreover, having distinct tiles seems like a good match for the game as it was written.

The next steps are to head back out to terra incognita and add this stuff to the project:

A more beefed-up definition of the hexmap definition (as a JSON, uh, schema).

A persistent store on the Scala side which is capable of storing and retrieving this definition (most likely via mongodb, since I’m familiar with it and it has a well-regarded Scala interface).

Simple REST service definitions on the Scala side, and, oh yeah, a web server framework to serve them up, most likely with Scalatra.

An Ajax call from index.html to get the persisted hexmap from the Scala server and display it.

I will note in passing that it has occurred to me that Barbarian Prince as she is spoke could probably be easily implemented purely through jQuery and HTML5 client-side storage (which I guess seems to mean indexedDB as of February 2012), keeping all game logic and map creation on the client side. This would certainly reduce network latency for players, as well as the need for highly-skilled developers on server side. This is an approach that somebody who isn’t trying to get experience in programming Scala should probably pursue.

After a good bit of head-scratching I’ve managed to make some progress on what I assumed was going to be one of the more difficult aspects of the hex-map project: actually displaying the hexes in a browser. This also turned out to be a good opportunity to finally dig into jQuery a little more deeply, too. I’ve come up with a rough idea of how to proceed from here, too; I’ll have the page make Ajax calls out to Scala to retrieve map details in JSON and it should be pretty easy to dynamically construct the hex grids from there.

Github, probably wisely, doesn’t display checked-in HTML directly in the browser, but I’ll attach a screenshot here; interested parties can check out the code on github under src/main/webapp. Currently the page generates a new random map on every page refresh, with each map being a mixture of roughly 2/3 desert tiles and 1/3 ocean tiles. The data structure is a simple 2-dimensional array:

Obviously for a full game implementation the elements would be complex JSON structures, not strings, but this suffices for now. To tile the hexes together, I’m simply laying them on top of one another via CSS absolute positioning; each image is a rectangle (actually, a square) with the diagonal edges cut out of the sides. I was a little worried this would make it cumbersome to attach jQuery event handlers to the right places, but I currently have a handler that displays the (x,y) coordinates of a hex when a user mouses over one and it seems very usable, if not perfect around the edges of hexes.

The code I’m using to position the tiles looks roughly like this (note: some cleanup is still pending in the actual code; what follows is the gist but see the code itself for details). Here tile is a string from the previous array, and x and y are row / column numbers.

This isn’t perfect, but it’s a good place to start. Some problems I’ve got with it:

The hexagon pngs are ones I’ve borrowed from the Battle for Wesnoth project, and are a bit irregular (literally – the top edges are shorter than those on the sides). This does seem to make the math easy, though. I should note here that the PNGs are published under the GPLv2, though to be honest, I have no clue how the GPLv2 could possibly apply to images.

I’m a little worried that overlaying further PNGs on top of these ones (for map highlighting, roads, player markers, etc) will interfere with the mouseover events, but I’ll need to experiment.

I had a lot of trouble trying to get little borders to appear between hexes by monkeying with the positioning values above.

Edit: having recently discovered jsfiddle, I uploaded a fiddle of this there. I’ve also manage to clean up the code so that it is currently in much better shape on jsfiddle than it is in github, but the github version will be changing and it will be nice to keep a clean version of the basic idea around.