Scala tips, tricks, observations, and learnings

Not just for in Lift web apps, the lift-json library serves as a useful, fast, and sophisticated method of handling JSON in your Scala projects. In general usage, the lift-json library is pretty smart about serializing/deserializing messages, which are based on case classes. However, under certain circumstances the case classes in question may be indistinguishable, and this is where type hints come into play. Let’s take a look…

Consider that we have case classes for a couple of kinds of animals and want to deal with serializing and deserializing lists of these animals:

Marshalled as JSON: [{"name":"Manchester","age":10},{"name":"Fido","age":7}]
net.liftweb.json.MappingException: No information known about type
at net.liftweb.json.Extraction$.instantiate$1(Extraction.scala:252)
at net.liftweb.json.Extraction$.newInstance$1(Extraction.scala:283)
at net.liftweb.json.Extraction$.build$1(Extraction.scala:301)
at net.liftweb.json.Extraction$.extract0(Extraction.scala:352)
at net.liftweb.json.Extraction$.extract(Extraction.scala:42)
at net.liftweb.json.JsonAST$JValue.extract(JsonAST.scala:300)
at net.liftweb.json.Serialization$.read(Serialization.scala:48)
at com.foo.LiftJsonTypeHints$.v1(LiftJsonTypeHints.scala:26)
at .<init>(<console>:6)
at .<clinit>(<console>)
at RequestResult$.<init>(<console>:9)
at RequestResult$.<clinit>(<console>)
at RequestResult$scala_repl_result(<console>)
at sun.reflect.NativeMethodAccessorIm...

Ruh roh.

Ok, we know we have a list of things that are of trait Animal, so let’s make a small change to explicitly read back a List of Animal:

Marshalled as JSON: [{"name":"Manchester","age":10},{"name":"Fido","age":7}]
net.liftweb.json.MappingException: No constructor for type interface com.foo.Animal, JObject(List(JField(name,JString(Manchester)), JField(age,JInt(10))))
at net.liftweb.json.Extraction$$anonfun$findBestConstructor$1$1.apply(Extraction.scala:212)
at net.liftweb.json.Extraction$$anonfun$findBestConstructor$1$1.apply(Extraction.scala:212)
at scala.Option.getOrElse(Option.scala:104)
at net.liftweb.json.Extraction$.findBestConstructor$1(Extraction.scala:212)
at net.liftweb.json.Extraction$.instantiate$1(Extraction.scala:247)
at net.liftweb.json.Extraction$.newInstance$1(Extraction.scala:283)
at net.liftweb.json.Extraction$.build$1(Extraction.scala:301)
at net.liftweb.json.Extraction$$anonfun$17.apply(Extraction.scala:291)
at net.liftweb.json.Extraction$$anonfun$17.apply(Extraction....

Ok, so we can’t just create a generic animal because Animal has no constructor, and lift-json doesn’t know any better. Let’s introduce type hints for our case classes–note the change in our implicit DefaultFormats to add ShortTypeHints.

Slick! The JSON-formatted list now includes “jsonClass” attributes to help clue-in lift-json to the type for unmarshalling back to Animal case classes. As an aside, one could use FullTypeHints instead of ShortTypeHints, which will fill in jsonClass as the class name including fully-qualified package structure. So if our classes are defined in package “com.foo”, we’d see “jsonClass”:”com.foo.Cat”.

Ok, let’s take this a step further. I originally was experimenting with this due to a desire to interact with a .NET-based JSON web service, which utilized type hints but with a different attribute name than jsonClass. Luckily, lift-json allows us to specify a custom type hint field name when we define our own Formats instance:

Ok, cool. But this didn’t quite meet my needs, because the API I’m dealing with also uses type hints that start with lowercase letters. To achieve this I can create my own case class extending the TypeHints trait, and then use that instead of ShortTypeHints:

I’m working on a project where I want to look at the last modified date of a URL for comparative purposes, and decided to come up with a brief method of fetching the headers and parsing it as a java.util.Date.

As with many HTTP-related operations, the go-to project here was the awesome Dispatch project, which is backed by the trusty (and proven) Apache HttpClient but throws on all sorts of Scala-y goodness though a nifty DSL of verbs for executors, requests, and handlers that wrap a number of common HTTP operations.

Without further ado, the “one-liner” (plus a couple of imports and a bit of line wrapping for less scrolly): (Note: See edit at bottom for enhanced version)

If this gives you what you’re looking for, you can stop here, but for educational purposes let’s break it down further.

The Http(...) bit defines a Dispatch executor. In this case, I’m using the singleton executor that blocks the current thread and can be safely shared across threads. Other types of executors can be used via direct constructors that offer options including executing on background threads or using NIO.

Within the executor, we essentially define a request and it’s handler. The request portion is :/("dispatch.databinder.net").HEAD, where /: is a verb used to define the host, one of many ways to specify the URL. The HEAD verb modifies the request so that we’re only requesting the HEAD elements rather than a conventional GET request. Other verbs are available to, among other things, add path elements, append query strings, or POST data to the URL.

After this we have our handler, in this case the most verbose portion of the code: >:> {h => (new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss zzz")).parse(h("Last-Modified").head)}. The first portion of this is the handler verb, >:>, which processes the header data as a Map passed to the function block that follows. Within this block, we hardcode a java.text.SimpleDateFormat that we know can parse out the date format used for “Last-Modified” dates–if this code was going to be run repeatedly we’d obviously define the date format object once and reuse it (see also edit at bottom for version using DateUtils). The h("Last-Modified").head actually pulls the value out of the Map of headers. The .head is required because the values portion of the Map actually contain sets of String to accomodate headers with multiple values, e.g. “Content-Type”. Since we know that we only will get a single “Last-Modified” date, we can confidently only care about the first (and in this case only) element of the Set.

Technically the return of this is a dispatch.Http.HttpPackage[Date], but this can be interacted with directly as if it were a java.util.Date

This could easily be adapted to process other header values at the whim of the coder, and all with much less effort than coding directly against HttpClient.

Edit:
The HttpClient library offers a DateUtils library that is able to parse various date formats that appear in the Last-Modified header. This is preferable to the method used above because (a) it accounts for different localities that format the date string differently and (b) SimpleDateFormat isn’t thread-safe so if you’re reusing it from multiple threads you will need to synchronize on it. Plus it’s less verbose. Given that, our one-liner can be rewritten as:

Upgrading a few existing sbt-based projects from 0.7.7 to 0.10.0 was relatively straightforward, but came with a few hurdles. One of these was the enabling of Scala compiler options such as -unchecked and -deprecation. Previously these could be declared by creating (or editing) a project definition file in project/build, something like the following:

This method no longer works in 0.10.x, where instead compiler options are declared in the build.sbt file of the project root (along with many other properties formerly defined in the project definition) using the scalacoptions property:

scalacOptions ++= Seq("-unchecked", "-deprecation")

This configuration parameter is of course well-documented on the sbt github page, but the change in names hung me up a bit. On the positive side, the new convention allows arbitrary compiler options more succinctly than previously, where non-built-in compiler options required use of the compileOptions(options: String*) method.

Aha! Type erasure! You see, Scala is forced to do type erasure due to the fact that the Java Virtual Machine (JVM) isn’t aware of generics, which underlie typed collections such as the Lists we’re looking at here. As such, any type of List will match the “List[String]” case and we never reach the expected case for Int lists.

However, my immediate needs in this area were limited in scope, only affecting a few lines of code and occurring within a single source file. I essentially wanted to send a List[String] to an Actor and have it handle this appropriately in a match block. I was reluctant to rely on a method that was termed “obscure” and “experimental”, and moreover wanted something lightweight that didn’t add a lot of code. Besides, it doesn’t seem like something like this should have to tap into the reflection libraries.

What I came up with was essentially wrapping List[String] inside of a single-argument case class, which could then be passed around and matched against:

The Scala RichString class provides some nice hooks for string manipulation beyond those provided by java.lang.String, including a capitalize method that returns a Java String with the first letter up-cased. For whatever, reason, an “uncapitalize” equivalent is not defined to downcase the first character. Using implicit conversions, one can relatively straightforwardly add such a method that can be executed against a Java String.

First define the following implicit def, which handles null/empty String values and otherwise downcases the first letter. I put this in a StringConversions object which can be imported for use in various other areas of code.

Determining local IP address is easy, but determining MAC address, and subnet mask in Java has always been one of those hairy problems that sends me Googling. Using some of those same Java utilities, we can add a few Scala-isms and shorten the whole thing up significantly.

To get the IP address we can just statically retrieve an java.net.InetAddress to localhost and get the address String from that .

Wrapping our localhost address object in a java.net.NetworkInterface, we can get the MAC address, or hardware address, formatted as an array of bytes, which we can convert to a Scala List, allowing us to format as hex and concatenate with the “:” separator with relatively little effort. Note that NetworkInterface.getNetworkAddress() requires Java 1.6.

We can use the same java.net.NetworkInterface instance to get us the subnet mask by retrieving the addresses for all interfaces (also a Java 1.6-only method) and for each of these retrieve the network prefix length, which essentially represents the subnet mask. We import scala.collection.JavaConversions._ to give us implicit Java-to-Scala list conversions, which I believe was introduced in Scala 1.8.

Hmmm…that’s not quite right. 24 and 64 aren’t what we expect to see for subnet masks. But worry not, we’re on our way, right now we have the CIDR notation version of the subnet masks, in IPv4 and IPv6 respectively. We then just need to filter out the IPv6 address and convert the CIDR notation to our expected format, which I do using the import org.apache.commons.net.util.SubnetUtils class from Apache Commons Net