We used to wait for Jenkins to produce us a Cobertura report and of course nobody read it until delivery time arrived and we realised we hadn’t met the SLAs. Enter the eCobertura eclipse plugin which provides you with visual code coverage directly from within the eclipse editor. Just run your unit tests and you immediately see the source lines in green and red. Wow! How did we ever do without this?

We came across a weird problem on ESX5i. Occasionally, one of our hosts would suddenly be unable to start any VMs – the running VMs were fine, but any attempt to start new ones would fail with an “Unknown internal error”. The first time this happened, I restarted the management agents and finally suspended all VMs and rebooted the server, after which everything was OK again for a few months and then the problem occurred again.

This time I decided to figure out what was going on. The log files in /var/log contain a lot of useful information and I was able to see that the problem was actually caused by ESX being out of disk space on the device used for /var/log. What was happening was that the driver for the Adaptec 5405z RAID controller in the machine was writing a huge log file which was not being rotated, so after a few months it consumed all the disk space.

The workaround was to add a line to the crontab (note: you also have to add a line to /etc/rc.local to readd the line to the crontab, otherwise it’ll be lost on next reboot) which deletes the adaptec log file periodically:
/etc/rc.local

Preamble: NFS works great for ESX datastores. Its a whole lot easier to manage than iSCSI and although iSCSI is generally considered to perform better, we find that the flexibility of using NFS more than makes up for the lost performance.

If you have sufficient budget, there are great solutions from Dell, HP etc. where you can get fault-tolerant ESX installations already setup in a rack, which not only provide data-store fault tolerance, but also VM failover and so on. But there’s a lot of people or companies out there who have several servers running the free ESX5i hypervisor and who would still like to have some fault-tolerance. This article is for those people.

What’s not so great about shared storage (like NFS or iSCSI) is that you generally have a lot of extra boxes around and complicated network configuration. If you want fault tolerance, you’ll need two ESX servers, two NFS servers and two switches (and a bunch of cables to connect them all). Furthermore, you’ll add a lot of complexity to configure those (often proprietary) NFS boxes. I recently configured a couple of Lefthand boxes for a customer and it not trivial to set up.

So I figured there must be an easier way – after all, ESX is an amazing platform for reducing the number of boxes in the rack, so why would I want to start adding boxes again if I don’t have to.

The first important point is that ESX provides the VMXNET3 10Gb virtual ethernet adapter, so that even if your ESX server does not have 10Gb network cards, the VMs running on the server can communicate with each other at 10Gb and, more importantly for our purposes, ESX itself can communicate with its VMs at 10Gb speeds. So if we run an NFS server as a VM on the ESX server, and use it as an NFS datastore for the ESX server, then the server will see it as a 10Gbps NFS server.

OK, but what about the fault-tolerance? For that, we need to replicate the NFS server’s disk to another server. So, if we don’t have a real 10Gb network, that’ll have to be across 1Gb. Does that slow things down? Apparently not much – we’re using DRBD asynchronously which causes a minimal performance hit.

So what we do is to clone the NFS server VM to a second ESX server and set up DRBD replication between the two VMs. We’re using Ubuntu 11.10 server (you’ll need a reasonably recent Linux distribution to get the 10Gbps support with the VMXNET3 virtual network adapter).

Because you only get the 10Gbps datastore access to the NFS server when the NFS VM is hosted on the local ESX server, this is not really shared storage (or at least its shared only at 1Gbps to other ESX servers). However, for our fault-tolerance purposes that doesn’t matter much. In fact, from a scalability point of view, it makes sense to provide each ESX server with a locally-running NFS datastore, accessed at 10Gbps and replicated to another ESX server. This scheme also has the advantage that each ESX server is autonomous – ESX servers with remote datastores always make me a bit nervous – any problems on the network and the VMs are likely to freak out. This way, the ESX server is completely self-contained – it only needs another ESX server for fault-tolerance. Even if the network fails, the local NFS datastore will be unaffected (except that fault-tolerance is temporarily suspended) and when the network is available again, the DRBD secondary will simply catch up again automatically, providing fault-tolerance again.

ESX Server 2 can do the same thing with another pair of NFS servers (a local one for fast access and a remote one on ESX Server 1 for fault tolerance). This idea can be scaled indefinitely – each ESX server having its own local NFS VM running its datastore and replicating to another ESX server. The major advantage of this approach is that its more scalable than a single fault-tolerant pair of shared NFS servers and you get 10Gbps access for free. Conversely, the price you pay for this is that you have a separate NFS server for each ESX server which makes administration more complex than for a single shared datastore (but hey, you can’t have everything, at least not for free).

You could additionally configure the nfs servers to fail over a shared ip address to the secondary – we haven’t bothered to do this since if the primary nfs server fails, its likely that the whole ESX server has failed. If that’s the case, we need to promote the DRBD secondary to primary manually, start the NFS server and register the VMs.

And how does it perform? Pretty well actually. The benchmarks below are made on an ESX5 host with a single i7 930 CPU, 24GB RAM, an Adaptec 5405z controller and 4x SATAII disks in RAID5.

Disk performance of a VM running directly on the ESX host (i.e. on the local datastore).

As you can see from the above, the VM running on the fault-tolerant NFS datastore is not as fast as the VM running on the local datastore, but it’s sufficiently fast for the subset of your VMs which require more fault-tolerance than that provided by daily backups.

Speaking of backup, we’re using ghettoVCB to backup 150 VMs (>1TB) every night via NFS to a Netgear ReadyNAS Ultra 6 device. We then rsync them to a remote data-center for offsite, versioned storage.

We had built up a mocked service layer for our Wicket Tester unit tests (with Mockito). This enables our unit test to start any page or panel in our app with reasonable test data. However, when it came to development, we were still using the real service layer – which meant that as our application got bigger it took longer and longer to start and navigate to the page under development.

Why not use the mocked services during development too? Good idea, we thought. We added development entry points to our unit tests which allow us to start a minimal Jetty container to run the pages under development in a mocked service sandbox.

Turns out this works great – start time goes down from 10 seconds to 1 second and you have all the power of the mock scenarios already developed for unit tests during development.

This not only saves time during development, but encourages better unit tests (since you build better mock scenarios during development and can then reuse them for unit tests).

It also finally allows us to decouple application development from service development, meaning that the application developers can start immediately with application development and contribute to the definition of the service layers from a client perspective by creating mocks as needed.

We use Maven to manage dependencies during development. This entails adding a pom.xml file to our Eclipse project which defines the jars which the application depends. Maven then takes care of fetching the right version of the jars from a number of repositories (central maven repository, vendor specific repositories, our own repository).

This works pretty well and its hard to imagine developing complex projects without this capability. However, when it comes to ensuring that an application is delivered to the production environment with all its dependencies, you’re pretty much on your own. You have to build either a war file or a jar-with-dependencies – both of which can be very tricky and lead to problems occurring in production which you never saw during development.

Additionally, our applications tend to have a lot of dependencies and the war files get huge.

So, we thought, why not just use Maven on production servers to fetch applications and their dependencies.

To do this we maintain a pom.xml file on the production server with the application listed as a dependency. We use the maven goals “versions:use-latest-releases” and “versions:commit” to update the pom file automatically to the latest release version. We then use the “dependency:build-classpath” goal to build a class path from the repository and finally run the application.

Wicket tester combined with Mockito is fantastic for unit testing your web application. However, developing the tests is harder than it should be because you have to refer to components by their path rather than their id.

i.e. instead of “ok”, I must refer to the button by its full path “border:form:pagetitlecontainer:ok”. This seems tedious – if I know that I only have one component on the page with the id “ok”, why can’t I just use “ok”. Wicket tester should notice that the component is not available and traverse the component hierarchy for it. If its ambiguous then it can complain, but if not, then it should be OK.

To get around this limitation, we override the getComponentFromLastRenderedPage method of the WicketTester class as follows:

My holiday reading this year was “Adapt” by Tim Harford. One of the most interesting parts was about Peter Palchinsky, a great russian engineer who was an advisor to both the Tsar and to Stalin. Although his uncompromising honesty got him exiled to Siberia by the Tsar, pardoned again and finally murdered by Stalin’s secret police, he nevertheless had time to formulate 3 principles for innovation:

Try lots of things, expecting many to fail.

Make sure the failures are survivable.

Learn from the failures.

As Tim Harford points out, this is roughly how evolution works in nature and it seems applicable to software development. Since customers don’t generally appreciate failures on their projects, this trial and error cycle needs to be carried out outside of production projects – in the “20%” projects we do on our own time.

I’m a big fan of Apache Wicket, but there’s an urgent need for a better-organized component-ecosystem around Wicket. To write a real web application you need a framework (that’s Wicket) and visual components (that’s JQuery, YUI, Recaptcha etc) and you need the two to work together smoothly.

A newcomer to Wicket should be impressed by the clean, component-oriented approach to producing the GUI for his application, but a production web application needs more than just HTML. So, inevitably, the next step is to try to find the stuff which will make your web-application look like all those other web 2.0 sites out there. However, very few of the Wicket components referenced from the Wicket home pages are “best-of-breed” and a certain amount of disappointment soon sets in.

Take the wicket example captcha implementation for example. There are two problems here: (a) the implementation produces a captcha image which is not only difficult for robots to read, but also pretty impossible for humans and (b) everybody else is already using recaptcha so why even bother distracting developers with yet-another home-spun implementation.

The Wicket community might react by saying “well, there are enough blogs describing how to use recaptcha with Wicket, so where’s the problem?”. Well the problem is that these blogs typically provide their description in a way which reflects their own expertise rather than that of the reader, which means that a novice Wicket programmer may or may not be able to follow the instructions to successfully integrate recaptcha into his application.

Or to put it another way, developers blogs are not a substitute for organized, tested and documented component libraries.

What could help, on the other hand, is a catalog of components, with ratings and a compatibility matrix indicating tested interoperability with other components/browsers etc.

By the way what prompted me to write this post was that yesterday I finally used a few of the Visural Wicket components (based on JQuery) and they are fantastic – well thought out APIs, styling etc. What I don’t understand is that mediocre-to-lousy versions of equivalent components (such as the Wicket extensions modal window or the wicketstuff lightbox) feature much more prominently on the Wicket sites, so the novice programmer is far more likely to use them (and be bitterly disappointed) than to use Visural Wicket (and be delighted).

Wicket adoption would likely be accelerated if the end-to-end experience of producing the first production web application involved a bit less trial-and-error and a bit more delight.

Yesterday we had a bug caused by the following situation: a class X had a variable “user”. Class X had an inner, anonymous class extending class Y. Inside the inner class a reference was made to the “user” variable from the enclosing class X. Somebody then added a protected variable “user” to class Y. The java compiler silently updated the reference to “user” within the inner class from X.user to Y.user.

This is pretty nasty and I hoped that some of the code-checking tools would warn about about it, so I checked the code with PMD, Findbugs and Checkstyle. Unfortunately, none of them flagged the issue.