Monday, August 23, 2010

As the Chinese curse has it, ‘May you live in interesting times’. These are certainly interesting times for the UK’s public sector. The coalition government has stated that it intends to cut non-health public services by 25 to 40% - the biggest cuts since the Second World War. They also say that most of these cuts can be had by efficiency savings. I personally think that they can cut that much from public sector IT and leave it healthier, but it needs to be done in the right way.

In the private sector money rules. Capitalism is an ongoing process of Schumpeterian creative destruction as firms are created, compete for profits and die. Survival depends on firms cost-cutting and innovating. The public sector exists in an entirely different context. How do you create a public sector framework that incentivises efficiency and encourages innovation, and more importantly, allows for failure? This has been a question for every government since the invention of taxes.

Well, there is an evolving self-organising system, not based on a monetary economy that often beats Capitalism at its own game. It produces more variety, better quality and out-innovates. It is, of course, the world of open-source software (OSS), one of the most profound and utterly unexpected outcomes of the internet. What is surprising is that it hasn’t been studied in more detail by policy wonks. I believe that public sector organisations could be transformed by learning lessons from OSS, and this especially applies to public sector IT.

Here’s what the government should do:

Every business is a software business

A common refrain you hear from managers, not just in the public sector, but they say it more than others, is, “We are not a software business”. Wrong. Every business in the 21st century is a software business. Much of the work of the public sector is information management: tax collection, benefits, regulation, health records, the list goes on. This work has IT at its core.

Unfortunately the prevailing trend has been to outsource software development and infrastructure. The UK civil service has gone from being a focus of expertise to being largely hollowed out. This needs to change. The civil service needs to put expertise in IT at its core. In the same way that the government is putting doctors back in charge of the NHS, it needs to put geeks in charge of IT.

Build IT Skills in the civil service

The civil service should work hard at becoming a major centre of UK IT skills. That means bringing the majority of IT expertise back in house and fostering a culture that makes the civil service a place where geeks want to work. Simple things like dress codes are important. There is simply no reason why a programmer should not be allowed to come to work in a t-shirt, shorts and flip-flops. In fact there’s no real reason for them to come to work at all – there is no need for people to be physically co-located in 2010. Insisting on this severely limits the pool of candidates. Public sector managers need to understand what motivates knowledge workers and handle them appropriately.

There also has to be a career path for developers that allows great coders to stay being great coders and not have to become managers. The UK IT workforce is warped. Because coding skills are undervalued, salaries tend to be low compared to jobs that require a similar level of expertise. This means that programmers almost always join the contract market at some point in their career, simply because it better reflects the real supply and demand for their skills. Many teams reflect this with the junior members being predominantly permanent staff and the senior ones being contractors. This is a problem, because permanent staff in are almost impossible to fire. When cost cutting is demanded, the contractors are laid off first. This guts the organisation of its core skills.

So pay rates for programmers need to be higher. But how do you know if a programmer is good enough to be paid these new high rates? There are no realistic qualifications in IT. I believe that a reputational system based on OSS is the best way to judge a programmer’s quality. It seems natural that a contributor to a successful, high-profile system should be valued more than someone without any demonstrable experience. Open-sourcing government IT would automatically build a transparent reputational market for IT skills.

Fix procurement

IT procurement in the public sector is broken. The seemingly laudable aim of regulating and monitoring procurement, especially of high value contracts has put a huge barrier in front of businesses that wish to supply the public sector. This distorts markets and encourages rent seeking behaviour. Only larger companies with deep pockets are able to afford the lengthy process of bidding for public sector contracts. And of course, once you’ve spent millions on securing a contract, you expect some payback. This leads to hugely inflated costs for software. Not only that, but the main effort and expertise of suppliers is focused on pre-sales and contract negotiation, once the contract is signed you often find that more junior managers and technicians take over.

As a freelance developer I have been hired several times by large suppliers to work on teams developing software for the public sector. There is no secret ingredient that they have. Often the project management and in-house development skills are poor to say the least. In many cases the team building the software is almost entirely freelancers, in which case the end-client could have hired them directly at a fraction of the cost, and probably experienced less friction in the process.

The outsourcing trend means that civil servants with little software development experience themselves have to manage contractual relationships with suppliers. They don’t know what questions to ask. They often have little idea of the cost of particular features or applications and have been acclimatised by suppliers to expect large numbers. Before long you have the situation where a few tens of thousands of pounds for some fields to be changed seems perfectly reasonable. I kid you not, I’ve seen it with my own eyes.

Because of the lack of internal IT skills and the corresponding reliance on external suppliers, there is a very negative attitude to open source software in the public sector. Of course you, dear reader, know full well that many open source software products are not only much cheaper (as in free) than their commercial equivalents, but are also often better quality. But without the internal expertise to evaluate them your only option is to listen to salesmen from commercial suppliers, and that is pretty much how most products get chosen in the public sector.

A civil service skilled in IT would be able to not only evaluate open source products, but also contribute to them and instigate new ones. The network effects would be significant and would have the potential to turn the UK into a world leader in open source development. Applications, infrastructure and innovations could be far more readily shared between departments, but it would also be a way for tax pounds to directly contribute to wider economy. Indeed, it only seems fair that a product produced at the taxpayer’s expense should be available to the taxpayer to use.

It should be policy to only consider a commercial product if it can be demonstrated that no practical open source alternative exists. Where there is no existing open source alternative, the first consideration should be to extend an existing project or create a new one from scratch. All software developed by the public sector should be open source by default. Only where there are demonstrable security concerns should this not be done.

By building software in the open, public sector organisations will be able to learn from each other in a real and direct way. This means being public about failures as well as successes. In the same way that I know the names of the developers of NHibernate and the Castle Project (two very successful open source projects), the names and reputations of the technologists and managers behind successful public sector projects will also be well known. When an organisation is considering a major new build they will know who to turn to, not on the basis of slick presentations, but on real measurable results.

An example of successful in-house development

I’ve just finished a very pleasant year working as a technical-architecture consultant with the UK’s Pensions Regulator (TPR) which is part of the Department for Work and Pensions. It is public knowledge that this department has been asked to prepare estimates for 25% to 40% cuts.

TPR’s work as a regulator is predominantly casework based. They monitor UK private sector pensions, risk-assess them, and provide advice to trustees. The case management process has been handled historically using a combination of off-the-shelf software products, spreadsheets and word documents. A dedicated case-management system was desperately needed.

Now you might think that you can buy a case-management system off the shelf, and yes, you can. But you can’t buy one specialised for managing pension regulation. Any product would have to be deeply customised. It would also need to be integrated with the rest of TPR’s systems to be effective. Some effort was expended looking at commercial alternatives, most with costs of around £100,000+. This does not include the customisations, which would, of course, have to be carried out by the supplier. The cost of the consultancy work could only be estimated from detailed specifications, the creation of which would have imposed considerable extra cost to TPR. There is little doubt that the internal and consultancy costs would have easily outweighed the ticket price.

But luckily TPR, for historical reasons, has a strong internal software development team. So after some debate, it was decided to build a bespoke case-management system from scratch. This took a team of two developers around 7 weeks to complete with a conservatively estimated cost of around £30,000. That’s right, building bespoke software in-house was at most 30% of the cost of using an external supplier. Because the system was built in-house there was no need for lengthy contract negotiations and detailed specifications, instead we built it incrementally with constant input from the case management team. They got exactly what they needed at a fraction of the cost.

Going forward, there won’t be any need for expensive maintenance contracts and huge costs for each change or refinement, because it can all be done in-house. Of course TPR need to maintain their development team, but I firmly believe that those guys are a lot cheaper than the outsourced alternative.

It’s a piece of software that I was very proud to have been a part of. It would have been useful to other public (and private) sector organisations, not simply as a casework system, but as a demonstration of a particular technology stack and methodology. It would also have been an excellent advertisement for the technical prowess of the team. It’s a real shame that this great success story can’t be told outside the confines of TPR. You will never get to see the beautifully factored code with over 1500 unit tests, or know the names of the people who put it together.

So, Mr Cameron, here’s something truly radical you could do that fits wholeheartedly with your aims for a more open and inclusive government:

Wednesday, August 18, 2010

If you’ve attended in the past, or plan to in the future, or even if you’re a Brighton based dev, please create an account for yourself and add your name to the list of members with a little info about yourself too.

It would also be very nice if we could update the wiki with stuff that we talk about, maybe a new page for each meeting.

I just read an old blog post by Wes Dyer showing how Linq Syntax uses Monads. I’ve been really struggling trying to understand Monads and learning Haskell over the last few months, and I’m really not there yet, but Wes’ post really lit a lightbulb. I’d naively thought Linq was simply all about IEnumerable<T> or IQueryable<T>, and for 99% of the examples and code out there, it is. But actually it’s far more generic than that.

Here’s a really silly example bastardised from Wes’ post. I’m going to wrap two integers in a context, or container, and then do some operation on them using Linq syntax so that I don’t have to explicitly dip into the container.

It just wraps an instance of some arbitrary type along with a message.

Next we need some extension methods, namely two varieties of SelectMany (which is effectively ‘bind’ from Haskell’s Monad), I’ve also added a ToContainer extension method, but it’s only to make things look nice and isn’t necessary for the Linq syntax to work:

Now we can use Linq Syntax to wrap two items (in this case integers) in containers, but still do operations on them without having to explicitly extract them from their containers. At the same time the containers themselves are combined as defined in the second SelectMany above:

Is that not just amazing? Think of all the times you need pass around something wrapped in some context and then keep digging it out to do some operation on it, this technique could really clean up that code.

I’d very much recommend reading Wes’ blog post in full, and have a look at Monads, they are mind bending, but have the potential to radically simplify many programming challenges.

For an in depth example, check out this post by LukeH where he uses Linq Syntax to build a monadic parser. Chris Patterson has an excellent working example of such a parser in his Magnum library.

Tuesday, August 03, 2010

Here’s a cunning way of writing a blog post, get other people to do it for you! Luckily for me, the people who (for some reason) follow me on Twitter are a really clever lot. Yesterday I asked the question, “What do people use for stress testing their web application?” the response was so good, that it has to be recorded as a blog post.

This is the tool I’ve been using as well. It works great, but users get upset when the website is stressed beyond capacity. Also I really want to see how far I can push Suteki Shop before it breaks. Up to now, it’s coped very well with the small number of users thrown at it. I want to see what happens when it gets Slashdotted – and no please, that’s not an invitation :)

“Web Capacity Analysis Tool (WCAT) is a lightweight HTTP load generation tool primarily designed to measure the performance of a web server within a controlled environment. WCAT can simulate thousands of concurrent users making requests to a single web site or multiple web sites. The WCAT engine uses a simple script to define the set of HTTP requests to be played back to the web server. Extensibility is provided through plug-in DLLs and a standard, simple API.”

Several people said they’d successfully used this. The major irritation with WCAT is that you have to script every request, so to be realistic you’d have to log all the requests in a typical user session and then hand-craft a script to be played back. I wonder how hard it would be to write an IIS log-file to WCAT script tool?

This is an cloud based service that you pay a subscription for, although they do have a free option for up to 25 concurrent users. There seem to be a few of this kind of service out there. I guess the advantage is that you get a much more realistic scenario, an external source hammering your application, than an internal tool. Beyond that, I’m not sure why I’d use such a service.

This tool isn’t really intended as a stress test tool. It’s primary use is for UI testing. It provides a nice API to launch and operate Firefox and IE so that you can write C# UI tests. However by launching multiple instances one of my correspondents has been successfully using it for stress testing. I guess my main question would be, how much hardware I would need to run enough instances of Firefox or IE to simulate high loads.

Selenium is another UI test tool, similar to WatiN. It’s what I use for UI based tests. Selenium Grid allows you run to many Selenium tests in parallel. Once again, it’s primary design goal is to aid UI integration build tests, but there’s no reason you couldn’t use it run stress tests.

Selenium 2 has merged in WebDriver which can drive the headless browser HtmlUnit (see below), this might mean that sufficiently high loads can be driven from minimal hardware.

This is a tool from Google. It’s a Chrome extension and looks like it’s primarily designed to profile AJAX applications, especially GWT based apps. For that it looks excellent. I wonder how easy to would be to use it for stress testing a web server though. At the very least it would face the same issues as WatiN and Selenium Grid; needing a large number of Chrome instances.

“HtmlUnit is a "GUI-Less browser for Java programs". It models HTML documents and provides an API that allows you to invoke pages, fill out forms, click links, etc... just like you do in your "normal" browser.”

JMeter, another Java tool that looks just the ticket. It’s designed specifically for load testing:

“Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load.”

If you’ve already paid for Visual Studio Ultimate (or been lucky enough to have been given a copy, thanks Liam!) then it’s probably worth checking this out. At first glance it looks pretty comprehensive and you get all the nice VS integration.

So thanks everyone for some great tips. If you’ve had a good (or bad) experience with load testing tools, please leave a comment. I still haven’t decided what to go with for load testing Suteki Shop. I’m leaning towards Selenium 2 + HtmlUnit, but I might give the VS LTA and JMeter a spin too. Watch this space.

Monday, August 02, 2010

Do you have an ‘Application Server’? For some reason it’s a very popular choice to separate web applications into a front-end UI/Controller deployment, a separate server for back-end ‘Business Services’ and have them communicate via web services. I’ve never understood why this is a good idea and I’ve come to believe that people do it because of a number of architectural misconceptions.

Before I start I want to make a clear distinction between web applications and rich client applications. Rich clients do require an application server. I’m arguing that there’s no sane reason to build one for a web application. I’m also not talking about application service ‘Application Servers’ such as WebSphere or AppFabric.

1. Don’t confuse Logical and Physical architecture.

Good software design 101 says that you should separate concerns. You shouldn’t mix up UI concerns with data access concerns, and you should separate infrastructure from business logic. This is all good advice and working out the best ways to do it is really the core task of any programmer. However, when we say ‘separate’ we mean, ‘logically separate’. This means that these concerns should be separated in the structure of your software, it doesn’t mean that they have to be in separate assemblies, and it most emphatically shouldn’t mean that they should be deployed them on separate servers. As Martin Fowler says in Patterns of Enterprise Application Architecture, ‘The first rule of distributed components is don’t distribute your components’.

Having your business logic deployed on a separate server behind web services leads to all kinds of weird design pathologies. Because your controller logic can’t access your domain logic directly you can often find either business logic leaking into your controllers and UI, and/or chatty or over-complicated web service interfaces. This often can kill performance, which is ironic since performance is often used to justify the whole application server idea.

2. You can deploy multiple copies of the same assembly.

A common argument for application servers is that they aid reuse. By having all the business logic deployed on a single server, multiple applications will be able to reuse the same web services. For example, a fat client web forms applications will be able to share the same business logic as a web application. Messaging components will also be able to access those same web services.

The unwritten assumption is that we can only ever have one deployed copy of an assembly. However, the in-process object-oriented API of your domain entities is far better for reuse than a bunch of web services. There is nothing to stop you from having one deployment of your business logic in your web application, another in your application server that supports your rich clients and yet another in your messaging component.

It’s also often the case that when you start to have multiple applications consuming your web services, you find that their requirements are diverse enough to mean that reuse is minimal. The requirements of a request-response web application are not the same as an event driven web forms client.

3. There should only be one access point to my database.

Shared databases are a plague on business IT. Somewhere in the depths of your organisation are a tribe of grumpy trolls called DBAs. They form a protective barrier around ‘The Enterprise Database’. No one is allowed to change the schema without twenty committee meetings and two hundred pages of documentation signed in blood. No one is allowed to access the database except through stored procedures two thousand lines long, lovingly crafted over the last ten years by said DBAs. The reason for this sorry state of affairs is the horrible 90’s idea that a single relational database can be the core repository for the entire enterprise. Of course you don’t want to repeat this mistake, so you mandate that there must only be one access point to the database; the application server.

This is very sensible, you should only have a single access point to a database. However we’re repeating the confusion between logical and physical deployment again. You should only have one logical access point, but there is no reason why you shouldn’t deploy it multiple times.

4. An application server will help me scale.

No it won’t, it will make things worse. If you have a single web server, the most scalable physical architecture is to put everything on the same box, including the database. Network calls are expensive, very expensive. Each extra network hop you require your application to make to server a request will slow it down. More resources will be taken making the network connection than you will save by moving processing to another server.

If you have multiple web servers, you gain nothing from forcing your application to talk to the database via an application server. SQL Server can easily handle multiple connections from a number of web servers, and if you have more than five web servers you are already in a new zone of scalability requirements and you should be looking at message based asynchronous architectures such as CQRS. The same applies if you have expensive processing requirements, hand them off asynchronously.

So to conclude, think carefully before you decide you need an application server for your web application. Putting all the application code on the web server is a far better approach. You can have a farm of servers sharing a single database and it will scale well. I did some consultancy for the UK property website findaproperty.com who use this approach very successfully, and Stack Overflow has a very similar architecture. Both serve around half a million page views per day, findaproperty with four web servers, Stack Overflow with two. Of course, if you are Google or Facebook, you need a different approach… and better advice than I can give.

Code Rant

Notepad, thoughts out loud, learning in public, misunderstandings, mistakes. undiluted opinions. I'm Mike Hadlow, an itinerant developer. I live (and try to work in) Brighton on the south coast of England.

All code is published under an MIT licence. You are free to take it and use it for any purpose without attribution. There is no warranty whatsoever.