software for councils

Not many people realise that we fund a proportion of our charitable work by carrying our commercial development and consultancy work for a wide range of clients.

Last year, we scoped, developed and delivered a real variety of digital tools and projects. Some of the projects were surprising. Some of them made us gnash our teeth, a bit, as we grappled with new problems. But all of them (and call us geeks if you like) got us very excited.

Here are just twelve of our personal high points from last year. If you have a project that you think we might be able to help you with in 2015, we’d love to hear from you!

2. We helped the MAS and the FCA protect financial consumers

Two of our projects helped people financially.

We built the Money Advice Service’s (MAS) first responsive web application, the Car Cost Calculator.

This tool takes one simple thing you know (the car you wish to buy) and tells you roughly how much it’ll cost to run that car against any others you might be interested in. It has been one of MAS’ most successful online tools in terms of traffic and conversion.

This tool helps users considering a financial investment to check a potential investment. Users enter information about the type of investment, how they heard about it and the details of the company offering it to them and get back tailored guidance and suggested next steps to help them ensure the investment is bona fide.

3. We Gave Power to the People of Panama (soon)

Working with the The National Authority for Transparency & Access to Information (ANTAI) and the Foreign & Commonwealth Office (FCO), we set up our first government-backed instance of our Freedom of Information platform, Alaveteli, in Panama.

This project will ensure that Panama’s FOI legislation is promoted and used, but it will also shine a light on ANTAI, who are responsible for ensuring ministries and organisations publish their information, and handling case appeals.

4. We Mapped All the Public Services in Wales

After we extended the Mapumental API to produce data output suitable for GIS (geographical information systems), the Welsh Government were able to map public services in Wales for their Index of Multiple Deprivation calculations.

Over the course of the year they have calculated travel times for over seventy thousand points of interest.

5. We Launched a New Organisation in Four Weeks

Simply Secure approached us in dire need of a brand, an identity and a website to accompany the launch of their new organisation to help the world build user-friendly security tools and technologies.

Cue four weeks of very intense work for mySociety’s designer, supported by members of the commercial team. And we did it.

6. We Printed Stuff BIG (and found people jobs)

mySociety developer Dave Arter figured out how to generate A1 sized maps from Mapumental for every job centre in the UK – all 716 of them.

Xerox will be using these with the DWP to help job seekers find work that is within reach by public transport. As a byproduct, Mapumental now handles high-fidelity print based outputs: get in touch if that is of interest.

7. We Opened Up Planning Applications

With Hampshire County Council we had the opportunity to build a new application to help assist members of the public and business better understand what was happening around them. For us, it was also the first application in which we worked closely with a provider of a linked data store, in this case Swirrl.

When Open Planning goes live, it will look to help improve social engagement and the economy of Hampshire through better understanding and transparency of planning data.

8. We Proved (Again) That FixMyStreet Isn’t All About Potholes

After a spate of cyclists’ deaths in London last year, we felt that the moment was right to build something that would support cycle safety in the UK.

11. We Fixed Yet More Potholes

That means that residents of those places can now make their reports direct from their council’s website, or via FixMyStreet, and either way they’ll have all the benefits of FixMyStreet’s smooth report-making interface.

12. We Showed Parliament the Way

And so, we end where we began. While Parliament were busy interviewing candidates for their new ‘Head of Digital’ position, we were commissioned to demonstrate what Hansard might look like were a platform like SayIt used instead of the largely print-based publishing mechanisms used today.

The result was shared internally. While SayIt may not be the end solution for Parliament, it’s great to have had some input into what that solution might be.

In yesterday’s blog post we talked about using our free, Open Source software, SayIt, to create collections of statements, like our collections of Party speeches.

That’s one use of SayIt – but we actually built it with a slightly different aim in mind: the storing and publication of transcripts.

SayIt really does transform transcripts – so, if you regularly take minutes of meetings at work, or in another capacity, it’s worth a look.

That’s easy for us to say, we know. But if you play with it for half an hour, we think you’ll see the benefits.

Making online transcripts better for your readers

Traditionally, transcripts of meetings are published as PDFs or Microsoft Word documents. The information is there; you’ve done your duty in making it available – but do you ever wonder if it’s really working for your readers?

For example, let’s say you are a clerk in the local council, and you routinely publish transcripts from council meetings online.

The chances are that residents access your transcripts when they have an interest in one specific topic. Typically your meetings cover many subjects, and readers have to wade through pages to find the part they want. On SayIt, searching is very easy, even for people who are not very familiar with internet technology.

Or suppose that you are a member of a pressure group, and you’ve transcribed a local community meeting to share on your website. You might want to highlight particular parts of the meeting. With SayIt, you can link to individual statements, so it’s simple to share them by email, social media, or on your website.

See some examples

If you’d like to see how your meeting transcripts will look, once they’ve been published on SayIt, have a browse through these two examples:

Getting started

Part of the URL (web address) for your site – for example, if you choose “TotnesCouncil”, your new URL will be http://TotnesCouncil.sayit.mysociety.org. Note that URLs can’t contain spaces or non-regular characters.

A title: this will appear in the top bar of your website. Don’t sweat too much: you can always change this later. In this example we might choose “Totnes Council meetings”.

A description (optional): this is a good place to explain the purpose of your site at a little more length. You might write something like “Transcripts from local council meetings in Totnes, UK, 2014 onwards”. Again, you will have the chance to change this later if you like.

2. Confirm your email address

If this is the first time you have used SayIt, you will need to input your email address, then go to your email and find our automated message so you can click on the confirmation link.

Keep a note of your password, as you will need it whenever you want to edit your site.

Inputting transcripts

SayIt is currently in Beta – that’s to say, it’s functional and live, but we’re still developing it.

In this phase, you can manually type (or copy and paste) each statement of your transcript in. Soon, it will also be possible to import a document of the entire meeting, as long as it’s in the required format – if you have a lot of existing transcripts and you’d like to try this, get in touch and we may be able to help.

In this post, we’ll look at the manual input of speeches.

Manual input

You will need either a copy of your transcript, or a recording of the meeting you wish to transcribe.

Here’s how to begin:

1. Click on the ‘add your first statement’ button.

2. You can paste, or type, your content directly into the box marked “text”.

In the fields below the text box, you have the option to add more details about this piece of text. None of these fields are mandatory, but all of them add functionality or information to your transcript:

Date and timeIf you know these, they are useful because they will help SayIt to order your speeches chronologically. Don’t worry if you don’t know them, though – SayIt automatically arranges speeches in the order that you input them, unless the timestamps tell it otherwise.

Event and location What sort of meeting was it, and where did it happen? For our example, we might input “Totnes Town Council Meeting” and “Guildhall, Totnes”.

SpeakerEnter a name, and then click on the underlined text to add it to your database. As with all text fields on SayIt, once you have added it, it will be offered as an auto-fill option for subsequent speeches. Attaching names to your speeches also means that SayIt can do clever things, like display everything said by one speaker.

If you are not sure who spoke, don’t worry – you can leave this field blank, or enter a name such as ‘Unknown’.

SectionMeetings often have distinct sections: an introductory period, apologies for absences, following up on agreed actions, etc. Or you might use Section to identify items on the agenda. If you use the Section field, SayIt will automatically arrange your transcript into groups of associated content.

Source URL If you are taking speeches from a source such as a news report or another website, you can add the web address so that interested people can see it in context.

Title and tags: These enable you to tag your content – for example, you might want to tag everything to do with road-building, and everything to do with tourism, et cetera. That means that your readers will be able to find the sections of the content they are most interested in.

When you’ve added everything you want to for this part of speech, click “Save speech”.

Well done! You’ve just added your first speech to SayIt.

You can go back and edit it at any time – and that applies to every field.

3. Continue adding speeches.

As you do so, SayIt will be making connections and organising things neatly.

Tip: If you click ‘add another speech like this’ then fields such as ‘event and location’ will automatically be filled for you – you can overwrite them if they are incorrect for your next speech.

Click on ‘Speakers’ to see an icon for everyone you’ve added:

– and click on any one of those icons to see just their speeches:

Clicking on ‘Speeches’ in the top bar will show you every speech you’ve input; if you used Sections, they will be divided up neatly:

Click on any of those sections to see its content:

You’ve done it

So there you are, now you’ve seen what SayIt can do – we hope you liked it enough to consider using it in the future. Remember, it’s completely free.

Let us know if you hit any problems, or if there are features you’d like us to add. SayIt is in active development at the moment, so your feedback will help shape it. We’d also love to hear if you are using it.

Importing content

Manual inputting is clearly only practical for shorter meetings (or people who have plenty of time on their hands!). As mentioned above, we’ll be adding the ability to import your transcripts.

They will need to be in the format that SayIt accepts, which is Akoma Ntoso, a schema for Parliamentary document types – you can read more about that here.

If you already have documents in Akoma Ntoso, get in touch and we can get them imported for you.

Hosting

You can host SayIt on your own servers, but for beginner users it’s quicker and easier to start by creating a version that we host, as described in the steps above.

If you decide later on that you want to host the content yourself, and perhaps embed it on your own website, that option will remain open to you.

SayIt is a Poplus Component – open-source software that is designed to underpin digital democracy projects. It can stand alone, or work with other Poplus Components. The source code is also available for developers to modify and improve, so if you are already imagining more ambitious ways that you might use SayIt on your website, let us know.

Every day, thousands of planning applications are submitted to local councils around the country by people applying to demolish a garage, erect a fence or convert a loft. More often than not these applications disappear into proprietary systems that, despite being publicly available, make it hard for members of the public to find out what’s going on in their area.

Last week, we kicked off the first sprint of an exciting new piece of work with the Hampshire Hub Partnership to build a prototype, open source web application to help members of the public find out more about planning applications in their area.

We jumped at the chance to work on this for a number of reasons.

Serving the needs of the public

Firstly, it has the needs of the general public as its focus. The planning process can be baffling if you’re new to it and this tool aims to help make it easier to understand. We’ll be helping people answer some of the most common questions they have about planning applications: What applications are happening near me? What decisions have been made in the past on applications like mine? How likely is it that my application will be dealt with on time?

A wireframe illustrating the potential functionality of the search results page

The site will help people browse planning application data by location — whether a postcode or a street address — and by type — whether it’s an extension, a loft conversion, or a major development like a retail park or commercial warehouse.

Built on Open Data

Secondly, it’s being made possible by the release of open data from local councils, once Ordnance Survey has granted the necessary exemption for locations derived from their data. Many of our projects rely on organisations publishing open data, so it’s great to have the chance to help demonstrate the value of releasing this kind of data openly.

The Hampshire Hub team has already spent a lot of time working with the LGA, DCLG and LeGSB to define a schema for how planning application data should be published. They’ve collaborated with local authorities, in particular Rushmoor Borough Council, to gather planning application data. And they’ve worked with Swirrl to set up an open data platform to collect all of this together, publish it openly and give us and others access to it.

Reuse, don’t rebuild

And finally, rather than build something from scratch, we’ll be using the fabulous PlanningAlerts.org.au open source codebase as a starting point. Planning Alerts is a piece of software built in Ruby on Rails by our friends down under at Open Australia. It gives us a lot of the functionality that we need for free. We plan in time to repay them for their kindness by submitting the features we develop back into their codebase (if they want them, of course).

We’ve just started our second sprint of work atop the Open Australia codebase, building the search functionality we need to help people find applications by location and category. We’re looking forward to seeing the tool grow, get into the hands of users and fill up with data.

This new functionality enables nominated members of staff to edit user reports from within the FixMyStreet front end.

It’s quick and easy, and allows you to react immediately to unwanted content on your site. Read on to find out more.

Screenshot of a problematic report in FMS

What’s wrong with this report?

So what is wrong with the report in the screenshot above?

If you run a site on the FixMyStreet platform, you’ll be familiar with this kind of report, and the chances are that you’ll already be twitching to edit it.

User-generated content is wonderful in many ways – but it can also present problems on a public-facing site. Let’s look at a few of the potential issues in the report above:

The user has included his phone number in the report description, and now it’s available for anyone to see.

The user’s name is also public. While this is the default option on FixMyStreet, users often get in touch to say that they meant to make their report anonymously (an option on FixMyStreet, but one which the user can only access at the point of submission).

There’s an inappropriate photo. This one is a statue of Carl Jung, which obviously has nothing to do with the report. But even relevant photos can be problematic: imagine if it was a graphic depiction of a dead animal, or some rude graffiti.

Profanity: in the example above, we’ll imagine that “pesky” is a mild profanity, but experience tells us that users don’t always hold back on their language.

There are other common problems too, not represented in this report. Users sometimes post potentially libellous information: naming someone they suspect of flytipping, for example, or giving an address where they believe planning permission has been flouted.

In the run-up to local election, councils have to be particularly sensitive to any content that might be construed as political – commonly they wish to remove any mention of any candidate.

Moderation in all things

Up until now, we’ve edited reports for our council clients, on request. However, this is clearly a long-winded way of getting sensitive material off the site, especially when time is of the essence.

So we’ll shortly be introducing the ability for client moderation of sites. Councils or other bodies who run FixMyStreet will be able to nominate trusted users and give them the ability to edit problematic reports from within the report page.

When logged in, these users will see a “moderate” button on every report – this feature will not be available to any user unless explicitly authorised.

As you can see, this panel provides the ability to:

Hide the report completely

Hide the name of the poster

Hide or show a photo (if one was originally provided)

Edit the title and body of the report.

For some reports, it might be necessary to make a number of edits, and finally submit the changes:

FMS Moderation in Progress

The moderator can also add a reason for the changes, so it’s recorded if a colleague needs to know the history of the report in the future.

This functionality gives a lot of power to admins to remove inappropriate information – but the user took the time to submit their report, and it’s only fair to let them know it’s been changed. So the system sends them an automatic email, as below:

FMS Moderation Email

Finally, the system automatically updates the report to show that it has been moderated. As well as a timestamp, it signals where any information has been removed in the title or body of the report.

FMS Moderation Displayed on Report

Updates can be just as problematic as reports, so the same functionality will apply to them.

We’d welcome feedback on this mechanism, so please let us know if you think we’ve missed any features.

Note: These screenshots are from our work in progress and do not yet display the slick design that we habitually apply right at the end of the build process. Please regard them as preview shots only!

You may be familiar with WhatDoTheyKnow, our website which simplifies the process of making a freedom of information request.

mySociety also provides the underlying software as a service for councils: it sits on the council website, templated and branded to fit their site’s style. When someone submits a request, it goes directly into the council’s own back-end processes.

Just like WhatDoTheyKnow, the system publishes all requests, and their answers, online. This helps the council show a commitment to transparency – it also has the effect of cutting down on duplicate requests, since users can browse previous responses.

Now, ordinarily, when we sign off a new project for a client, we write up a case study for our blog. But this time, we were delighted to read an interview by Matt Burgess on FOI Directory, which has done all the hard work for us. With Matt’s permission, we are reproducing the piece in full.

The number of Freedom of Information requests public authorities receive is generally rising and central government dealt with more requests in 2012 than in any year since the Act was introduced. One council has decided to try and open up access to their requests using custom software from mySociety.

Brighton and Hove City Council have implemented a custom version of the popular WhatDoTheyKnow website where more than 190,000 requests have been made.

The council hope it will allow others to easily browse requests that have been made and make them more accountable.

We spoke to council leader Jason Kitcat about why the council decided to implement the new system – which was soft-launched at the beginning of November.

Why did you decide to implement the new system?

JK: I personally, and we collectively as a Green administration, believe passionately in openness and transparency. That’s the primary motivation. So digital tools to support making it easier for citizens to access council information I think are strongly in the interest of our city and local democracy.

We also were seeing an increase in the number of FOI requests, many of them similar. So using a system like this helps people to find the information that’s already published rather than submitting requests for it, when it’s actually already been published.

How does it work?

JK: It’s a customised version of the mySociety WhatDoTheyKnow site, delivered by mySociety for us in the council’s branding. It allows anyone to submit their FOI request in a structured way through the web and others can see the requests and any responses. The requests are linked in with the main WhatDoTheyKnow site to help further reduce duplication of requests and enable consistent commenting.

Behind the scenes it also offers workflow management to assist the council team who are responding to the requests.

What benefits will the system have to those answering and making FOI requests?

JK: It opens up the process, helps others to see what is going on even if they aren’t making requests themselves. Particularly important is that it by default puts requested information out there on the web without any more effort by the council or those making the requests.

Were there any obstacles in setting the system up and how much did it cost the council?

JK: Obstacles were mainly stretched resources within the council to prepare for the changed workflow, making sure our information governance was ready for this and that our web team could support the minor integration work needed.

Given this is a web-based ’software as a service’ offering it’s pretty straightforward to implement in the grand scheme of things. I don’t have the final costs yet as we’ve been doing some post-launch tweaks but, as is the way with nimble organisations like mySociety, I think pricing is very reasonable.

Do you think it will improve the council’s performance in responding to FOI requests and make the council more transparent to the public?

JK: Yes absolutely. Not only will the council’s FOI performance be more publicly accountable but I’m hoping we can reduce duplicate requests through this so that our resources are better focused.

Would you say it has been worth creating and why should other public authorities follow suit?

JK: Yes it’s worth it. I think we as councils have to be ever more open by default, use digital tools for transparency and relentlessly publish data. I believe this will result in better local democracy but also is one of the ways we can truly challenge cynicism in the whole political system.

N.B.: The website current shows a large number of requests that appear to be unanswered. We asked about these and it includes the number of historic requests that were loaded into the site.

————————————

Many thanks to Matt of FOI Directory for allowing us to reproduce this interview in full.

When we built FixMyStreet in 2006, our primary focus was to create a tool for citizens. We wanted to make it easy, quick, and effective to report street problems, even if the user had no prior knowledge of where their reports should go. And while the tool obviously had to work for the councils who were receiving reports, it never crossed our minds to research, or try to key into, prevailing council strategies.

But over the last few years, and to the benefit of both sides, council strategy has become strongly aligned with several of the qualities that FixMyStreet was founded on. The development of our specialised software, FixMyStreet for Councils, cemented that further, based, as it is, on consultation with local authorities.

If your current strategy focuses on any or all of the following points, then FixMyStreet is extremely well-positioned to help you.

Channel shift

UK local authorities are fully aware of the channel shift theory by now: put reporting online, make it self-service, and see efficiency rise while costs fall.

It sounds simple, but it hinges on one important factor – you have to get the reporting interface right. Otherwise, all those hassle-free online transactions turn into irate residents on the phone, seeking help.

Citizen engagement

On first impressions, many assume that FixMyStreet is just a public platform for grumbling – so it can be quite a surprise to discover that it often has the opposite effect. By allowing everyone to see what the problems are in their own community, it provides a platform for engagement, debate – and, sometimes, solutions.

FixMyStreet is a superb tool for councils who are looking for ways to encourage residents to take a stake in their own communities.

Usability

Any council web team worth its salt will be anxious to maximise usability across the website. FixMyStreet was designed with the user at its heart: from minimising the number of clicks it takes to make a report, to making sure that every step is as easy and comprehensible as possible.

Transparency

Modern society is demanding transparency across a vast array of organisations, not least government. By putting a record of every report online, FixMyStreet helps you fulfil those demands. And there are side benefits, too.

First, FixMyStreet brings previously ‘hidden’ work into the open, allowing your residents to understand the degree and quantity of work you do on their behalf.

And second, having reports online allows citizens to see at a glance whether their problem has already been reported, thus cutting down on duplicates – and saving you time.

Lowering costs

FixMyStreet is efficient when used on a desktop; it also works very easily on mobile devices, meaning that your residents help you crowd-source information. You’re effectively multiplying your inspection capabilities by a factor of hundreds, and your residents become your ‘eyes and ears on the ground’, as one of our client councils has said.

FixMyStreet.com has always tried to make it as simple as possible to report a street problem. When we built FixMyStreet for Councils, we wanted to simplify things for local authority employees too.

So, as well as offering the option to integrate with council back-end systems, we also put together this nifty dashboard (right – click to see full-size). It’s one of several extra features councils get when they purchase the FixMyStreet for Councils package.

What do councils need?

At-a-glance statistics, for all kinds of reporting. Perhaps the local newspaper have asked how many potholes have been fixed this year, or internal staff need a report on which types of problem are most rapidly fixed.

The top half of the dashboard allows for this sort of analysis. The drop-down category list means you can filter the view to show one category of problem – say, fly tipping – or all of them. Results are shown across a variety of timeframes.

FixMyStreet for Councils allows councils to designate their own progress statuses, beyond our standard ‘fixed’ and ‘open’. So, in this case, the statuses include ‘in progress’, ‘planned’, ‘investigating’, etc. Each of these is shown separately.

A realistic picture of how long it takes to deal with issues. The ‘average time to council marking as fixed’ is a great measure of just how much time it is taking to get reports resolved.

Perhaps just as important, though, is the ‘average time to first council state change’ – that could just mean the report has been acknowledged, or that its status has changed to ‘under investigation’ – but these are still valuable mileposts for keeping residents informed of progress.

Quick access to problems, as they’re reported. At the foot of the dashboard, there are links to all problems reported within the council boundaries.

There’s an option to filter them by any of the statuses, as above.

Access for multiple people, in different locations. The dashboard is web-based, so it can be accessed by any employee with internet access – or several at once.

But at the same time, complete security. It’s password-protected, so it’s only accessible to those who have been granted access.

A responsive provider. mySociety believe that the launch of new software is only the beginning of the story.

When people start using new products, they often do so in surprising ways. They often ask for features that would never have occurred to us, and indeed might never have previously occurred to them.

We will remain in active development, of the dashboard, and of FixMyStreet for Councils as a whole. We’ll be soliciting feedback, and listening to it very carefully.

The FixMyStreet for Councils dashboard is only available to councils as part of our FixMyStreet for Councils package – find out more here.

More specifically, we’ve been helping councils use user-centred techniques to kick-start the process of digital transformation: taking existing services that cause unnecessary frustration, figuring out how they should work for the customer in an ideal world, identifying the process changes needed, and helping make them happen.

How do you know where to start?

Most consultancies in this area will publicise their patented 5-step approach, or shower you in platitudes about talking to users and involving service managers, but I thought it would be more useful to walk through in detail what we actually do on a project like this. In this post, I’m going to describe only the first step (I’ll talk about others in future posts): given all the stuff that councils do, how do you know where to start?

Clearly, not every council service is susceptible to digital transformation. If you work in children’s services or benefits advice, your service is more likely to rely on cups of tea and conversation than on your website. But there are high volume transactions that involve exchanges of information or of money that do not, or rather *should not*, require any human intervention. Unfortunately, because of mistakes in how websites are structured and processes organised (that often go right back to decisions about management structure and procurement priorities), unnecessary demand is placed on contact centres.

What are your users trying to do?

So if you want to know what mistakes you’re making with your online presence, the first place you should look is the volume of calls to your contact centres and what questions the callers are asking. Here’s a complete list of all the places you can look for useful data on what your customers are actually trying to do and what you might be doing wrong:

Contact centre logs: the records of what people who call you are actually asking about. This is the best place to look to identify the areas where your web presence is under-performing.

Internal site search terms: the things people type in most often in the search box on your website. Generally speaking, use of search on a website is an indicator that your navigation and page structure have failed. Therefore the search terms people use on your site are another very interesting indicator of things you’re not doing well enough.

Referring search terms: the most frequently used search terms that drive traffic to your website. What are people looking for and what words have they actually put in to Google (or indeed any other search engine) for to arrive at your website?

Popular pages: data on the most frequently visited pages and sections of your website doesn’t tell you what you should improve or how, but it does give you a feel for where the demand is.

If you look at all of those things, you’ll have a lot of data to go through and make sense of. If you’re short on time, focus on the first one – it’s the juiciest source of insights.

Talking to service managers

Another approach we pursue in parallel to this one is to talk to a group of service managers and ask them for their opinions: if the decision on where we should focus our redesign efforts was up to them, what single thing should we start with that would make the biggest difference? How this actually happens in practice is that we get a group of people in a room together and ask them to write down (almost certainly on post-it notes) the top 3 – 5 services that they think are in need of a digital redesign. We then discuss and consolidate all of these before grouping them, trying to identify those that are the most susceptible to automation and where the complexity of the change needed internally is low enough to be approachable.

Decision time

The final part of figuring out where to start is to make a decision: which of these areas are you going to start redesigning first? You now have two sources of data on where to start: the results of your analysis of customer behaviour and the views of your employees who are closest to the action. Here we’ll make a recommendation, but leave the final decision to our council client: they know their organisation a lot better than we do.

With a focal point for the transformation efforts decided on, so begins the daunting-yet-exciting task of researching and designing the changes to be made: the bit where you actually talk to users, make prototypes or mockups of what the service’s digital touchpoints should look like (no specification documents here please) and then figure out together what process changes need to be made for it all to work in practice. Which, of course, are the topics for future blog posts.

This is the third of our recent series of Open311 blog posts: we started by explaining why we think Open311 is a good idea, and then we described in a non-techie way how Open311 works. In this post we’ll introduce our proposed extension to Open311, and show how we use it in FixMyStreet.

The crux of our suggested improvement is this: normal people want to know what has happened to their problem, and Open311 currently isn’t good enough at telling them whether or not it has been dealt with. To be more specific, our additions are all about reports’ status change, by which we mean something like this:

That pothole?

I just totally fixed it.

That’s robot-311 from the previous post, if you’ve dropped in here without reading the previous posts. Once again we’re blurring the distinction between client and user (the girl you’ll see below) a little, to make things simpler to follow.

Create→send→fix→update

Every month in the UK, thousands of problems are reported on www.fixmystreet.com and, moments later, sent on to the councils who will fix them. Here’s what happens with a problem report for something like a pothole or a flickering streetlight:

FixMyStreet sends that report to the right department at the right council.

That body puts it into its own back-end system.

Later, when the council fixes the problem, FixMyStreet is updated, and everyone knows it’s fixed.

On the face of it, you might think we need only care about 1 and 2. But really, FixMyStreet isn’t just about dispatching reports, it’s about helping to get things like potholes actually fixed. And neither citizens nor local governments benefit if work gets done but nobody finds out about it – which is part 4 on the list above.

What do we mean by “status change”?

The example at the top of the page shows the robot effectively changing a problem’s status to “fixed”.

Actually, statuses can be simple, such as either OPEN or CLOSED, or more detailed, such as “under investigation”, “crew has been dispatched”, “fixed”, and so on. But since we’re only concerned here with the status changing, that specific vocabulary deployed doesn’t really matter – it can be anything.

In situations where FixMyStreet is not integrated with council systems (i.e we just send email problem reports) FixMyStreet problems still frequently get marked as fixed, because anyone can change the status of a report just by visiting the page and clicking the button. Obviously, though, we prefer to have FixMyStreet directly connected to the local government back-end databases, so that news of a fixed report can be automatically bubbled from the back-office up into FixMyStreet and out onto the net.

And here’s where the problem lies: Open311 doesn’t quite support this business of getting problem updates from the back office out to the public. So first, we’ll show you how it can be done today, using Open311, and we’ll explain why this isn’t a good option. Then we’ll show our preferred solution, which we’ve proposed as an extension.

Looking at everything just to spot one change (bad)

One way to notice if any problems’ statuses have changed is to use Open311 to ask for every single service request, and see if any of them have a different status since the last time you checked.

Tell me all the service requests you’ve ever received

OK:

request 981276 the pothole on the corner by Carpenter Street is now CLOSED (I filled in the pothole)

request 988765 the pothole by bus stop on Nigut Road is now CLOSED (I filled in the pothole)

request 998610 gaping hole at the end of Sarlacc Road is now OPEN (the pothole fell through)

request 765533 where the street was cracked outside Taffey’s Snake Pit is now CLOSED (I filled in the pothole)

. . .

continues for thousands of requests

Um, OK. Now I’ll look at all these and see if any have changed since I last asked *sigh*

Obviously there are some problems with this. Even though Open311 lets you ask for quite specific service requests, you have to ask for all of them, because by definition you don’t know which ones might have changed. Remember, too, that problems can potentially change status more than once, so just because it’s been marked as CLOSED once doesn’t mean it won’t become OPEN again later. This exchange is very wasteful, very slow and ultimately (with enough reports) may become de facto impossible.

Asking for just the changes (good)

So here’s a better way of doing it. We’ve actually been doing this for some months, and now seems the time to share.

The client asks the server for just the updates on a regular basis, so any requests that have recently changed get updated on FixMyStreet automatically, usually just a few minutes later.

Have you changed the status of any of service requests today?

Yes, request 981276 was CLOSED at 3 o’clock (I filled in the pothole)

Or, more practically for keeping FixMyStreet up to date:

Have you changed the status of any of service requests in the last 15 minutes?

Nope.

This is handled by our extension to Open311, GET Service Request Updates. There’s also an optional equivalent call for putting updates into the server (POST Service Request Update), which would apply if the client changed the status after the service request had been submitted.

Note that the server identifies the problem with its own reference (that is, 981276 is the council’s reference, not a FixMyStreet ID, for example). This is important because not all these requests necessarily came from this particular client. Remember that all service requests are available through the Open311 GET Service Requests call anyway (as shown above). So the server doesn’t send each service request back in its entirety: just its ID, the new status, when it changed, and a brief description.

In practice the client wouldn’t usually ask for “today”. In fact, we typically send a request asking for any updates in the last 15 minutes, and then at the end of the day ask for the whole day’s updates, just to check none were missed.

The technical bit

From a client’s point of view, this is simply an extra call like others in the Open311 API. So it’s just a request over HTTP(S) for XML (or JSON, if required).

We deliberately make the client poll the server for updates and pull them in, rather than expecting the server to push updates out. This frees the server from any obligation to track which clients (for there may be more than just one) care about which updates. The requests themselves are sent with unique IDs, allocated by the server, so the client can dismiss duplicates. It’s also robust in the event of connection failures, so if there are timeouts or retry logic, that’s for the clients to worry about, not the server. Basically, this is all to make it as light on the server as possible: the only real issue is that it must be able to provide a list of updates. This usually means adding a trigger to the database, so that when a problem’s status is updated a record of that update is automatically created. It’s the table of those “service request update” records that incoming requests are really querying.

Is that it?

Yup, that’s it.

This extension is in addition to the Open311 specification — it doesn’t break existing implementations in any way. Obviously this means FixMyStreet’s Open311 implementation is compatible with existing Open311 servers. But we hope that others working on Open311 systems will consider our extension so that clients are kept better informed of the status of the problems being fixed.

Why are statuses so important that it is worth extending the Open311 spec?

mySociety didn’t originally build FixMyStreet because we wanted to get potholes fixed. We built it because we wanted nervous, politically inexperienced people to know what it felt like to ask the government to do something, and to be successful at that. We wanted to give people the buzz of feeling like they have a bit of power in this world, even if the most tiny amount.

If the government fixes a problem and the citizen doesn’t find out it’s a double loss. The citizen becomes disillusioned and weakened, and the government doesn’t get the credit it is due. Everyone loses. We think that Open311 is a key mechanism for making large numbers of people feel that the government does respond to their needs. It just needs a bit of an upgrade to do it better. We hope very much that the wider community tests and endorses our extensions, and it can be folded in to the next official version of the Open311 standard.

In the previous blog post we explained why we think Open311 is a good idea. In this post we’ll explain what it actually does.

Open311 is very simple, but because it’s fundamentally a technical thing it’s usually explained from a technical point of view. So this post describes what Open311 does without the nerdy language (but with some nerdy references for good measure). At the end there’s a round-up of the terms so you can see how it fits in with the actual specification.

We’re using an unusual example here — a blue cat stuck up a tree — to show how applicable Open311 is to a wide range of problems. Or, to put it another way, this is not just about potholes.

So… someone has a problem they want to report (for this discussion, she’s using a service like FixMyStreet).

There’s one place where that report needs to be sent (in the UK, that’s your council). That administrative body (the council) almost certainly has a database full of problems which only their staff can access.

I have a problem :–(

the “client”

I fix problems!

the “server”

In this example, FixMyStreet is an Open311 client and the council is an Open311 server. The server is available over HTTP(S), so the client can access it, and the server itself connects to the council’s database. In reality it’s a little bit more complicated than that (for now we’ll ignore clients that implement only part of Open311, multiple servers, and decent security around these connections), but that is the gist of it.

Although it’s not technically correct to confuse the client with the user, or the server with the council, it makes things a lot easier to see it this way, so we’ll use those terms throughout.

Service discovery

To start things off, the client can ask the server: what services do you provide?

Until the client has asked the server what problems it can fix, it can’t sensibly request any of them.

FixMyStreet can use the response it gets from such a service discovery to offer different categories to people reporting problems. We actually put them into the drop-down menu that appears on the report-a-problem page.

In the Open311 API, this is handled by GET Service List. Each service has its own service_code which the client must use when requesting it. Note that these services and their codes are decided by the server; they are not defined by the Open311 specification. This means that service discovery can easily fit around whatever services the council already offers. The list of services can (and does) vary widely from one council to the next.

Service definitions

Some services require specific information when they are requested. For example, it might be important to know how deep a pothole is, but it’s not relevant for a streetlight repair.

Tell me more about the PET service!

I can get pets down from trees, but when you request the service, you *must* tell me what kind of animal the pet is, OK?

In the Open311 API, this is handled by the GET Service Definition method. It’s not necessary for a simple Open311 implementation. In fact, it only makes sense if the service discovery explicitly told the client to ask about the extra details, which the server does by adding metadata="true" to its response for a given service.

Requesting a service

This is where it gets useful. The client can request a service: this really means they can report a problem to the server for the body to deal with. Some submissions can be automatically rejected:

My hoverboots are broken :–( I need BOOT service!

404: Bzzzt error! I don’t fix hoverboots (use service discovery to see what I *do* fix)

Hey! Blueblue is up a tree! I need PET service (for cats)!

400: error! You forgot to tell me where it is.

If the report is in good order, it will be accepted into the system. Open311 insists that every problem has a location. In practice this is usually the exact position, coordinates on planet Earth, of the pin that the reporter placed on the map in the client application (in this case FixMyStreet.com).

I need PET service (for cats)! Blueblue is stuck up the biggest tree in the park :–(

200: OK, got it… the unique ID for your request is now 981276

In the Open311 API, this is handled by POST Service Request. You need an API key to do this, which simply means the server needs to know which client this is. Sometimes it makes sense for the server to have additional security such as IP address restriction, and login criteria that’s handled by the machines (not the user).

Listing known requests

The server doesn’t keep its reports secret: if asked, it will list them out. The client can ask for a specific report (using the ID that the server gave when the report was submitted, for example) or for a range of dates.

Did anyone ask you for help yesterday?

Yes, I got two requests:

request 981299: TELE dirty teleport at the cantina (I’m waiting for a new brush)

request 971723: POT pothole at the junction of Kirk and Solo (I filled it in)

In the Open311 API this is handled by GET Service Request(s). The client can indicate which requests should be listed by specifying the required service request id, service code, start date, end date or status.

Does Open311 work?

Oh yes. On the Open311 website, you can see the growing list of places, organisations, and suppliers who are using it.

The technical bit

In a nutshell: Open311 responds to HTTP requests with XML data (and JSON, if it’s wanted). There’s no messing around with SOAP and failures are reported as the HTTP status code with details provided in the content body.

You can see the specification for Open311 (GeoReport v2). It doesn’t feature blue cats, but if you look at the XML examples you’ll be able to recognise the same interaction described here. And remember the specification is an open standard, which means anyone can (and, we think, should) implement it when connecting a client and server in order to request civic services.

Coming next…

In the next blog post we’ll look at how FixMyStreet uses Open311 to integrate with local council systems, and explain why we’re proposing, and utilising, some additions to the Open311 specification.

mySociety

We are a not-for-profit social enterprise. We are based in the UK, where we run a number of projects designed to give people the power to get things changed; we also work internationally to support partners who deploy our technology in countries around the world. This work is partially supported by providing software and development services to organisations that can benefit from our experience in civic technologies.