Tuesday, September 27, 2005

Building it Front-to-Back

Wow. I wish I'd tried this sooner. Conventional wisdom in software development says that first you design your architecture, then you lay the foundation, build the back-end, and somewhere near the end of your project, features start to bubble up through the user interface (UI). Maybe it's because we keep comparing software development to construction, where first the foundation goes in, then the framing, and the color of the paint and countertops doesn't even need to be decided until the very end. I'm not sure.

How many times have you experienced this: A gaggle of developers working feverishly for months, all the while, stakeholders asking, "Can we see it yet?" "Nope.", comes the reply, "We're still building the system that the UI is going to sit on top of." Then, one day, "Tada! Here's the UI!" "But that's not what we wanted!" Oh oh. (Or best case, the UI is functionally sound, but looks like ass because you're too pressed for time to polish it up and make it esthetically appealing and ergonomically friendly).

I never really thought to question it before. Not until I had the complete freedom to try it a different way, without having to fight to convince anyone that doing things in reverse might actually be a better way. I'm building a large system front-to-back right now (user interface to data storage), and am thrilled with all of the benefits this approach is offering.

Most people in development know that the last things on the project schedule are the things at most risk of being cut in a time crunch. Most development projects see the bulk of the user interface being developed after all the system work has been done. But the user interface is the only thing the customer sees - it's what they buy. See the problem? You're leaving the most important thing to last. This is way too risky, for you and your customer.

The idea that you can reduce an application design down to a bunch of nicely written Word documents is nice, but not realistic. Try and try again, with this approach, it'll never match having a living breathing example (UI) of what you're building, while you build all of the system architecture level things that no-one else gets to see but us developers.

Not that you shouldn't spend a really sizable chunk of time designing and building your architecture. (See my last post on architecture). But rather, the architecture should be derived from a living mockup, or UI design. Not just screen caps (not enough), and not necessarily a full prototype (too much, and too likely to get thrown into production), but something in the middle.

Here's the steps I followed with my current project (a web-based modelling tool, built to be hosted ASP style; = large scale deployments):

Started in Excel: 'cause it's a data modelling type tool, so I wanted to layout the main tables and views that people would have available to them

Moved to Photoshop: to put some lipstick on the pig and make it look pretty (don't forget, esthetics sell)

Got feedback on the concept - Was the concept right? How does it look? Good. Ready to proceed.

Moved to HTML/CSS/Javascript: This step might be unique to me, but I really wanted to have the richness of the desktop in my web-based application (drag/drop, inline editing, etc.), so I built a bunch of re-usable, cross-browser Javascript controls for this and demonstrated them on a few sample pages of the application

Got feedback on the interactivity - how does this behave compared to regular web-apps? What do you think? Good. Ready to proceed.

Moved back to PowerPoint: I wanted to mock-up every single screen in the application so that I could get feedback and make any changes quickly without having to rip-out months worth of coding. PowerPoint is great for UI design because you can throw in image pieces and move them around with text and rectangles quickly. And with one-slide-per-screen, you can still have 60 screens in one file, with annotations, printer-friendly, and sharable.

Implemented each screen in HTML/CSS/Javascript: At this point, I took the time to do some "real" development. Not hacking, not prototyping, but the real take-the-time-to-do-it-right design and development, but ONLY for the UI. The goal was to have every screen built to do nothing but show dummy data and move around to the other screens.

Got feedback - This was the single most important feedback stage, and allowed me to get the detailed kind of comments that would only otherwise be available at the end of a development cycle (when it's too late), because the application "appeared" to be complete, from the user's perspective - just that it was all dummy data. At this point, beyond looking at screen caps, they could actually click around and "use" it.

Business Logic and Back-end: Now things start to look like a normal development cycle from here-on-in. Do your architectural design, but this time with a living specification staring you in the face (one that the customer has actually approved), and proceed to implement it.

With this approach, I've pretty well eliminated the risk that the product won't match customer expectations because they've actually been able to see it and play with it before I go do the lion's share of the development. Not only that, but now that I'm building the back-end, while I still want to build a somewhat future-proof architecture and be fairly generic with it, I have an extremely specific, living example of the application I ultimately want to build with it.

Some might call this rapid prototyping, but there are a couple important differences:

Only the UI was mocked up: do we really need to demonstrate that we can read and write from a database? No. Skip that stuff. Just mockup the screens and connect them so the user can get a feel for the navigation. Populate the screens with dummy data.

There was nothing "rapid" about it: while the PowerPoint mockups were done quickly, they could also be changed quickly. When it came time to move the screens into HTML/CSS/Javascript, it was solid development practices the whole way. Not prototyping, not throw-away, but solid design and re-usability. This was production UI code.

While the goal of this approach was similar to one of the goals of Extreme Programming (get the customer in early), I think the approach is stronger because while you get all the benefits of the user being involved early on to validate what you're doing, you don't have to sacrifice the benefits of solid up-front design for each tier (UI, Biz Logic, Data Storage), by making it up as you go along and trying to convince yourself you can re-factor quickly.

I can't tell you how useful this front-to-back approach has been. Virtually all of the requirements and design unknowns have been squashed, and makes building the back-end a dream. Descreased risk, increased predictability, increased quality.

This approach definitely took more time upfront than a "document only" approach to design, but I believe the time saved later, and the risks mitigated make it an exceptional investment. I would highly recommend it to anyone.

7 Comments:

Hi Craig, I like the idea behind what your saying. It certainly seems to make sense. I do have a question though. How do you plan on re-socializing the customers who expect to work from the old style "requirements document". This approach seems to require a cultural change not only on behalf of the development team but also on behalf of the client

Good question. It definitely requires cultural change on the part of the team/company developing the product. That's the hard part. I wouldn't suggest that you should jump into building a UI before you've even tried to put requirements for the product down on paper, but rather that there is an intermediate step between requirements and "build the whole thing at once." That step is the building of the UI for sign-off before the bulk of the back-end gets built.

As for change on the customer side, yes that may require some cultural change as well, but I don't think that customers are usually the ones saying "we want you to produce a big shaggy requirements document." They'll give you requirements in a variety of ways, from ad-hoc conversations, to napkins, to PowerPoints, to group meetings. It's usually the producing company that then tries to put everything into one doc in order to manage the chaos, get a contract done, feed the dev team, etc.

Traditional "requirements/design docs only" approaches are the vendors' way of steering the ship. (And with the experience they have producing software, that's a better alternative to letting the customer completely steer the ship).

Some people following the XP (Extreme Programming) methodology are trying just the opposite, they're trying to let the customer steer the ship and absolve themselves of any responsibility, with extreme practices like "customer works on-site with you in your office".

What I'm advocating is a balanced approach: Do your homework upfront (requirements/design), but manage expectations by getting the UI done first, and make sure you've got that right before biting off the remaining 6 months of development work. That way you and the customer are on the same page, but you're still steering the ship.

Craig, You've hit the nail on the head and pretty much laid out all of my frustrations with this article.

Things are starting to move more towards design driven development, but we still have a long way to go before we, as designers, get true satisfaction.

I'm am astounded by the fact most software projects still think design is something that is tacked on at the end if time permits. Design is much more than a Photoshop comp and a few css rules. It needs to iterate along with the development, not lumped in at the very last.

I am a developer also, so I've been using the process you've outlined for internal projects and a couple of others I'm now working on.

I don't think you'll have to do a lot of convincing to get clients to understand what it is your trying to do. I find it good to discuss screens and basic processes and then map out the UI.

Once you have a clear understanding of what screen X does, how it fits in with the flow of the application you then create the UI for it. When you can present clients with what they'll actually be seeing instead of a bunch of jargon full documentation you'll certainly get better feedback.

Design is not just for whats on the screen, it's where you go from this screen and how you get there. Design for the process, not just the page.

Most of the time the designer is brought in way to late in the development cycle to really be effective. When your brought in this late all you can really do is polish what's already there.

People need a serious wake up call so they can realize how important a designer and the design process is to a software project. After all, I don't know many people who's first thoughts are: "Boy this multi-threaded intelligent query caching mechanism made this application well worth the money spent!".

i see the benefit of design driven development but it mostly seems like to work if your client is reachable.

what if you are developing an application that will target an unreachable audience before you launch your product? say for example, a web application like flickr,etc ? any ideas on that kind of development process?

Good point. It's not as obvious if you're developing a mass marketed product, where no single company is commissioning your work. That's actually the situation I'm in right now (developing a mass marketed product), and the process works exactly the same. Instead of necessarily having a handful of real customers in that case, you can assemble a group of people from your intended target audience and interact with them exactly the way you would a customer. They can be the voice of your future customers and give you all the same feedback if you choose them wisely.

Depending on how much time and input you need, you may have to offer them something of value for their time, but usually a discount on the end-product or maybe a free license for their troubles will do. Most people are quite willing to sit down for coffee and let someone pick their brain for a while. Everyone loves to give their opinion.

In fact, this approach (building front-to-back) may be even MORE important for mass marketed products, because you're not actually working off of a set of requirements from a client - you're the inventor and believe you have found something that the market will want. It's critical that you test your idea with your target audience first!

As a software developer, this is what I've always liked about design and usability work: I can code with confidence, knowing the bugs have already been wrung out of the UI. Our software department has mostly adopted this approach enthusiastically for the same reason.

Tog has a great article about this -- scroll down to "how user testing saves money."

About

I can't believe that 70% of software projects fail. It's embarrassing. Something is fundamentally broken here. Left to their own devices, people have a way of over-complicating things. It really doesn't have to be that hard. The trick is knowing what to look for and what key factors to manage. The rest is noise.

About Me

Hi, I'm Craig. I'm currently building a company called Devshop.com. Devshop has a new approach to bringing software in on time, and therefore helps eliminate the total cost of late software.
I started coding when I was knee-high to a grasshopper and I have worked for 5 different software companies in the last 12 years, as head of product development.
In early 2004, I became very frustrated with the fact that general purpose project management methodology is not working for the software industry. My personal belief is that to solve this problem, we need to start focusing on specific types of projects, rather than relying on age-old one-size-fits-all practices and techniques.