I saw an entry posted the other day where someone said sie was disappointed with (among other things) DW's development pace slowing down: new features being released more slowly, things that we were working on delayed/postponed, etc. And there were totally some valid criticisms in there, don't get me wrong! (In fact, I'm not linking it because I don't want there to be an overwhelming impassioned defense of DW in hir comments.) But that's one criticism that made me realize I've been doing a poor job of explaining precisely what's been going on in DW development and why there's been a paucity of user-facing changes, which can look to an outsider like there's a massive slowdown going on in DW development.

The answer is at once both very simple and very complicated: we've spent the past six months or so concentrating on paying down our technical debt.

Technical debt, as detailed in that Wikipedia article, is the collective IOU you-the-developer write to your future self. Let me just do this fast now, and do it right later, you think: let me do the quick but messy way now and do the correct way later. But of course, the minute you ship, you're moving on to the next big thing, and you never do go back to fill in those FIXMEs and TODOs in the code, until the next time you have to touch that area and what would otherwise be a five-hour fix has turned into a twenty-hour fix because you have to "pay back" the debt you incurred last time.

More than that, you've got to pay the debt plus interest: it's a common truism that code you yourself wrote six months ago is as impenetrable as code written by a complete stranger, and you have to spend a great deal of time puzzling out what the heck you were thinking back then. (Code that is brilliant, flawless, and crystal clear when you write it slowly morphs into idiotic, bug-ridden, and clear as mud over time. This is a well-known process. I suspect pixies in the source code repository, working their anti-magic while nobody's looking.)

Now, like real-world, financial debt, sometimes there is prudent technical debt: sometimes it makes sense to incur those future obligations in order to accomplish something you otherwise couldn't have. We saw a lot of instances of that during the ramp-up to open beta, where we did it one way then to "get it out the door" and opened bugs for later refactoring so we didn't lose track of the debt we were incurring. (There's a good article about the technical debt quadrant that discusses these various types of technical debt, and when they might be useful.)

There's also scenarios where you don't realize you're committing to technical debt until later: you do something one way based on your best understanding at the time, and down the road -- due to new technology becoming available, new people joining your team with new strengths that allow you to work on things you couldn't before, new sources of funding appearing, new progress in hardware capabilities, etc -- you realize, oh, hey, we could have done it this way instead, and it would have worked much better.

In short, there's no way to completely eliminate technical debt in software design, and more than that, you wouldn't want to even if you could. Technical debt, like the responsible use of revolving lines of credit or obtaining a mortgage to buy a house in the real world, is an important part of the software lifecycle, and responsible use of technical debt is a tool that can enable a software project to succeed where they otherwise might have failed.

In a 'typical' two-year-old software project, the amount of technical debt you've built up -- if responsibly managed, which I like to think we've been pretty good at with the things we've made so far -- is fairly negligible, and you can probably get away with spending 10-20% of your coding time and effort to making payments on your technical debt without problems, thus leaving you 80-90% of your time to advance new user-facing features and improvements.

The problem is, Dreamwidth isn't a two-year-old software project. We've only been open for two years, true -- but when we opened, we forked from LiveJournal, which sprang into life in Brad Fitzpatrick's dorm room in 1999. In forking our code from LiveJournal's, we inherited that decade's worth of features, fixes, and improvements, but we also inherited a decade's worth of deferred technical debt. Dreamwidth isn't a two-year-old software project; it's a twelve-year-old software project. And the face of technology has changed quite a bit in those 12 years.

In the decade LJ was under development before we forked our code from theirs, there were of course payments being made on that technical debt; it's necessary in order to move forward. Really big payments, though -- the technical equivalent of paying down your credit card in one lump sum because you've come into a windfall -- were mostly deferred.

So, one of the things we need to do in order to move forward in a lot of instances, to take advantage of the advances that have been made in technology since various features and bits of site design were first coded, is to do all the work necessary to get to a place where we can take advantage of them.

I'll give you an example: when we forked from LJ, the code "out of the box" would only work on the Apache 1.3 series of web server software -- and early releases of 1.3, at that. (The web server is the program that runs on your machines and handles how to serve web pages to your browser when you ask for them -- if you don't have a web server running, nothing else works.)

The first release of Apache 1.3 was June 6, 1998 -- it was cutting-edge when Brad started LJ in 1999. The latest version that the LJ code would work with was (I believe -- I may be wrong) 1.3.17, released January of 2001. Subsequent versions of Apache wouldn't work with the LJ code -- it would cause horrible errors and prevent the site from running at all. To upgrade to later versions of Apache (which had more features, fixed security holes, and in general were more technically advanced) would have taken a lot of work on the code.

Even at that time, Apache was working on Apache 2.0 -- a much more technologically advanced version. The first 2.0 release was in March of 2000. The problem was, it was mostly incompatable with the optimization tricks used under Apache 1.3, and LJ code was highly optimized to take advantage of the 1.3 series. To port LJ over to the Apache 2 series would take an incredible amount of effort -- aka, technical debt.

By the time we forked the code, in mid-2008, Apache 1.3 was nearing the "end of life" -- the point past which Apache would refuse to support it, refuse to issue any additional updates, and generally say, look, c'mon, it's been nearly a decade, upgrade already. (It was formally EOL'd in February of 2010, but everyone agreed it was well past time.) So if we accepted the requirement for Apache 1.3, we were already tying our hands, and making it incredibly hard for our developers: the Apache 2 series had been the standard since at least the middle of the decade, and if someone wanted to run DW, either as a production website or to do development work, they would first have to spend hours actually downgrading their server in order to make the code work.

Before you guys even saw DW, before we could even get to the point where people could install and start hacking on the code, mark had to spend months making the code work under the Apache 2.0 series -- he had to sit down and pay the technical debt. The same thing happened with Perl, the language DW is written in. LJ had been written in an earlier version of Perl, and later versions had some backwards-incompatabilities; DW had to modernize before we could upgrade.

The things we need(ed) to do aren't all that obvious, of course. Those of you who came from LJ know that each page you see on LJ has the suffix ".bml" on the end of it. BML stands for "Better Markup Language" (or "Brad's Markup Language"), a templating system that produces the framework to generate a page. (Things like the site skins --Tropospherical, Celerity, Gradation, etc -- are a function of BML; the contents of the page are the same no matter what, but the templating system builds the various 'looks' of the page so you can just swap them in and out and the contents don't change but the display does.)

In 1997-1998, when Brad was working on FreeVote (his project before LJ) and the first iterations of what would become LiveJournal, there wasn't anything better out there, so he had to "roll his own". He got it to a place where it would work for LJ, and then -- because there wasn't any real need to advance things further, since it already did everything he needed it to do -- mostly stopped work on it. The face of the web changed a lot since that point (this may be a wee bit of understatement), and today there are a lot of incredibly powerful templating systems out there that do way, way more than BML does -- and are under active development, so there will be future awesomeness coming out of them.

Additionally, making people learn BML -- which is a fairly impenetrable system -- in order to contribute to DW would be silly -- it's a barely-documented custom language that only exists on less than half a dozen websites in the world. The other templating systems out there are in wide use, and it's way more likely that someone will already have the skill set necessary to contribute. (Not to mention, it'd be nice to have a templating system that has actual books written about it rather than a few web pages here and there.) Switching to a more standard version makes an incredible amount of sense.

It's also an incredible amount of work. (We've been plugging away at it, bit by bit, for over a year. At this point, the pages you see on DW itself are half generated by BML, half by Template Toolkit (the templating system we chose) -- you never see the difference, if we do our work right, but our end goal is to get rid of BML entirely.)

There are a lot of examples of things like this, from the "big project" issues to smaller things (like the need to take duplicated code -- where the same block of code is repeated in multiple places -- and move it into a function that can be called from anywhere instead, so that people only have to update one area instead of many). There are cases where functions and features that were revolutionary when they were implemented on LJ have aged over time while the technology has advanced in leaps and bounds -- a good example there is the implementation of the inline cut-tag expander; when "lj cuts" were first introduced on LJ waybackwhen, that technology didn't exist, or would've been too much of a pain in the neck to implement. Over time, it became easy; from becoming easy, it became expected, until the point where a site that didn't implemented it started to look clunky or backwards.

People have come to expect a lot of things from their social websites, in terms of 'standard' technical abilities -- I'm sure you can think of about a dozen things that other, more recently-designed websites implement that DW doesn't. We'd love to have those features. (I probably curse their lack about twice a week.) The problem, again, is that the backend code for those pages was written so long ago that before we can drop in those features and functions, we have to modernize everything. You'll never see the work. But we have to do it before we can move forward.

Some people have asked us why we committed to the LJ platform when we knew we'd be accepting the IOUs of ten years of the programmers who came before us. The answer is twofold:

1) In many cases, we were those programmers. We knew the code, knew what it could and couldn't do, had a reasonable perspective on what we were in for, and were incredibly familiar with the way it worked, the way it ran, and the way it was put together. (I say 'we', but I mostly mean mark there. I didn't get much into development until we started DW. But I still followed along with the technical discussions, and I had a pretty strong grasp of the technical end of things even though I hadn't been doing the coding myself.)

2) In addition to the ten years of technical debt, we also inherited the benefit of ten years of bugfixes, security fixes, architecture/performance improvements (in another 10 years when historians write the history of the early 2000s on the internet, I fully believe they will point to LJ as the technical pioneer that made a very great deal of Web 2.0 happen; the problems LJ solved back then are universal to any high-load system, and the solutions they/we came up with are still in use today), and feature development.

We believed, and continue to believe, that the LiveJournal system and code contains some of the most incredible social features out there, to the point where even today, ten years later, there is no other site that does everything the LJ code does and does it as well. A lot of that amazingness is buried, now, under a lot of "usability problems" that are actually relics of the fact that nobody went back to modernize things once the first draft was released. (To be fair, there are tons of usability problems that are actual usability problems, and were at the time the feature was released, as well. And I don't want to intimate that LJ-now is ignoring these problems either; they've been doing a lot of work on their own technical debt lately, as evidenced by the number of people who accuse them of not working on any new features either.)

One of our major goals with DW is to take the awesomeness that is inherent in the LJ codebase and bring it into the "modern era" of web design and function. We've made some great strides, but we're still only part of the way, and every time we set out to do something new, another whole chunk of problems that we have to address first pop up. It's the technical equivalent of having to learn to crawl before you can learn how to walk: before we can complete the new update page redesign, for instance, we need to do an incredible amount of work in order to make it possible to intermingle JQuery (the Javascript library that we need for the modernized widgets on the update page) with the existing Javascript the site uses. (Among many other things.)

So, when you see a code tour that's full of nothing but backend improvements, don't think of it as "DW isn't doing any feature development". Think of it as "DW is doing the necessary background work to enable awesome feature development in the future". The work we're doing now is going to pay off in the future, and it's going to allow us to do epic things.

no subject

Thank you. I have been quite content...but I'm no developer. I am comfortable here, and if you say the back end needs cleaned up, great. But in taking the time to explain, so that even a non-tech like me can get it? Thank you.

no subject

You're not alone -- there are a lot of people who dislike change (hell, I'm still using a multichat client that was EOL'd in like 2005 or so, because I hate the user interface of every alternate client I've tried with the burning passion of a thousand suns), but I do honestly think there are ways to improve and modernize things without a) pissing your userbase off, b) screwing things up ten ways to sunday, or c) making it worse than it was when you started. It all revolves around making sure that the changes are done in an organic way, that they add value rather than making things new for the sake of making things new, and carefully observing the actual behavior of your users instead of what you would like that behavior to be in an ideal world.

I mean, there are tons of things about DW (and about LJ, since we inherited a lot of them) that are a usability nightmare. We don't notice them, for the most part, because we've been using them for ages, so we've trained ourselves how to make them work -- but someone new looks at it all and runs screaming! Our overall goal is to improve those usability disasters, while at the same time not losing what makes things so special, you know?

Funny you should mention it!

I've spent a month looking at shopping carts to replace my ancient, delaminating, often hand-coded OSCommerce site. When I look at the primitive state of PHP language back then I must have been hitting it with rocks and grunting at it to make it work. I can't do that now. And I have experienced that thing with the incomprehensible code that I myself wrote once on a time.

(I'm working with a developer to create a craftsperson's cart for wordpress)

Don't do those social network things, please? If there were one thing I would ask you for-- it would be to implement a page or series system for the fiction writers, so that they could link chapter posts together outside of the chronology. If you did that, I would totally transfer my writer's blog and archive to DW. (Are URLS implemented yet?)

Re: Funny you should mention it!

I really do believe there are pixies that live inside the computer and move around the code while we're not looking. It's the only possible explanation.

I got into it a bit with ilyena_sylph up there, but there are basically a lot of usability "hot spots" -- things that we, who have been using the site for ages and ages, know full well how to operate, but someone new would have no idea. And they take one look at it and go screaming, and wind up on another service that does things better (I'm studying Tumblr pretty closely right now, for instance, since it's the new up-and-coming place that a lot of people are heading to.) So we need to improve usability enough and to modernize enough that people won't dismiss us straight out of hand, because a service can't live just by being geared towards people who are disaffected with LJ -- we need to be able to attract and retain new users from elsewhere.

Still, I totally believe that there are ways to modernize without destroying what makes DW special, and without annoying people to death!

"Pretty" URLs aren't implemented yet, but it's in Bugzilla, and I've been poking at it on and off and wondering if I have the skillz to accomplish it. (Right now, I don't think I do, but hey, I work best when I'm working just two steps out of my comfort zone.) We also have a bug open for "index pages", which will be sort of like a mega links list, where you can create a meta-page that links to a bunch of other entries, which would work quite nicely for what you're talking about with the chapters thingy -- I've been wanting that for a while!

no subject

And yeah, the Apache 1.3 dependency was the first thing we did (it sort of had to be) -- down the road, LJ even used our implementation of it as a reference for their own work. (They got off the 1.3 tree somewhere in 2009, I think.) It was painful.Then there's JQuery-ifying the site, and converting to Template Toolkit, and half the code wasn't even OO, and just, ugh. Ten years of deferred payments -- and that's just on the backend; I'm not even going to get into the mess that half the frontend is in.

Sometimes I look at all of it and get totally overwhelmed by just how much we have left to do, but then I stop and make myself look at how far we've come. We've actually diverged from LJ so much that we can't adopt their patches wholesale anymore; we need to convert them to work on our end. If that's not a sign of how far we've come, I don't know what is!

memcached: man, some of the things Brad came up with were total duds, and his code is occasionally a bunch of sheer wtf, and rolling your eyes and mocking Acts of Brad was a common pastime even back in the LJ office of 2004, but in his own way the guy is a genius, and the things he came up with in order to keep LJ running on a shoestring from 2002-2005 or so, before Six Apart could just throw more hardware and people at the problems, were occasionally works of art. I learned more about architecture from watching him than I ever realized. I really do hope that the historians of the future give him the credit he really deserves.

no subject

None of this is news to me because I am a Speshul Fucking Snowflake who learns by osmosis and absorbed a lot of these concepts while very young from my father, but it's always nice to see someone lay it out clearly for the layperson. This is well written and communicates the concept of technical debt pretty simply for people who have no desire or need to understand software engineering in any kind of depth.

I've been really excited to see all the refactoring and modernising of code and I had kind of forgotten that it might not look very exciting to most people.

Anyway! Yay this entry and I'm glad you're going to go into this in a little more detail in the next news update than have gone into past updates.

no subject

It was just this morning I was reading about "mature software environments" in a novel--and the description in the novel made it sound like an environment where they never did pay off their technical debt, just layered over it with other stuff. :-)

no subject

That is so true! My employers are attempting to develop a 'service oriented architecture' across the organisation, and it's like the layers of an onion: shiny RIA and mobile clients/UIs (2010's tech), then a lot of Java and webservices (2000's tech, buffed up by IBM), then as you dig deeper you find the C++ code from the 1980s and 1990s, and then at the bottom it's all still COBOL.

no subject

Code that is brilliant, flawless, and crystal clear when you write it slowly morphs into idiotic, bug-ridden, and clear as mud over time. This is a well-known process. I suspect pixies in the source code repository, working their anti-magic while nobody's looking.

Yes! Darn them. They get into everything!

This is why I like writing flippant comments in my code. It reminds me of my mindset at the time I was first writing it and helps me look at things the right way to pick up and work on it again. Yes. That's my story and I'm sticking to it.

no subject

Ayup. It's a big part of the reason the project I'm currently working on is 51% comments, 17% whitespace, 34% code (in the source/script files; not counting the explicitly-documentation files). Okay, part of it is that I babble when I get tired, but most of it is that I want whoever tries to modify it next to be able to make sense of it without the ability to read my mind, so it gets edited instead of ripped-and-replaced ... especially if that next person is just me again in a year. There are fewer "WTF were they thinking?" moments when the comments explain what you were thinking, why you went one way instead of another.

Unless the code pixies f$#% with the comments as well, of course, in which case, well, I'm screwed.

no subject

My father has the worst 'code come back to haunt me' story I've heard yet.

He worked for the electric power company in Akron, OH in the late 1950s, where he wrote some Fortran to help analyze power distribution networks. After about five years, he left that job and moved to Duluth, MN, where he went to work for a different electric power company.

Fast-forward to sometime in the mid-80s. Dad is working on distribution planning (how much power are we going to need to provide where in 5, 10, or 20 years, and what do we have to do to our network to meet those needs), and needs to some specialized network analysis. He goes rummaging into the software library, and finds a program that should do just what he wants, with a little tweaking.

The specific section he needed to modify was about 30 years old. According to the changelog, it was untouched since it had been written. He looked at it, went 'I remember writing this code, and I haven't the faintest idea how it works'. He tore the whole section out and re-wrote it from scratch.

Yeah, I intuitively understood the concept of technical debt pretty much the moment you used it.

no subject

Thanks for this explanation-- it was a really helpful behind the scenes look at development of projects like this. Good to think about when I see how much technical debt we're having to chip away at in my day job as well.