Tag: refactoring

Take a look at the code you work in day-to-day. How long has it been there? How old is it? Six months old? A year? Maybe five years old? Ten? Twenty?! How much of the code is old? Less than 10%? Half? Or as much as 90%? Curious to know the answers to these questions I’ve been investigating how long code sticks around.

Software archeology

Work for any company that’s been around for more than a couple of years and there will be source code that’s been there for a while. Writing software in an environment like this is often an exercise in software archeology – digging down into the application is like digging down into an old city, slowly uncovering the past layer by layer.

Once you get past the shiny new containerized micro-services, you start delving into the recent past: perhaps the remnants of the company’s first foray into a Service Oriented Architecture; now a collection of monolithic services with a complex tangle of business logic, all tied together with so much Spring. Dig deeper still and we get back to the EJB era; some long-forgotten beans still clinging on to life, for lack of any developer’s appetite to understand them again. Down here is where the skeletons are.

If it ain’t broke, don’t fix it

What’s so bad about old code? It’s fulfilling some important purpose, no doubt. At least, some of it probably is.

If you look at code you wrote a year ago and can’t see anything to change, you haven’t learnt a thing in the last year

We’re always learning more: a better understanding of the domain, a better understanding of how our solution models the domain, new architectural styles, new tools, new approaches, new standards and new ideas. It is inevitable the code you wrote a year ago could be improved somehow. But how much of it have you gone back and improved recently?

The trouble with old code is that it gets increasingly hard to change. What would happen if a change in business requirements led you all the way down to the Roman-era sewers that are the EJBs? Would you implement the change the way it would have been done a decade ago? Or would you spend time trying to extract the parts that need to change? Perhaps building another shiny, new containerized micro-service along the way? That change isn’t going to be cheap though.

And this is the problem: paying back this “technical debt” is the right thing to do, but it will be slower to change this ancient code than the stuff you wrote last week or last month. The more ancient code you have the slower it will be to make changes, the slower you can develop new features. The worst part of maintaining a long running code base isn’t just paying back the debt from the things we know we did wrong; it’s the debt from things that were done right (at the time), but only now seem wrong.

How old is our code?

I’ve been looking through a variety of source code: some commercial, some open source. Across a variety of languages (Java, C#, Ruby). Generally it seems that most code bases follow a pretty similar pattern:

About 70% of the lines of code you wrote today will still be in head, unchanged, in 12 months time

Perhaps unsurprisingly, code changes most often in the first couple of months after being written. After that, it seems to enter a maintenance mode, where code changes relatively rarely.

I found this pretty surprising: after a year around 75% of the code I’ve written is still there. Imagine how much better I understand the problem today. Imagine the since-forgotten design ideas, the changed architectural vision, the new tools and libraries it could be refactored to use today. Imagine how much better every single one of those lines could be. And yet, even in a code base where we’re constantly working to pay back technical debt, we’re barely making a dent in how old the code is.

The how

How did I do this analysis? Thanks to the magic power of git, it is surprisingly easy. I can use git to do a recursive git blame on the entire repo. This lists, for each line currently in head, the commit that introduced it, who made it and when. With a bit of shell-fu we can extract a count by day or month:

I can then re-run the recursive git blame. By comparing the count of lines last checked in each month I can see how much of the code written before 2015 still exists today. With more detailed analysis I can see how the number of lines last touched in a given month changes over time to see how quickly (or not!) code decays.

Conclusion

Code is written to serve some purpose, to deliver some business value. But it quickly becomes a liability. As this code ages and rots it becomes harder and harder to change. From the analysis above it is clear to understand why a code base that has been around for a decade is mostly all archaic: with so little of the old code being changed each year it just continues to hang around, all the while we’re piling up new legacy on top – most of which will still be here next year.

What are we to do about it? It seems to be a natural law of software that over time it only accumulates. From my own experience it seems that even concerted efforts on refactoring make little impact. What are we to do? Do we just accept that changes become progressively harder as source code ages? Or do we need to find a way to encourage software to have a shorter half-life, to be re-written sooner.

Advertisements

Share this:

Like this:

Ever since the “Gang of Four” book, everyone and their uncle is an expert in patterns. Software is all about patterns – the trouble is, it seems very little of note has happened in the intervening 20 years.

Why patterns are good

Patterns help because they let us talk about code using a consistent language. Patterns make it easier to read code. Ultimately, reading code is all about being able to reason about it.

If I change this, what’s going to break?

This bug got reported in production, how the hell did it happen?

If I can see the patterns the code implements, it lets me reason about the code more easily, without having to carefully analyse exactly what it does. Without them, I have to carefully understand every single line of code and try and reverse engineer the patterns.

The trouble is, the patterns often get mixed up with the implementation. This makes it hard to discern what the pattern is, and whether it’s actually being followed. Just because a class is called TradeManagerVisitorFactory doesn’t mean I actually know what it does. And when you open up the rats nest, you realise it’s going to take a long time to know what on earth it’s doing.

Patterns Exist

Patterns exist in our code, whether we explicitly call them out or not – there are hundreds, probably thousands of patterns that we’re all familiar with. Wouldn’t it be great if these patterns, at least within a single code base, were consistent? Wouldn’t it be great if we could help keep developers on the straight and narrow so that when they instantiate a second instance of my “singleton” we can break the build because it’s obviously changed something that might break all sorts of assumptions.

If we could start identifying the patterns in our code, and (somehow) separate that from how they’re implemented, we might actually be able to change how certain things are implemented, consistently, mechanically, across an entire enterprise-scale code base. Let me guess, an example would help?

The Builder Pattern

Ok, it’s a really simple pattern – but it makes a nice example. I have a class, that I need to be able to create, so I use the builder pattern. However, really, a builder is a type of object construction pattern. If I start thinking about different ways objects can be instantiated I can quickly come up with a handful of alternatives:

A constructor, with a long parameter list

A builder, with a lot of setXxx methods

A builder, with a lot of withXxx methods, that each return the builder for method chaining

A builder with a list of properties for object initialisation (C#)

Now, this means I have four ways of implementing “the builder pattern”. And you know, it makes precisely zero difference, functionality-wise, which I choose. It’s an entirely stylistic choice. However, I probably want it to be consistent across my codebase.

When it boils down to it, my pattern has three elements:

The actual pattern – e.g. a builder with method chaining

The logic & configuration for the pattern – the list of fields, the types, any type conversions or parsing etc

The actual rendering of the first two – this is all we have today: source code

I don’t want to get rid of the last (code generation is great, until you have to, you know, actually use the generated code). This should all work from existing, real code. But maybe we could mark up the code with annotations or attributes to describe what pattern we’re following and how it’s configured. Some of the logic and configuration would have to be inferred – but that’s ok, once we know it’s a builder pattern we can take entire method bodies as “logic”.

Pattern Migration

But you know what would be really awesome? If I could change how the pattern is implemented. Maybe today I’m happy with all my objects being constructed from a long parameter list. But maybe tomorrow I decide to change my mind and use builders and withXxx method names. Wouldn’t it be awesome if I could simply change the global config and have all my code automagically refactored to match? If all my builders are annotated and identified, and all enforced to be written the same way – I can extract the configuration & logic from the implementation and re-render the pattern differently. I could programmatically replace a constructor call with a builder or vice versa.

Ok, the builder is a trivial example that probably isn’t of much use. But how many more patterns are there? Say today I have classes with lots of static methods. Well, really, they’re singletons in disguise. But none of the cool kids create singletons nowadays, they use IoC frameworks like spring to create singleton beans. Wouldn’t it be nice if I could mass-migrate a legacy codebase from static methods, to singletons, to spring beans (and back again when I realise that’s made things worse!)

Controllers

What about a more complex example – let’s compare spring-mvc and struts. Two frameworks to accomplish basically the same thing – but both with a very different philosophy. The answer isn’t to build a framework to capture the common parts of both – trust me, been there, done that: you get the worst of both worlds.

But, could you describe a pattern (probably several) that struts actions and spring-mvc controllers follow? Ultimately, both spring-mvc and struts let you respond to web requests with some code. Both give you access to the HTTP request and the session. Both give you a way of rendering a view. I wonder if you could describe how to extract the pattern-specific parts of a struts action and a spring-mvc controller? The config comes from different places, how to render views is different, how the view is selected is different, even the number of actions-per-class can be different. But, could you extract all that functionally irrelevant detail and separate the pattern configuration from the underlying source code.

If you could, maybe then you could describe a transformation between the two (sets of) patterns. This would give you a purely mechanical way of migrating a struts application to spring-mvc or vice versa.

Pattern Database

Now, the cost for me to describe, in great detail, all of the patterns for spring-mvc and struts in order to automatically migrate from one to the other is probably more than the effort to just get on and do it by hand. I could probably tackle a bunch with regexes and shell scripts, then hand finish the rest. However, what if we could define these patterns and share them? If there was some kind of open source pattern database? Then, the cost for the community is spread out – the cost for me is minimal.

Could we get to a point where our patterns are documented and codified? Where we can migrate between frameworks and even architectures with a manageable cost? An impossible dream? Or a better way of crafting software?

Share this:

Like this:

I always aspire to write well-crafted code. During my day job, where all production code is paired on, I think our quality is pretty high. But it’s amazing how easy you forgive yourself and slip into bad habits while coding alone. Is shame the driving force behind quality while pairing?

We have a number of archaic unit tests written using Easy Mock; all our more recent unit tests use JMock. This little piece of technical debt means that if you’re changing code where the only coverage is Easy Mock tests you first have to decide: do you fix up the tests or, can you hold your nose and live with / tweak the existing test for your purposes? This is not only distracting, but it means doing the right thing can be substantially slower.

This process is pretty mechanical, so as part of 10% time I started using my scripted refactoring tool – Rescripter – to mechanically translate our EasyMock tests into JMock. Rescripter let’s you run code that modifies your Java source. But this is more than just simple search & replace or regular expressions: by using Eclipse’s powerful syntax tree parsing, you have access to a fully parsed representation of your source file – meaning you can find references to methods, locate method calls, names, parameter lists etc. This is exactly what you need given the nature of the translation from one library to another.

This was inevitably fairly exploratory coding. I wasn’t really sure what would be possible and how complex the translation process would eventually become. But I started out with some simple examples, like those above. But, over time, the complexity grew as the many differences between the libraries made me work harder and harder to complete the translation.

After a couple of 10% days on this I’d managed to cobble together something awesome: I’d translated a couple of hundred unit tests; but, this was done by 700 lines of the most grotesque code you’ve ever had the misfortune to lay your eyes upon!

And then… and then last week, I got a pair partner for the day. He had to share this horror. Having spent 10 minutes explaining the problem to him and then 15 minutes explaining why it was throwaway, one-use code so didn’t have any unit tests. I was embarrassed.

We started trying to make some small changes; but without a test framework, it was difficult to be sure what we were doing would work. To make matters worse, we needed to change core functions used in numerous places. This made me nervous, because there was no test coverage – so we couldn’t be certain we wouldn’t break what was already there.

Frankly, this was an absolute nightmare. I’m so used to having test coverage and writing tests – the thought of writing code without unit tests brings me out in cold sweats. But, here I was, with a mess of untested code entirely of my own creation. Why? Because I’d forgiven myself for not “doing it right”. After all, it’s only throwaway code, isn’t it? It’s exploratory, more of a spike than production code. Anyway, once its done and the tests migrated this code will be useless – so why make it pretty? I’ll just carry on hacking away…

It’s amazing how reasonable it all sounds. Until you realise you’re being a total and utter fucktard. Even if it’s one-use code, even if it has a relatively short shelf-life

the only way to go fast, is to go well

So I did what any reasonable human being would do. I spent my lunch hour fixing this state of affairs. The end result? I could now write unit tests in Jasmine to verify the refactoring I was writing.

Not only could I now properly test drive new code. I could write tests to cover my existing legacy code, so I could refactor it properly. Amazing. And all of a sudden, the pace of progress jumped. Instead of long debug cycles and trying to manually find and trigger test scenarios, I had an easy to run, repeatable, automated test suite that gave me confidence in what I was doing.

None of this is new to me: it’s what I do day-in day-out. And yet… and yet… somehow I’d forgiven myself while coding alone. The only conclusion I can draw is that we can’t be trusted to write code of any value alone. The shame of letting another human being see your sorry excuse for code is what drives up quality when pairing: if you’re not pair programming, the code you’re writing must be shameful.

Advertisements

Share this:

Like this:

Are you a Java developer? Do you use Eclipse? Ever find yourself making the same mindless change over and over again, wishing you could automate it? ME TOO. So I wrote an Eclipse plugin that let’s you write scripts that refactor your source code.

It does what now?

Some changes are easy to describe, but laborious to do. Perhaps some examples would help:

Renaming a method and renaming every call to that method (ok, Eclipse has built-in support for this)

Replacing a call to a constructor with a static factory method (ok, IntelliJ has built-in support for this, but Eclipse doesn’t)

Moving a method and updating callers to get a reference to the target class

Replacing use of one library with a similar one with a different API

Basically anything that involves the same, small change made multiple times across your source code.

How does it work?

After installing the Rescripter Eclipse plugin write some JavaScript to describe your change; when you run this, the script can make changes to your source code using Eclipse’s built-in refactoring and source code modification support.

Why Javascript? Because its a well understood language which, via Rhino, integrates excellently with Java.

As part of some refactoring, if I want to change how these numbers are created or implemented I may want to replace all calls to the constructor with calls to the static factory method. For example, this would let me introduce a class hierarchy that the factory method knows about, or any number of other changes that cannot be done if client code calls the constructor directly.

Unfortunately, Eclipse doesn’t provide a way to do this refactoring – however, Rescripter let’s you do it. We can describe what we want to do quite simply:

Line 4 introduces a loop over each reference, filtering to only matches in source files (we can’t change .class files, only .java files).

Line 7 adds our static import.

Lines 8-11 replace the constructor call with the text “valueOf”

Our original call now looks like:

MyNumber myNumber = valueOf("42");

Now having replaced all uses of the constructor I can make it private and make whatever changes I need to the static factory method.

That’s great, but can it…?

Yes, probably. But maybe not yet. Rescripter is very basic at the minute, although you can still run some pretty powerful scripts making broad changes to your source code. If you’re having problems, have a feature suggestion or bug report – either post a comment or drop me a mail.

Share this:

Like this:

In a a blatant rip-off of the T.S Eliot quote: “if I had more time, I would have written a shorter letter” I had a thought the other day, perhaps:

If I had more time, I would have written less code

It seems to me the more time time I spend on a problem, the less code I usually end up with; I’ll refactor and usually find a more elegant design – resulting in less code and a better solution. This is at the heart of the programmer’s instinct that good code takes longer to write than crap. But as Uncle Bob tells us:

The only way to go fast is to go well

How then do we square this contradiction? The way to go fast is to go well; but if I sacrifice quality I can go faster – so I’m able to rush crap out quickly.

What makes us rush?

First off – why do developers try to rush through work? It’s the same old story that I’m sure we’ve all heard a million times:

There’s an immovable deadline: advertising has already been bought; or some external event – could be Y2K or a big sports event

We need to recognise the revenue for this change in this quarter so we make our numbers

Somebody, somewhere has promised this and doesn’t want to lose face by admitting they were wrong – so the dev team get to bend so those who over committed don’t look foolish

What happens when we rush?

The most common form of rushing is to skip refactoring. The diabolical manager looks at the Red, Green, Refactor cycle and can see a 33% improvement if those pesky developers will just stop their meddlesome refactoring.

Of course, it doesn’t take a diabolical manager – developers will sometimes skip refactoring in the name of getting something functional out the door quicker.

It’s ok though, we’ll come back and fix this in phase 2

How many times have we heard this shit? Why do we still believe it? Seriously, I think my epitaph should be “Now working on phase 2”.

Another common mistake is to dive straight in to writing code. When you’re growing your software guided by tests, this probably isn’t the end of the world as a good design should emerge. But sometimes 5 minutes thought before you dive in can save you hours of headache later.

And finally, the most egregious rush I’ve ever seen is attempting to start development from “high level requirements”. As the “inconsequential little details” are added in, the requirements look completely different. And all that work spent getting a “head start” is now completely wasted. You now either bin it and do-over, or waste more precious time re-working something you should never have written in the first place.

Does rushing work?

This is the heart of the contradiction – it feels like it does. That’s the reason we do it, isn’t it? When compelled by the customer to finish quicker, we rush – we cut corners to try and get things done quicker.

It feels like I’m going quicker when I cut corners. If only it wasn’t for all those unexpected delays that wasted my time then I would have been finished quicker. Its not my fault those delays cropped up – I couldn’t have predicted that.

It’s the product manager’s fault for not making the requirements clear

It’s the architect’s fault for not telling me about the new guidelines

Its just unlucky that a couple of bugs QA found were really hard to fix

When you cut corners, you pay for it later. Time not spent discussing the requirements in detail leads to misunderstandings; if you don’t spot it later on, you might have to wait until the show and tell for the customer to spot it – now you’ve wasted loads of time. Time not spent thinking about the design can lead to you going down an architectural blind alley that causes loads of rework. Finally time not spent refactoring leaves you with a pile of crap that will be harder to change when you start work on the next story, or even have to make changes for this one. Of course, you couldn’t have seen these problems coming – you did your best, these things crop up in software don’t they?

But what if you hadn’t rushed? What if rather than diving in, you’d spent another 15 minutes discussing the requirements in detail with the customer? You might have realised earlier that you had it all wrong and completely misunderstood what she wanted. What if you spent 30 minutes round a whiteboard with a colleague discussing the design? Maybe he would have pointed out the flaws in your ideas before you started coding. Finally, what if you’d spent a bit more time refactoring? That spaghetti mess you’re going to swear about next week and spend days trying to unravel will still be fresh in your mind and easier to untangle. For the sake of an hour’s work you could save yourself days.

How much is enough?

Of course, it’s all very well to say we will not write crap any more; but it’s not a binary distinction, is it? There’s a sliding scale from highly polished only-really-suitable-for-academics perfection; to sheer, unmitigated, what-were-you-thinking-you-brain-dead-fuck crapitude.

If spending more time discussing requirements could save time later, why not spend years going through a detailed requirements gathering exercise? If spending more time thinking before coding could save time later, why not design the whole thing down to the finest detail before any coding is done? Finally if refactoring mercilessly will save time, why not spend time refactoring everything to be as simple as possible? Actually, wait, this last one probably is a good idea.

Professional software development is always a balancing act. It’s a judgement call to know how much time is enough.

How long does it take?

When working through a development task I choose to expend a certain amount extra effort over and above how long it takes me to type in the code and test it; this extra time is spent discussing requirements, arguing about the design, refactoring the code etc etc. Ideally I want to expend just enough effort to get the job done as quickly as possible (the lazy and impatient developer). If I expend too little effort I risk being delayed later by unnecessary rework; if I expend too much effort I’ve wasted my time.

However, I don’t know in advance what that optimum amount of effort is – it’s a guess based on experience. But I expect the amount of effort I put in to be within some range of the optimum – the lowest the point on the graph:

All other things being equal, I’d expect the amount of time it actually takes to complete the task to be within some margin based on how close I got to the optimum amount of effort. Sometimes I do too little and waste time with rework. Other times I spend too long – e.g. too much detail in the design that didn’t add any value – so the extra time I spent is wasted.

This is by no means all the uncertainty in an estimate; but this is the difference between me not doing quite enough and having to pay for it later (refactoring, debugging or plain rework); versus doing just enough at every stage so that no effort is wasted and there’s no rework I could have avoided more cheaply. In reality the time taken is likely to be somewhere in the middle: not to great but not too shabby.

There’s an interesting exercise here to consider the impact experience has on this. When I was first starting out I’d jump straight into coding; my effort range would be to the left of the graph: I’d spend most of my time re-writing what I’d done so the actual time taken would be huge. Similarly, as I became more experienced I’d spend ages clarifying requirements, writing detailed designs and refactoring until the code was perfect; now my effort range would be to the right of the graph – I’d expend considerable upfront effort, much of which was unnecessary. Now, I’d like to think, I’m a bit more balanced and try to do just enough. I have no idea how you could measure this, though.

Reducing effort to save time

What happens when we rush? I think when we try to finish tasks quicker, we cut down on the extra effort – we’re more likely to accept requirements as-is than challenge them; we’re more likely to settle for the first design idea we think of; we’re more likely to leave the code poorly refactored. This pulls the effort range on the graph to the left.

To try and get done more quickly, I’ve reduced the amount of effort by “shooting low”: I cut corners and expend less effort than I would have done otherwise.

The trouble is this doesn’t make the best-case any better – I might still get the amount of effort bang on and spend the minimum amount of time possible on this task. However because I’m shooting low, there’s a danger now I spend far too little extra effort – the result is I spend even longer: I have to revisit requirements late in the day; make sweeping design changes to existing code; or waste time debugging or refactoring poorly written code.

This is a classic symptom of a team rushing through work: simple mistakes are made that, in hindsight, are obvious – but because we skipped discussing requirements or skipped discussing the design we never noticed them.

When I reduce the amount of extra effort I put in, rather than getting things done quicker, rushing may actually increase the time taken. This is the counter-intuitive world we live in – where aggressive deadlines may actually make things go more slowly. Perhaps instead I should have called this article: