I’ve inherited 200K lines of spaghetti code—what now?

It's a monstrous task with gigantic answers.

This Q&A is part of a biweekly series of posts highlighting common questions encountered by technophiles and answered by users at Stack Exchange, a free, community-powered network of 80+ Q&A sites.

kmote asks: I am newly employed as the sole "SW Engineer" in a fairly small shop of scientists who have spent the last 10-20 years cobbling together a vast code base. (It was written in a virtually obsolete language: G2—think Pascal with graphics). The program itself is a physical model of a complex chemical processing plant; the team that wrote it has incredibly deep domain knowledge but little or no formal training in programming fundamentals. They've recently learned some hard lessons about the consequences of nonexistent configuration management. Their maintenance efforts are also greatly hampered by the vast accumulation of undocumented "sludge" in the code itself. I will spare you the "politics" of the situation (there's always politics!), but suffice it to say, there is not a consensus of opinion about what is needed for the path ahead.

They have asked me to begin presenting to the team some of the principles of modern software development. They want me to introduce some of the industry-standard practices and strategies regarding coding conventions, lifecycle management, high-level design patterns, and source control. Frankly, it's a fairly daunting task and I'm not sure where to begin.

Initially, I'm inclined to tutor them in some of the central concepts of The Pragmatic Programmer, or Fowler's Refactoring ("Code Smells", etc). I also hope to introduce a number of Agile methodologies. But ultimately, to be effective, I think I'm going to need to hone in on 5-7 core fundamentals; in other words, what are the most important principles or practices that they can realistically start implementing that will give them the most "bang for the buck."

So that's my question: What would you include in your list of the most effective strategies to help straighten out the spaghetti (and prevent it in the future)? Related: When is code "legacy"?

Answer: The (very) long answer (237 Votes)

Foreword

This is a daunting task indeed, and there's a lot of ground to cover. So I'm humbly suggesting this as somewhat comprehensive guide for your team, with pointers to appropriate tools and educational material. Remember: These are guidelines, and as such are meant to be adopted, adapted, or dropped based on circumstances.Beware: Dumping all this on a team at once would most likely fail. You should try to cherry-pick elements that would give you the best bang-for-sweat, and introduce them slowly, one at a time.Note: Not all of this applies directly to Visual Programming Systems like G2. For more specific details on how to deal with these, see the Addendum section at the end.

Executive summary for the impatient

Define a rigid project structure, with:

project templates

coding conventions

familiar build systems

and sets of usage guidelines for your infrastructure and tools.

Install a good SCM and make sure they know how to use it.

Point them to good IDEs for their technology, and make sure they know how to use them.

Couple the build system to continuous integration and continuous inspection systems.

With the help of the above, identify code quality "hotspots" and refactor.

Now for the long version... Caution, brace yourselves!

Rigidity is (often) good

This is a controversial opinion, as rigidity is often seen as a force working against you. It's true for some phases of some projects. But once you see it as a structural support, a framework that takes away the guesswork, it greatly reduces the amount of wasted time and effort. Make it work for you, not against you. Rigidity = Process / Procedure. Software development needs good processes and procedures for exactly the same reasons that chemical plants or factories have manuals, procedures, drills, and emergency guidelines: preventing bad outcomes, increasing predictability, maximizing productivity... Rigidity comes in moderation, though!!

Rigidity of the project structureIf each project comes with its own structure, you (and newcomers) are lost and need to pick up from scratch every time you open them. You don't want this in a professional software shop, and you don't want this in a lab either.

Rigidity of the build systemsIf each project looks different, there's a good chance they also build differently. A build shouldn't require too much research or too much guesswork. You want to be able to do the canonical thing and not need to worry about specifics: configure; make install, ant, mvn install, etc. Re-using the same build system and making it evolve over the time also ensures a consistent level of quality. You do need a quick READMEto point to the project's specifics, and gracefully guide any user/developer/researcher. This also greatly facilitates other parts of your build infrastructure, namely:

So keep your build (like your projects) up to date, but make it stricter over time, and more efficient at reporting violations and bad practices. Do not reinvent the wheel, and reuse what you have already done. Recommended Reading:

Rigidity in the choice of programming languagesYou can't expect, especially in a research environment, to have all teams (and even less all developers) use the same language and technology stack. However, you can identify a set of "officially supported" tools and encourage their use. The rest, without a good rationale, shouldn't be permitted (beyond prototyping). Keep your tech stack simple, and the maintenance and breadth of required skills to a bare minimum: a strong core.

Rigidity of the coding conventions and guidelinesCoding conventions and guidelines are what allow you to develop both an identity as a team, and a shared lingo. You don't want to err into terra incognitaevery time you open a source file. Nonsensical rules that make life harder or forbid actions explicitly to the extent that commits are refused based on single simple violations are a burden. However:

a well thought-out ground ruleset takes away a lot of the whining and thinking: nobody should break under any circumstances

a set of recommended rules provide additional guidance

Personal Approach: I am aggressive when it comes to coding conventions, because I do believe in having a lingua franca for my team. When crap code gets checked-in, it stands out like a cold sore on the face of a Hollywood star: it triggers a review and an action automatically. In fact, I've sometimes gone as far as to advocate the use of pre-commit hooks to reject non-conforming commits. As mentioned, it shouldn't be overly crazy and get in the way of productivity: it should drive it. Introduce these slowly, especially at the beginning. But it's way preferable over spending so much time fixing faulty code that you can't work on real issues.

Some languages even enforce this by design:

Java was meant to reduce the amount of dull crap you can write with it (though no doubt many manage to do it).

Python's block structure by indentation is another idea in this sense.

Go, with its gofmt tool, which completely takes away any debate and effort (and ego!!) inherent to style: run gofmt before you commit.

Make sure that code rot cannot slip through. Code conventions, continuous integration and continuous inspection, pair programming, and code reviews are your arsenal against this demon. Plus, as you'll see below, code is documentation, and that's another area where conventions encourage readability and clarity.

Rigidity of the documentationDocumentation goes hand in hand with code. Code itself is documentation. But there must be clear-cut instructions on how to build, use, and maintain things. Using a single point of control for documentation (like a WikiWiki or DMS) is a good thing. Create spaces for projects, spaces for more random banter and experimentation. Have all spaces reuse common rules and conventions. Try to make it part of the team spirit. Most of the advice applying to code and tooling also applies to documentation.

Rigidity in code commentsCode comments, as mentioned above, are also documentation. Developers like to express their feelings about their code (mostly pride and frustration, if you ask me). So it's not unusual for them to express these in no uncertain terms in comments (or even code), when a more formal piece of text could have conveyed the same meaning with less expletives or drama. It's OK to let a few slip through for fun and historical reasons: it's also part of developing a team culture. But it's very important that everybody knows what is acceptable and what isn't, and that comment noise is just that: noise.

Rigidity in commit logsCommit logs are not an annoying and useless "step" of your SCM's lifecycle: you DON'T skip it to get home on time or get on with the next task, or to catch up with the buddies who left for lunch. They matter, and, like (most) good wine, the more time passes, the more valuable they become. So DO them right. I'm flabbergasted when I see co-workers writing one-liners for giant commits, or for non-obvious hacks. Commits are done for a reason, and that reason ISN'T always clearly expressed by your code and the one line of commit log you entered. There's more to it than that. Each line of code has a story and a history.The diffs can tell its history, but you have to write its story.

Why did I update this line? -> Because the interface changed. Why did the interface change? -> Because the library L1 defining it was updated. Why was the library updated? -> Because library L2, that we need for feature F, depended on library L1. And what's feature X? -> See task 3456 in issue tracker.

It's not my SCM choice, and may not be the best one for your lab either; but Git gets this right, and tries to force you to write good logs more than most other SCMs systems, by using short logs and long logs. Link the task ID (yes, you need one) and a leave a generic summary for the shortlog, and expand in the long log: write the changeset's story. It is a log:It's here to keep track and record updates.

Rule of thumb: If you were searching for something about this change later, is your log likely to answer your question?

Projects, documentation, and code are alive. Keep them in sync, otherwise they do not form that symbiotic entity anymore. It works wonders when you have:

Where this tracker's tickets themselves link to the changesets in your SCM (and possibly to the builds in your CI system)

A documentation system that links to all of these

Code and documentation need to be cohesive.

Rigidity in testing

Rules of thumb:

Any new code shall come with (at least) unit tests.

Any refactored legacy code shall come with unit tests.

Of course, these need:

to actually test something valuable (or they are a waste of time and energy)

to be well written and commented (just like any other code you check in)

They are documentation as well, and they help to outline the contract of your code. Especially if you use TDD. Even if you don't, you need them for your peace of mind. They are your safety net when you incorporate new code (maintenance or feature) and your watchtower to guard against code rot and environmental failures. Of course, you should go further and have integration tests, and regression tests for each reproducible bug you fix.

Rigidity in the use of the toolsIt's OK for the occasional developer/scientist to want to try some new static checker on the source, generate a graph or model using another, or implement a new module using a DSL. But it's best if there's a canonical set of tools that allteam members are expected to know and use. Beyond that, let members use what they want, as long as they are ALL:

Productive

NOT regularly requiring assistance

NOT regularly adjusting to your general infrastructure

In areas like code, build system, or documentation

NOT affecting others' work

ABLE to timely perform any task requested

If that's not the case, then enforce that they fallback to defaults.

Rigidity vs. versatility, adaptability, prototyping, and emergenciesFlexibility can be good. Letting someone occasionally use a hack, a quick-n-dirty approach, or a favorite pet tool to get the job done is fine. Never let it become a habit, and neverlet this code become the actual codebase to support.

It's about the code, not about the developersMake developers conscious of the quality of their code, But make them see the code as a detached entity and not an extension of themselves, which cannot be criticized. It's a paradox: you need to encourage ego-less programmingfor a healthy workplace but to rely on ego for motivational purposes.

From scientist to programmer

People who do not value and take pride in code do not produce good code. For this property to emerge, they need to discover how valuable and fun it can be. Sheer professionalism and desire to do good is not enough: it needs passion. So you need to turn your scientists into programmers(in the large sense). Someone argued in comments that after 10 to 20 years on a project and its code, anyone would feel attachment. Maybe I'm wrong, but I assume they're proud of the code's outcomes and of the work and its legacy, not of the code itself or of the act of writing it. From experience, most researchers regard coding as a necessity, or at best as a fun distraction. They just want it to work. The ones who are already pretty versed in it and who have an interest in programming are a lot easier to persuade to adopt best practices and switch technologies. You need to get them halfway there.

Code maintenance is part of research work

Nobody reads crappy research papers. That's why they are peer-reviewed, proofread, refined, rewritten, and approved time and time again until deemed ready for publication. The same applies to a thesis and a codebase! Make it clear that constant refactoring and refreshing of a codebase prevents code rot and reduces technical debt, and facilitates future re-use and adaptation of the work for other projects.

Why all this??!

Why do we bother with all of the above? For code quality. Or is it quality code...? These guidelines aim at driving your team toward this goal. Some of these points help by simply showing your team the way and letting them do it (which is much better) and others take them by the hand (but that's how you educate people and develop habits). How do you know when the goal is within reach?

Quality is measurableNot always quantitatively, but it is measurable. As mentioned, you need to develop a sense of pride in your team, and showing progress and good results is key. Measure code quality regularly and show progress between intervals, and how it matters. Do retrospectives to reflect on what has been done, and how it made things better or worse. There are great tools for continuous inspection. Sonar being a popular one in the Java world, but it can adapt to any technologies; and there are many others. Keep your code under the microscope and look for these pesky annoying bugs and microbes.

But what if my code is already crap?

All of the above is fun and cute like a trip to Never Land, but it's not that easy to do when you already have (a pile of steamy and smelly) crap code, and a team reluctant to change.

67 Reader Comments

A key point in the original question is the users are "scientists." For R&D and software functionality that is not well defined, unit testing is not useful. The research function might go through many iterations before the behavior is stable enough to have reliable unit tests.

One thing to be aware of is that you need to have massive buy-in to change something like this.

A good development model massively slows development speed in the short run, causing small changes to take hours instead of minutes. (Add a formal test for 50 lines of code? Change logs? Commenting 50 lines? I just want to write the code and go home. And I need the change by tomorrow morning!)

Of course, the hundreds of hours that are saved by taking shortcuts don't make up for the thousands of hours lost later, but the key point is that people *feel* the hundreds of hours that they are "losing" by doing things properly *right* now.

So, expect heavy push-back. Also, make it clear to managers that they cannot expect the same level of "productivity" as before. (That's often the biggest hurdle.)

Having (once long ago) seen a manager nearly fired after he adopted good practices because the upper-management couldn't understand that good practices pay off in months, not days, I no longer fault managers for being heavily concerned with short-term metrics...

A key point in the original question is the users are "scientists." For R&D and software functionality that is not well defined, unit testing is not useful. The research function might go through many iterations before the behavior is stable enough to have reliable unit tests.

Agreed. Applies also to any team using software to solve unusual problems. In such cases, you're dealing with very smart people who can both do their science and use the tools. I think OP's situation is one in which the scientists are better at their science than at programming, but that doesn't mean they're bad at programming. It sounds like a giant sandbox, built DIY to address a specific concern long ago, now evolved into something of a beast. In this case, adopting practices from a more structured project is likely to frustrate and stymie the scientists.

I think OP is on the right track by looking at how the scientists are using their tools. In other words, their habits. It may be sufficient to teach them to use their existing tools and expertise more effectively. This is where Pair Programming might be beneficial... OP might want to spend a few days sitting with each developer one on one, studying their work and the problems they solve, and observing their natural tendencies, then offering tweaks here and there (ah! i see what you're doing, have you tried this?) but also analyzing the larger pattern and coming back with larger suggestions (we need source control, and here's how it might work for us).

An individual in a technology role can often work most effectively by trying to understand as well as possible what the user is trying to achieve, and then bringing the experience to bear on those problems from the user's perspective. We can come in with all our experience and say how it's going to be, but if we're trying to tell a carpenter to start using a welding torch, it doesn't help anybody. Something to keep in mind.

In order for someone to take on this initiative, you need to convince management that there's VALUE in refactoring code. There's no direct profit correlation, hence it's hard to justify this activity. (It's about making a PROFIT, not a ego trip).

In some cases, you can make a justification because the code is becoming unmaintainable or cannot be improved for new features. That's when you strike the iron while it's still hot.

Remember, if you break the load because you "refactor" code, who's to blame? Why would anyone be that stupid to stick their head out ready to be chopped off?

Besides, what makes you think you can do so much better than someone who have spent 5-10years working on the code. Just because you are a software engineer doesn't mean you are Mr. Know-it-all. Domain knowledge my friend.

Don't blow up a mountain with a nuclear bomb.

Refactor when necessary, and maintain the rest. Remember, all projects come to an end at some point, so when a new project replaces this current project, you can leave your mark and perform properly software development practices. But not too much, because being all proper are for academics. In the real world, it's about compromises and delivering in a TIMELY manner.

I personally wouldn't "refactor" anything because you're just risking breaking it without any possible gain. What I would do is maintain the existing structure and slowly replace sections of code as they develop errors or need new features. That's actually what I do with the horribly designed (basically not designed) access database (thousands of lines of code and 1500+ objects.) mess that I have to work with every day.

As a worst case scenario you can develop a replacement, in fact if you get spare time it might be worth starting now, because most people don't understand how long it takes to write a replacement system.

"Well, -this- is the exact code I used to write Nevyn 2003, and -this- is the exact code I used to write Nevyn 2004...." Diff says: almost identical code.

But the actual prof in charge has five grad students - all, effectively, doing their own versioning. Not -quite- hardcoding, but not focused on "Hey, I could take a week from my actual work and make this program -much- more general." Instead, generic program shoved -this- way for this project. Shoved -that- way for project two. No real reason to merge the diffs - the two program have already provided the answers they needed to generate - now they're just archival fluff.

- Require Doxygen-style documentation comments for every function! As the code is written!

I can't stress that enough. First off, writing documentation right in the code is very easy, and having it be very easy means that it will get done. If you have a separate documentation system, that probably means that the docs won't get written until much later, thereby losing much of their value.

Having easily-found docs during the coding stage helps prevent lots of bugs from being created in the first place. As any developer knows - writing code is easy; making it work is hard. Bugs can take huge amounts of time to track down, and they often come from overlooked details. If those details are mentioned in the docs, they'll be much less likely to be overlooked in the first place.

Using doxygen-style docs isn't strictly necessary, so long as the docs are easy to find. The advantage of using doxygen-style docs is that they can be easily made into real, formal hyper-linked documents. You can link to them from other web pages that provide system overviews and block diagrams.

A final nugget: If a function takes a "size" parameter, then either have the whole library use the same kinds of sizes (size in bytes, size in 32-bit words, size in records), or else name the size parameter accordingly (sizeInBytes, sizeInU32s, sizeInRecords, etc.). Having various API's take size parameters of different sizes is a gold-plated invitation for bugs to come.

I personally wouldn't "refactor" anything because you're just risking breaking it without any possible gain. What I would do is maintain the existing structure and slowly replace sections of code as they develop errors or need new features. That's actually what I do with the horribly designed (basically not designed) access database (thousands of lines of code and 1500+ objects.) mess that I have to work with every day.

That is the definition of refactoring. I think what you're referring to is a total rewrite, ie start a new project.

Make the automated testing system. Keep it up to date. Add tests for each new bit of functionality. Run the tests daily or before each change commit. (This may require a "quick" test and a "long" test, various targeted tests, or what have you; set up what you need.)

It's a big pain to write tests. However, doing so is usually straightforward. It's a bigger pain to find obscure bugs. Testing also helps avoid really embarrassing stupid mistakes, which are surprisingly easy to make with computers.

- Require Doxygen-style documentation comments for every function! As the code is written!

Be careful though: you have to make sure people update the comments when they update the function (which is much easier said than done). The comments must deal with the "Why" rather than the how (you can tell the how by looking at the code) or other external/navigational information.

These kinds of comments can be a detriment and make life harder if they refer to an older version of the function -- subtly misleading you to base new code on false assumptions -- unless you update the comment too!

Not sure if this applies here's, but have your scientists considered using a commercial software. That software wouldn't have all their know how I am pretty sure, but it might have a lot of the more generally known stuff already coded in and have tools to build on that basis.I work for a software company that produces "chemical process simulation" software and many of our customers use our software as the basis to add their own know how.

Where is the empirical support for "defining a rigid project structure" for this domain? Scientists tend to be skeptical, especially of authority. Justifiably, few scientists are going to put their research output at risk just on the whims of software engineers.

In a startup company, you're not going to see alot of documentation. End result is key and performance is priority 1.Documentation are for people who don't know how to code at an expert level.

If you are running a large company with multisite teams, by all means write documentation. you don't need to document every function, just the ones that you interface with external sources (ex. API Doc). Code changes ALL the time, bug fixes, etc ... inline documentation is a recipe for disaster if it's not kept up to date.

Write code that doesn't use complex syntax that is rarely used, or design patterns that are highly complex. It's good for your ego, but terrible for maintenance. Not everyone is a guru. The ones that inherit your code will be scratching their heads wondering what the hell you are doing. I guess it's good for job security, but morally wrong to do so.

CityZen wrote:

Here's my best low-level programming practice suggestion:

- Require Doxygen-style documentation comments for every function! As the code is written!

I can't stress that enough. First off, writing documentation right in the code is very easy, and having it be very easy means that it will get done. If you have a separate documentation system, that probably means that the docs won't get written until much later, thereby losing much of their value.

Having easily-found docs during the coding stage helps prevent lots of bugs from being created in the first place. As any developer knows - writing code is easy; making it work is hard. Bugs can take huge amounts of time to track down, and they often come from overlooked details. If those details are mentioned in the docs, they'll be much less likely to be overlooked in the first place.

Using doxygen-style docs isn't strictly necessary, so long as the docs are easy to find. The advantage of using doxygen-style docs is that they can be easily made into real, formal hyper-linked documents. You can link to them from other web pages that provide system overviews and block diagrams.

A final nugget: If a function takes a "size" parameter, then either have the whole library use the same kinds of sizes (size in bytes, size in 32-bit words, size in records), or else name the size parameter accordingly (sizeInBytes, sizeInU32s, sizeInRecords, etc.). Having various API's take size parameters of different sizes is a gold-plated invitation for bugs to come.

In a startup company, you're not going to see alot of documentation. End result is key and performance is priority 1.Documentation are for people who don't know how to code at an expert level.

If you are running a large company with multisite teams, by all means write documentation. you don't need to document every function, just the ones that you interface with external sources (ex. API Doc). Code changes ALL the time, bug fixes, etc ... inline documentation is a recipe for disaster if it's not kept up to date.

Write code that doesn't use complex syntax that is rarely used, or design patterns that are highly complex. It's good for your ego, but terrible for maintenance. Not everyone is a guru. The ones that inherit your code will be scratching their heads wondering what the hell you are doing. I guess it's good for job security, but morally wrong to do so.

CityZen wrote:

Here's my best low-level programming practice suggestion:

- Require Doxygen-style documentation comments for every function! As the code is written!

I can't stress that enough. First off, writing documentation right in the code is very easy, and having it be very easy means that it will get done. If you have a separate documentation system, that probably means that the docs won't get written until much later, thereby losing much of their value.

Having easily-found docs during the coding stage helps prevent lots of bugs from being created in the first place. As any developer knows - writing code is easy; making it work is hard. Bugs can take huge amounts of time to track down, and they often come from overlooked details. If those details are mentioned in the docs, they'll be much less likely to be overlooked in the first place.

Using doxygen-style docs isn't strictly necessary, so long as the docs are easy to find. The advantage of using doxygen-style docs is that they can be easily made into real, formal hyper-linked documents. You can link to them from other web pages that provide system overviews and block diagrams.

A final nugget: If a function takes a "size" parameter, then either have the whole library use the same kinds of sizes (size in bytes, size in 32-bit words, size in records), or else name the size parameter accordingly (sizeInBytes, sizeInU32s, sizeInRecords, etc.). Having various API's take size parameters of different sizes is a gold-plated invitation for bugs to come.

One particular problem with R&D contracts is that there is generally a hard limit on the amount of money that can be spent, and this limit is often too small to both (1) follow complex software development processes and (2) build anything that works.

Large defense contractors tend to do (1) and not worry about (2). So at the end of the contract they can tell the government, "We followed all the rules, and here's the paperwork to prove it. And yeah, we spent $X million and nothing works, but if you give us another $X million we'll try again." This is because [insert-huge-defense-contractor] can say to the government, "Well, you've already spent billions on this project, you wouldn't want all that to go to waste, because then it would be an embarrassing public failure, and how would that make you look?"

On the other hand, small companies tend to do the opposite: hack like crazy and hope to get something demoable within the limited time and money available. But, of course, the resulting code will be difficult to maintain...

tenuki raises a very valid point in the first post. Not all software engineering processes, best practices, and tools are suitable for use by scientists. As someone who is educated and practicing as both a software engineer and scientist, I have heard and contributed many complaints about software engineering recommendations that are not appropriate to the scientific domain: I once gave an invited talk/rant on the subject at IBM's CASCON 2010.

In short, most of the issues have to do with the transient nature of the scientific software, the fact that the software is heavily reliant on floating-point computations (a fact that complicates testing), and the huge volumes of data that are involved. These realities undermine the usefulness of a lot of software engineering approaches that do not take them into account.

Good research work in this area has been done by a number of folks including Diane Kelly, Greg Wilson, Spencer Smith, Steve Easterbrook, Judith Segal, and Paul Dubois, Do a Google search for their names with key words "computational science" and "software engineering". There was a workshop series at ICSE for a few years called "Software Engineering for Computation Science and Engineering" (SECSE) that brought a lot of the research into one place.

Circling back to the tenuki's original point about unit testing: my graduate work focused on scientific software testing. As part of my research I spent some time with a scientist setting up unit tests for a piece of image analysis software. We found that unit test were helpful for the low level numerical routines that don't change very much once they're were implemented. However, they didn't help very much with the more transient, research oriented parts of the code.

Also, CityZen raises a very important point in his post. Once a scientific code has settled down a bit (I assume that would be true of the code that started this whole discussion) then it is absolutely essential to get some automated tests up and running. I have found this to be true in my own experience, and research confirms it -- e.g., Steve Easterbrook's studies of climate modelling software highlight the value of automated regression tests.

In a case like this, I have to say it might be better to go to square 1, though taking cues from the older code. 200K lines of code is way too much for almost any program I can think of to do what it needs to do.

In a startup company, you're not going to see alot of documentation. End result is key and performance is priority 1.Documentation are for people who don't know how to code at an expert level.

Your last statement could not be more wrong.Your first statement is the reason that lots of startups will run into problems down the line.

Sure, for a given project, you might be able to wrap your head around the entire thing and won't feel that documentation is necessary. That's fine if no one else has to look at the code and you won't have to look at the code once you're done with it. But those are rare circumstances if you're working on a project with any kind of life expectancy.

web2dot0 wrote:

If you are running a large company with multisite teams, by all means write documentation. you don't need to document every function, just the ones that you interface with external sources (ex. API Doc). Code changes ALL the time, bug fixes, etc ... inline documentation is a recipe for disaster if it's not kept up to date.

Write code that doesn't use complex syntax that is rarely used, or design patterns that are highly complex. It's good for your ego, but terrible for maintenance. Not everyone is a guru. The ones that inherit your code will be scratching their heads wondering what the hell you are doing. I guess it's good for job security, but morally wrong to do so.

The documentation that I was referring to is the API documentation, not the inline comments. While a lot can be inferred from a decent function and parameter names, things like corner cases and error handling are usually not so obvious, and knowing these can be critical to avoiding bugs.

Yes, I do work in a reasonably large company with multi-site teams. I don't think this is a special case. I've often come back to small projects that I wrote over a year ago and wished it had decent documentation. No matter how good you are, you can't remember everything.

- Require Doxygen-style documentation comments for every function! As the code is written!

Be careful though: you have to make sure people update the comments when they update the function (which is much easier said than done). The comments must deal with the "Why" rather than the how (you can tell the how by looking at the code) or other external/navigational information.

These kinds of comments can be a detriment and make life harder if they refer to an older version of the function -- subtly misleading you to base new code on false assumptions -- unless you update the comment too!

That's why it's best when the docs are right in the code. It makes it easy to update when you change the code. Having the docs live in some separate area helps guarantee that it will get out of date.

You do know that startup companies rewrote code all the time right? I've worked in many startups so I am well awarded of it. There's nothing wrong with little to no docs because Comanies need to be agile. Your example is when developers write crappy code. No docs will help you fix crap. Startups fail because the code is architected/written poorly, not because of lack of docs. Been there.Trust a guy with experience bud.

CityZen wrote:

Your last statement could not be more wrong.Your first statement is the reason that lots of startups will run into problems down the line.

Sure, for a given project, you might be able to wrap your head around the entire thing and won't feel that documentation is necessary. That's fine if no one else has to look at the code and you won't have to look at the code once you're done with it. But those are rare circumstances if you're working on a project with any kind of life expectancy.

web2dot0 wrote:

If you are running a large company with multisite teams, by all means write documentation. you don't need to document every function, just the ones that you interface with external sources (ex. API Doc). Code changes ALL the time, bug fixes, etc ... inline documentation is a recipe for disaster if it's not kept up to date.

Write code that doesn't use complex syntax that is rarely used, or design patterns that are highly complex. It's good for your ego, but terrible for maintenance. Not everyone is a guru. The ones that inherit your code will be scratching their heads wondering what the hell you are doing. I guess it's good for job security, but morally wrong to do so.

The documentation that I was referring to is the API documentation, not the inline comments. While a lot can be inferred from a decent function and parameter names, things like corner cases and error handling are usually not so obvious, and knowing these can be critical to avoiding bugs.

Yes, I do work in a reasonably large company with multi-site teams. I don't think this is a special case. I've often come back to small projects that I wrote over a year ago and wished it had decent documentation. No matter how good you are, you can't remember everything.

What does the code do, precisely? Is it guiding missiles? Is it one-off code written as an experiment? Will it be used in a commercial product? Is it going to be rewritten at any point?

It's nice to refactor things and write tests, but if it doesn't serve the goals of the organization, it's a waste of everybody's time.

What the original poster needs to do here is identify the goals, create a clear plan towards achieving them, and then get buy-in from the rest of the organization. He needs to engage his brain rather than just blindly applying a Methodology.

A lot of the responses here are just people pushing their own ideology rather than actually trying to figure out what the organization really needs. And unfortunately, people like this are why big software projects often fail.

You do know that startup companies rewrote code all the time right? I've worked in many startups so I am well awarded of it. There's nothing wrong with little to no docs because Comanies need to be agile. Your example is when developers write crappy code. No docs will help you fix crap. Startups fail because the code is architected/written poorly, not because of lack of docs. Been there.Trust a guy with experience bud.

You seem to be saying "if, like many startups, you write bad code that's going to be thrown away soon, you don't need to document it".

I would tend to agree with that. However, I'd suggest you don't write bad code in the first place. "Agile" doesn't mean that all code is throw-away code. At some point, assuming your project is going somewhere, you've got to move from prototype to production. The team is going to grow. If you don't start with some decent programming practices, you'll be paying the price down the line, assuming you get there.

I'm not trying to suggest that if you don't write up a full formal header for every function you're doomed. But if you don't point out the weird cases in the docs, you'll be spending hours chasing down bugs that could have been avoided easily had you done so. Write up the most important stuff first, fill in the rest from time to time. Though don't wait until you've got plenty of spare time, since that never happens.

You have to be careful when you are working with old procedural code. Often times that's all they'll produce is spaghetti code. In fact if the engineers tried to get some object oriented behavior from it, then it's 100% guaranteed you got spaghetti on your hands. Just comes with the territory and why we have 4G languages to begin with. Not everything is going to be pretty as Java or C# or even C++ for that matter so don't force it.

Maintaining a huge codebase is a difficult task, especially if it has been left to grow without looking at how the higher-level architecture is structured and/or the smallest fix/change is done to address an issue (e.g. removing all references to a file, but not the file itself).

There are several important things to take note of when looking at refactoring code:

1. A refactoring is an isomorphic change -- it can be reversed.

2. The behaviour of the application does not change after a refactoring -- this is not a rewrite.

3. Each refactoring step should be committed to source control -- it should be easy for a reviewer (or you) to understand what you did at each step. This becomes important when trying to track down regressions as it allows you to (a) track down the commit using tools like `git bisect` and (b) easily verify the commit once identified.

4. Automated tests are critical in reducing the chance of introducing bugs during a refactoring, but often this is not possible (e.g. for UI code) so you need to do a manual test at each step.

5. Refactoring should be done with a goal in mind, creating a roadmap or plan for the changes. This is important because it is easy to apply automated refactorings without understanding where you are going in terms of the higher level architecture (especially since refactorings can be reversed) and you may end up making it harder to refactor (e.g. splitting a large function into smaller ones where you could have looped through some data).

6. Keep a note of things that need refactoring (or better yet, put them in your bug tracking system) and make sure they are up-to-date and monitored. Having them in the bug tracker allows you to link other items to them (e.g. bug X is difficult to fix due to code Y being complicated). Do this for the larger items or the items that cannot be fixed there and then.

7. Refactoring should be part of the development process -- when adding new code or fixing bugs -- refactoring the related code to make it easier to implement the new functionality or fix the bug.

8. Refactoring can start simple (tidy up the code, document the system, remove unused types, classes, files, functions, etc.) and become more complex as you better understand the system.

It is a good idea when adding new functionality to look at what parts of the code you are changing and why. Can this be simplified? Where is the code difficult to modify and why? This will help you get a feel for what the high level architecture should look like.

It is a good idea when fixing a bug (as well as writing a test for it) to look at why the bug was introduced. Can you change the design to pick up these bugs earlier (e.g. using templates/generics instead of objects in containers)? Can you detect the bug using a code analysis tool (and find other instances of the bug)?

Is this code still science? If so, it is a lost cause. Remember that science is repeatable. They can't publish based on hundreds of K of undocumented tangle.

So, has it become engineering? In that case, can it be replaced off the shelf. I know that scientists tend to regard grad student labor as free, but if this has become engineering it presumably has a budget. Why are these guys maintaining a product sized code corpus if that is not their value-add? Surely a lot of it can be done off the shelf. Having seen scientists do stuff like this, I'd bet a lot of what they did is off the shelf libraries in the MatLab/Octave world. And anything you can't find in that rich ecosystem is probably something special enough to be worth the group writing, which is likely to be a small enough project for you or them to do nicely.

Don't start by spec'ing the code. Start by spec'ing what they do with it. What they want to do with it. Then go shopping to get what they really want and work on transferring existing processes to use the new code. Sure, it will be painful. But really it is pointless to refine a body of code these guys were amateur at. There is something they are being paid for (and by extension, you) and it is not to train themselves to reach novice class programmers on a body of code which is a distraction.

And once you have got them on the right track, go find a better job. This place has no career path. Be kind, teach them to fish, but don't stick around forever gutting and cleaning the catch.

Your explanation that startups fail because of poor documentation is BS. Startups don't fail because of poor documentation. It's because

A) Bad IdeaB) Poorly written code that doesn't scaleC) Hire the wrong people and don't know when to get rid of themD) Get rid of people that cannot keep up with the changing environment.

The list goes on. But poor documentation is at the BOTTOM of the list.

Your BS explanation that "well, don't write crap then" is like saying ... "if you want to win, run like Usain Bolt".Yeah I know. We all know. But that doesn't mean it can be done.

Nobody wants to write crap on purpose. It's like saying I want to do my worst because I want to underachieve and proud of it.

I'm not going to bother trying to elaborate further because you are clearly ideological. Start your own company, be successful at it, then come back to me and pound your chest and educate me in how things should be done. But don't be a armchair Quarterback and tell me how things should be done when you haven't done it yourself.

I've done it and have lived through it. So don't tell me what it takes to succeed. Because you don't.

CityZen wrote:

You seem to be saying "if, like many startups, you write bad code that's going to be thrown away soon, you don't need to document it".

I would tend to agree with that. However, I'd suggest you don't write bad code in the first place. "Agile" doesn't mean that all code is throw-away code. At some point, assuming your project is going somewhere, you've got to move from prototype to production. The team is going to grow. If you don't start with some decent programming practices, you'll be paying the price down the line, assuming you get there.

I'm not trying to suggest that if you don't write up a full formal header for every function you're doomed. But if you don't point out the weird cases in the docs, you'll be spending hours chasing down bugs that could have been avoided easily had you done so. Write up the most important stuff first, fill in the rest from time to time. Though don't wait until you've got plenty of spare time, since that never happens.

Could the both of you miss the point any more glaringly? It's pretty simple: 20 years of code without 'real' CM. The advice was to add in some version control, task tracking and unit testing to newly checked in code. Not to refactor or to retrofit documentation. How the bleep did you segway into best coding practices for a startup? If very basic CM is incompatible with the scientific process, I don't know what to tell you other than this project deserves all the pain it has coming to it.

One thing to be aware of is that you need to have massive buy-in to change something like this.

A good development model massively slows development speed in the short run, causing small changes to take hours instead of minutes. (Add a formal test for 50 lines of code? Change logs? Commenting 50 lines? I just want to write the code and go home. And I need the change by tomorrow morning!)

Of course, the hundreds of hours that are saved by taking shortcuts don't make up for the thousands of hours lost later, but the key point is that people *feel* the hundreds of hours that they are "losing" by doing things properly *right* now.

So, expect heavy push-back. Also, make it clear to managers that they cannot expect the same level of "productivity" as before. (That's often the biggest hurdle.)

Having (once long ago) seen a manager nearly fired after he adopted good practices because the upper-management couldn't understand that good practices pay off in months, not days, I no longer fault managers for being heavily concerned with short-term metrics...

They call this technical-debt. Technical-debt can kill any software project and it is to be avoided at all times, without fail if possible. That said, your first step there is to get people to even recognize that they are creating technical-debt. Most chaotic environments have programmer, architects and managers that don't even know they are creating it when they do. Usually, if your enhancement starts off with a small book of needing to know the history, you're paying technical-debt off. If you say "Just get it done", you're creating technical-debt. The interest on technical-debt eventually becomes a massive stifling drain on resources that prevents the team from getting anything accomplished.

Your explanation that startups fail because of poor documentation is BS. Startups don't fail because of poor documentation. It's because

A) Bad IdeaB) Poorly written code that doesn't scaleC) Hire the wrong people and don't know when to get rid of themD) Get rid of people that cannot keep up with the changing environment.

The list goes on. But poor documentation is at the BOTTOM of the list.

Your BS explanation that "well, don't write crap then" is like saying ... "if you want to win, run like Usain Bolt".Yeah I know. We all know. But that doesn't mean it can be done.

Nobody wants to write crap on purpose. It's like saying I want to do my worst because I want to underachieve and proud of it.

I'm not going to bother trying to elaborate further because you are clearly ideological. Start your own company, be successful at it, then come back to me and pound your chest and educate me in how things should be done. But don't be a armchair Quarterback and tell me how things should be done when you haven't done it yourself.

I've done it and have lived through it. So don't tell me what it takes to succeed. Because you don't.

You're reading much more into my comments than I wrote. I never said startups fail due to poor documentation. I'm just saying that it takes less time to write worthwhile documentation than it does to track down bugs that wouldn't have been there had the documentation been written. If done right, the benefits of decent documentation far outweigh the cost of creating them.

You may feel free to disagree with me. I'm not giving orders. But I do write code for a living.

First time i hear that some scientist needs a software engineer. Usually scientist can make their own software without trouble.

This case is weird, since they choose an obsolete language and they were disorganized.

Hahahaha no.

When I hear that a scientist is programming, I assume they're making a mess. Unless you A) program as a hobby, B) majored in a software related-field in college, or C) have done a load of reading on programming best practices, you'll start out making a mess for any project of reasonable complexity.

Logic and critical thinking skills (along with some basic knowledge of syntax) are enough to make a program work, but not anywhere near enough to make it maintainable.

First time i hear that some scientist needs a software engineer. Usually scientist can make their own software without trouble.

The very first sentence tells you that they've been happily producing their own code for a long time, but they've brought in a software engineer to help them sort out their mess having realised they actually need the help. For what it's worth, it's really not that unusual either. There tend to be fairly large egos and entrenched attitudes in the scientific community but they're not that stubborn (usually). You'll find software developers working alongside scientists all over the world.

Quote:

This case is weird, since they choose an obsolete language and they were disorganized.

The question only suggests that the language is currently obsolete, not that it was obsolete at the time the code was written

You'll find a lot of scientists still actively using code written in the most bizarre and esoteric languages and some poor blighter trying to keep the systems running that it was written for. I know a sysadmin who has to maintain a system with Turbo Pascal on it, just for one critical piece of code no one seems interested in rewriting. I've no doubt in 20 years time someone will suddenly find themselves having to support something written using SciPy 0.6 and wondering if it's still even possible to get hold of Python 2.4!

It's not true that it takes less time to write worthwhile documentation. Documentation take ALOT of time. Crappy documentation is worse than no documentation because it can lead to misleading statements causing the developer to be even more confused.

If you actually worked in a startup company, code churn is daily. Are you going to rewrite documentation over and over? Nonsense.At the end, all the documentation will do you no good if the code is crap to begin with.

I can give a new grad all the time in the world to write code and document the shit out of it, but I'd rather hire a senior engineer with good coding skills with minimal documentation. The world have changed with the web. Development time is reduce to such a great degree that it makes no sense to document unless you are running a very large project which involves multisite teams or giving it out to external customers. Even then, it depends on how often the code churns.

Everyone complains about Facebook API having terrible documentation. But no one is stopping development and the company continues to churn out new features. Do you see websites use Facebook Plugins? The stuff works. Clearly, their code scales, otherwise, nothing would work and everything would come to an halt.

That's what happens in real life.

The key is right good test cases and hire the right staff. Minimal documentation is all you need.

I'm not trying to say that that's the only way things should work, but go ahead, document the shit out of your code. But I can bet that you are going to lose to another developer just as talented, if not way more talent than you doing the same thing, but with better features. People go with what works best, not whether their documentation is great or not.

Just my 2 cents.

CityZen wrote:

You're reading much more into my comments than I wrote. I never said startups fail due to poor documentation. I'm just saying that it takes less time to write worthwhile documentation than it does to track down bugs that wouldn't have been there had the documentation been written. If done right, the benefits of decent documentation far outweigh the cost of creating them.

You may feel free to disagree with me. I'm not giving orders. But I do write code for a living.

It's not true that it takes less time to write worthwhile documentation. Documentation take ALOT of time. Crappy documentation is worse than no documentation because it can lead to misleading statements causing the developer to be even more confused.

It takes almost no time to write a comment above a function documenting what the function is, what its arguments represent (and any restrictions on them), and what output it produces. In a scientific context, you might want to reference any research the code is heavily based on. In old code, relevant gotchas and known bugs^Wfeatures. As documentation goes, it's not much, but it helps enormously when reading code. Changing the high-level operation of a function, its inputs or outputs should be a pretty major change anyway, so hopefully such documentation shouldn't need to be updated often.

Quote:

If you actually worked in a startup company, code churn is daily. Are you going to rewrite documentation over and over? Nonsense.At the end, all the documentation will do you no good if the code is crap to begin with.

This isn't about crappy startups who will die or be acquihired within a year. This is about a decades-old codebase.

It's not true that it takes less time to write worthwhile documentation. Documentation take ALOT of time. Crappy documentation is worse than no documentation because it can lead to misleading statements causing the developer to be even more confused.

It takes almost no time to write a comment above a function documenting what the function is, what its arguments represent (and any restrictions on them), and what output it produces. In a scientific context, you might want to reference any research the code is heavily based on. In old code, relevant gotchas and known bugs^Wfeatures. As documentation goes, it's not much, but it helps enormously when reading code. Changing the high-level operation of a function, its inputs or outputs should be a pretty major change anyway, so hopefully such documentation shouldn't need to be updated often.

Quote:

If you actually worked in a startup company, code churn is daily. Are you going to rewrite documentation over and over? Nonsense.At the end, all the documentation will do you no good if the code is crap to begin with.

This isn't about crappy startups who will die or be acquihired within a year. This is about a decades-old codebase.