Meet the New Problem… Same as the old Problem

“Investing a large amount of resources into creating a tool or language to simplify a particular problem never makes much economic sense precisely because the problems that exist today are never the problems that exist tommorow [sic].” — ocean

No. This is just completely wrong.

In the first place, I’d just like to ask why we have all of this mind numbing unceasing tech churn? Evidently there is some economic sense to this whole language/tool creation thing. I mean… even the obvious examples like Java, VB, Erlang, XML, NAnt, SubVersion, etc. should settle this. To be a software developer is to be in a constant state of assessing and evaluating tools. This is about as “mucky” a problem as they come, and where there’s muck there’s brass… so there’s an entire industry striving to meet this need.

But maybe I’m taking ocean’s remarks out of context here. Let’s blur our vision for a second and pretend that all the holy wars and competing tool sets have gone away. All we’re left with is just a typical man on the street developer trying to keep up with a huge number of change requests and bug reports. Here’s a snippet from a random job posting:

“Creating web application ordering systems for many different clients (each client has a unique ordering process) -All new development work is in C#, dozens of existing applications are in Visual Basic .NET -Candidate must be able to switch between languages without issue.”

What are we going to tell the guy that ends up with that job? “Hey, we can’t wait to have you here. We’ve got so much to do. But look man, we know we’ve got 50 web pages running here that have a lot of the same code, but don’t even think of making a code library to share between all of the projects. We tried that once and it just didn’t work.”

Are we going to look over the guy’s shoulder and randomly pipe in and say, “Hey… stop that. We decided last year that refactoring was a waste of time. Take all of those carefully named subroutines and pack them all back into one giant procedure like we had it before.”

Are we going to cry “foul” if he starts coding up a macro for his text editor?

I suppose we should fire him if he starts working up framework or a templating system to use as a basis for all of his “store” projects. And if he even toys with the idea of a human readable custom scripting language for handling certain triggers or events we should go ahead and just burn him at the stake.

Shoot, to be consistent, let’s ban OOP. A sufficiently sophisticated object model starts to look like a language anyway… let’s outlaw the use of the .Net framework as far as we can.

Okay. Alright. We’re not going to do any of that stuff. We’re not going to just randomly start banning the nuts and bolts of basic software development. Of course there’s an entire strata of developers that will never develop their own object model to solve a problem– even though they spend all day poking around in other people’s object models. There’s an even larger strata of developers that will never make up their own language to solve a problem– even though they might need to learn a dozen just to work a lame web project. But just because we aren’t comfortable with metalinguistic abstraction, it doesn’t mean we need to shoot down anybody that wants to explore its possibilities. And at some point, there were folks that had some computing problems and then wrote C and awk in order to get around them– so by definition, a new language/tool has to be the right answer sooner or later.

But if you look back at that job posting and think about it… that’s almost exactly the same problem that Paul Graham solved back in the eighties working on what became Yahoo Store. It’s ten years or more and we’ve still got pretty much the same sorts of problems. And if you’re in the business of solving a certain class of problems, you’ll want more of them, too. Why not invest in developing tools/frameworks/languages that can help you manage more problems with less effort? Why not continue investing in those tools so that you can expand the scope of the problems that you consider trivial? If you insist on working crappy development projects, you’re sure to get away with it for a time. But somewhere, someone is working on the tools that will make most of your painstakingly crafted custom solutions irrelevant. They are doing everything they can to make it easy for the people that write your paycheck to ditch you and move to their system.

10 Responses to “Meet the New Problem… Same as the old Problem”

Good points. Though I feel like pointing out that developing your own simple language to solve a specific kind of problem you work with often leads to releasing that language/tool to others who decide they need it to do a ton of general-purpose-programming kind of things you never intended, and it builds up an ugly pile of hacks, and eventually you have PHP.

Your argument doesn’t make much sense. My proposal to avoid creating your own tools is an argument **for** reuse. A developer who recognizes that his product will be very different 6 months down the road would realize that it makes much more sense to adopt other’s tools and libraries rather than write his own. That’s the reason developers are constantly evaluating tools: they are constantly solving problems and don’t have the time or resources to write their own tools. Try to imagine the extraordinary amount of work that goes into creating languages like Java or VB and then consider the resources available to your average developer.

The question here is not about the stupid strawman of whether developers should *ever* create their own tools and languages. I even point out the conditions under which it makes sense to write your own DSL. The question is about whether creating new tools and language is the sole purpose or “highest dream” of software development as Warfield suggests. And, as I suggest, the reason we don’t go about creating tools willy nilly is because (1) common sense. Spending a lot of time solving problems that will change tommorow doesn’t make much sense. (2) As engineers our principle tactic is to take new problems and reduce them to previously solved problems. This is what engineers actually do and in practice it has very little to do with inventing new paradigms and exercising our creative juices.

I’m sorry if I took your comments out of context, but I’m not so sure that I did.

You’re repeating the idea that making tools is about “solving problems that will change tommorow doesn’t make much sense.” If people find themselves doing the same task a hundres times a day, they will come to us developers for help. If we do the same task fifty times in a row, we start working on tools to help take the work out of doing that type of task. That’s far from “willy-nilly” and not a case of YAGNI– especially not when your trying to sell a hundred more such solutions to people with similar problems.

Yes, it took a huge amount of effort to come up with Java and VB. Java was made to “solve” certain problems with C++. VB was made to make it easy to write Windows applications without having to use massive numbers of obscure API calls. Huge undertakings. But that doesn’t mean that writing an spreadsheet style expression language for customizing one of your apps is going to be so tough. It doesn’t mean that your only choice for configuring your application is to have a crapload of tables in the database. And it doesn’t mean that there aren’t programming languages, tools, and environments that make constructing new language elements easier.

If we aren’t investing in creating new tools and languages, we’re painting ourselves into a corner and creating a maintenance nightmare in the process. We need every abstraction technique we can get our hands on to stay on top of this.

Ok, I have a better idea where you’re coming from. I will say I’m all for developers leveraging good design, well thought out abstractions and automation. What I’m suggesting though is that good engineering isn’t about inventing new tools to solve new problems it’s about reducing new problems such that they can be solved using existing tools. Insisting that the emphasis of software engineering ought not to be on the creation of new tools isn’t quite the same as saying developers should never invent new tools. I’d suggest that focusing on creating new tools and languages will lead to YAGNI because more than likely you’re trying to solve a problem that has either already been solved well enough or is purely secondary to the domain. I think it’s actually because developers are so intent on creating their own tools and languages that so little real progress is made in development.

I’ve usually seen that developers are so intent on forcing their problems to fit some existing framework or XML-powered whatsit that they lose sight of what their problem looks like. I’ve found that developers who take pains to make their own tools and DSLs are usually competent and successful. What’s really bad is when developers neither use existing tools nor create their own.

I think that a developer’s job is to evaluate existing tools and create custom tools as needed, taking into account the scope and lifetime of the project. There’s a lot of leverage that can be had by creating a language or a framework, and I think history has shown that we shouldn’t leave tool creation solely to the tool companies.

Though I’m not really in the mood for defending .Net I feel the need to correct your premise that a job that requires a programmer to work in C# and VB.Net somehow makes it impossible to integrate the two. This is not the case. So long as they’re compiled into separate assemblies one language can use the other’s stuff very easily. That’s one of the things .Net was designed for.

Your examples of “what are we going to tell the developer?” are based on reality for many shops, but I don’t see the connection to the ad. A lot of programming ads are like this, even if they encourage best practices.

I think your argument boils down to computing architecture (in software); use what’s appropriate. ocean’s basic point is “don’t reinvent the wheel”, I think. Both are valid points, IMO. The thing is our industry has a screwed up way of determining what’s the best architecture for a job. By architecture I mean the language, the VM, and any frameworks that are used with it. Often what gets picked is out of what’s popular, and that’s the sole criteria, not what’s most appropriate for the technical problem. You know as well as I do that the web architectures most places use is overly complex and I think it’s fair to say “rickety”. It’s improved from the days of Perl and C++, but the challenge is still how to keep track of session state. Seaside running in Squeak/Smalltalk handles this well by almost totally eliminating the need for the programmer to keep track of session state. Ruby on Rails does a good job of removing the complexity of modeling a database app. on the web. Though I haven’t reviewed them, I’m sure there are Lisp web frameworks that do a good job as well of managing these things. Here’s the rub–hardly anyone uses these technologies. So what you get is the result of the “worse is better” mentality. So while “don’t reinvent the wheel” is valid, I think we need to be looking at the quality of the wheel before we make that argument. If the “wheel” we’re using is squares on axles, and we’re using that because that’s what everyone else uses, then we deserve what we get…

Mark… we don’t seem to be connecting on this one. I don’t have any axe to grind with .Net’s language integration features or even with the fact that “better” tools are being set aside for “popular” ones.

I also don’t have any problem with the ad or with the people that run shops like that. I was looking for a specific example to give a real context to the discussion. I chose that one in particular after reading though about 6 or 8 postings because it sounded sorta like the kind of pressures that gave rise to Ruby on Rails– and as a bonus it was more or less the same problem that Paul Graham solved back in the nineties up in his attic with a couple of other Lisp hackers. It sounded to me like a situation where custom tool construction even at the lone developer level would pay dividends in a matter of months.

If I was applying for the job, my main question would be who would have “ownership” of the code. Would I be allowed to refactor the code or not?

If I took the job, I would stay in maintenance mode for a while. If I saw repetitive patterns accross the code base, I’d start making a library. I’d review existing tools and libraries to make sure I wasn’t reinvinting the wheel. I’d look at ways of increasing the configurability/adaptability of the classes/routines.

I’d start using the new library on all new development work… and I would consider applying it to the older web pages if I ever had to go into them for a set of major changes or additions. If it’s possible to write Emacs macros to make automate some of the rewriting, I’d do it even if it takes a little longer.

If the library was a big success and if even doing minor maintenance on the legacy pages was a pain, I’d consider sneaking in and changing them some time. If my library eliminated a certain class of bugs altogether, I’d have all the more reason to do this. If I didn’t think I could pull the change off without anyone noticing, I’d hesitate to do this.

At this point I’d look at the code base and start trying to imagine the tool that would make my work even easier. The place to look is… after I done all of the easy refactoring and made my library as powerful as I can… what’s left? Where’s the muck? It’s at this point that we might be entering DSL and/or code generation territory. (If we drink the .Net cool aide, we may even want to look at Workflow foundation here.)

Then we have the phase of dreaming about every aspect of the problem, reading books on the weekends, and building small proof of concept applications. When the idea is hammered out enough, you have a business proposal on your hands: we can hire a “junior” programer to maintain all the old sites if we have this framework– he’ll be able to do so much without very much programming. All the hard stuff is done. I get to work on extending the framework so that we can expand the domain of people that we’re selling to… and we take this business to the next level. Something like that.

The tool vs. “not tool” dilemma is not really a big problem. We should always have a “tool” mindset even when we’re just trying to get stuff done. The framework can slowly grow over time without a huge investment of time all at once… and it doesn’t have to be implemented until it can justify its existence.

Alright. I misunderstood you. It just seemed like you went right from the ad to implying a host of problems with the job. It left me wondering, “Hmm. Perhaps Lispy is a bit jaded and is venting?” I didn’t follow the links. Perhaps that would’ve filled me in.

I agree with you about ownership of the code and trying to make improvements to the architecture (like using your library) over time. My main obstacle with some jobs has been I don’t have the time to do the project right. The budget is constrained by the customer, so even if I wanted to do it up right and had ownership, the budget doesn’t allow it unless I feel like doing it gratis. On bigger corporate projects I’ve been able to sneak in refactoring as part of bug fixing, but I agree, it has to be innocuous.

I think the reason employers get queezy about developers creating their own languages is the prospect that from then on every developer they hire will have to learn it. I think that’s an expectation most employers should have anyway, but in this age when knowing a popular language is the expected norm in business it can be difficult to get over that hump. I think something that can set their mind at ease is if you make a commitment (this may require stressing the importance of this with your employer) to write documentation for the language, and do it well, so that others can learn it easily. The reason I say “stress the importance” is it needs to be a project task. Unless it’s real simple it’s not something that can be written up in a half-hour.

I should mention that another obstacle is the baggage that a language brings with it, like how it flags errors, and how it enables debugging. When you’re creating a language you’re literally creating a new model for computing, so it pays to take a holistic approach to it.

[…] I wasted some time blogging about programming practices and things gradually culminated into a manifesto of sorts. While I now had enough technical know-how to address some of the biggest problems […]