Monday, December 04, 2006

I hope you're happy, taxpayers.

I am sulking, because they have me doing a stupid repetitive task and I can't figure out how to either automate it or kick it down to someone less expensive. I keep mentioning that, and they keep saying some crap about wanting it done right.

The thing that especially bothers me is that if they would just let me listen to This American Life, I would actually look forward to an afternoon of zoning out and grooming data. But, no streaming media for us. I am cruelly used indeed.

You could always contract the work out to me. I'll do it for $100/hr. I'm guessing it's an excel spreadsheet? That's easy to deal with, though it's been a long time since I've dealt with any automation in WIndows.

I am a social scientist who is currently a doctoral student in a highy computational PhD program, and even though I have taken classes in C, C++, Java, and C#, I am *never* going to be the kind of person who will learn a "simple language" for the purpose of making hypothetical tedious tasks easier. My brain is just not built in the way that certain other peoples' are, where the task of creating little scripts to do gruntwork is enjoyable and rewarding.

That said, Megan, I think what you REALLY ought to do is find yourself an undergrad whose brain IS built that way, and get him/her to automate the automatable parts of your task for cheap.

Oh Megan, you think we only write code when it is faster than the rote task that code replaces? Hell no. I'll spend an hour fine tuning code that will accomplish a task that would have taken me about an hour to get done.

Quirkybook: half your problem is that the four languages you list are a) very close cousins b) quite low level and c) with the exception of `c', somewhat nasty. Things don't have to be that tedious, really.

A cleaner, higher level language is a lot easier to get things done quickly it, even if it may not be quite as efficient (which often doesn't matter, anyway). Even something as half-assed in design as perl or python or VB will be much, much more efficient in programmer time to whip together little tasks.

Ack, please no language arguments. It only turns off the newbies. These poor people only want to solve their problems and you're asking them to evaluate something extremely complex like language tradeoffs before even getting started. That said, S is right: whoever told quirkybooks to start with c-derived languages ought to get hit with a cattleprod.

The truth is, if Megan's using Excel, then the tool of choice is <holds nose> VB. Ack! I have become a tool of the evil empire. Luddite that she is, Megan probably will never make it to Haskell, as lovely as that would be.

Also, moving data in and out of Excel is asking for various kinds of subtle but deep pain. Question: what are the quoting and escaping rules for Excel CSV? Answer: it depends! Was 1900 a leap year? Those of us using the Gregorian calendar would say no, but Microsoft begs to disagree! Blech. Once you start to move beyond Excel it's best to leave it behind entirely.

If only those poor Excel using technophobes knew the kind of evil they're perpetuating. Try to explain it to them and all you get is a funny look. Believe me.

"I can't figure out how to automate it" is a hint that it may be time to move beyond Excel.

All that said, I believe the solution for Megan in this case is to silently rebel by spending more work time writing blog posts. Tell them you got bogged down by all the tedium, so it took you a whole week to do. "So and so can do it in an hour? Wow, sounds like they're the go-to person for this kind of thing." That's the only way the powers that be are ever going to learn.

Megan -- get out the holy water, crucifex, spikes and garlic - there's somebody here who knows Haskell. I speak from first hand experience, I've not only coded in it, I was forced to beta test it. The language is perverse. Whoever else that is must be pure evil.

The only thing that really worried me was the excel. There is nothing in the world more helpless and irresponsible and depraved than a man in the depths of an excel binge, and I knew we'd get into that rotten stuff pretty soon.

I love Excel every day of the week and twice on weekends. All you sophisticates might be all "Oh, it is sooo limited, and people don't understand real power.", but most of the time, Excel does me exactly right.

TJ and I had a silly argument about this automation concept. He likes to automate things, and I got mad at him for implying that I _don't_ and that I'm some kind of neanderthal that likes to hit things with rocks until they fall into place. The truth is that if I was a computer science dude, I too would automate things because I would know how. And I would do it out of habit, like Justin and TJ do. Because it's fun and a little challenge, and all that.

But sometimes it's a trap, like in the situation Amanda describes. Or, like in the following scenario wherein TJ decides to spend an entire weekend writing a Sudoku-solving program -- because Sudoku is a GAME whose only purpose is to entertain a person during the process (the result of the game is useless to all). So he writes the program and perfects it.

The next time we're on the plane (the only place either of us does the Sudoku in the magazine), he says, "Let's race and see who can do it faster." And then he pulls out his laptop and starts feeding the puzzle into it.

Me: Um, you're going to use your program?

Him: Yeh!

Me: I don't think we should race, then, because you already spent the weekend writing the program, so you're two days in the hole as it is.

But I'm sure that writing the program was more fun than doing the Sudoku. I've considered writing such a program myself (didn't because they are already publicly available, and it is less fun if you know you are reinventing the wheel).

Many years ago, an ex-boss engaged me for a few hours to write a puzzle solving program for him, to help impress some telecom executive who had bought the puzzle for his son, and could NOT (after months of trying) put it back together.

The puzzle was a Soma(TM) cube (think of a 3x3x3 cube made of 7 pieces, 6 of which are made of 4 unit cubes, and one of 3) on steroids. It was a 4x4x4 cube made of 13 pieces (12 of 5 unit cubes, 1 of 4). I think it might have been called a "Bedlam cube".

As a test, I had my crude program solve the Soma cube. It took less than 5 minutes (this was back in the 80286 days, or 386 at best). Then the larger cube....several optimizations later, TWO computers succeeded in solving the puzzle after TWO DAYS of crunching.

Haskell does have toilet-cleaning functionality. It's all in the IO monad (badump, ching! I kill myself).

I suppose I wouldn't want to beta-test Haskell either, but you have to agree that they're doing some very cool stuff.

Megan: I agree that Excel is the appropriate tool in a variety of situations (if your data isn't too big and doesn't have a lot of structure, if you don't get affected by the occasional odd legacy behavior, etc.) The 1900 weirdness doesn't affect you? Good! How do you know?

That kind of question is extremely difficult to answer once you have a large number of spreadsheets.

I think we'd all agree that moving beyond Excel would involve a lot of work. The next step is quite tall, and the effort involved is more than most people are willing to put in, and that may be the rational choice in most cases. Plus, if you leave Excel behind your coworkers will all think you're snooty or something.

I'm sure that with your Excel hammer in hand most calculations appear very nail-like. What's the benefit that you'll get from learning to use the power saw? It depends on how much carpentry you're going to be doing, and it depends on how often you find yourself attacking boards with the claw end of the hammer.

Of course, I've found it very rewarding, but then again my problems are not yours and vice versa.

And then there's the reason that most people use Excel: everyone else uses it! The network externalities are strong enough to perpetuate a sub-optimal solution. This too is rational, at least in the short term.

Honestly, I don't expect you to give up Excel. Clearly you enjoy it, and why would I want to take that away? The fact that you can use it to solve problems gives you a feeling of power, just like my tools do for me. That may be overstating it a little but it's true.

You said that you weren't sure how to automate your problem; maybe this doesn't happen very often but if it does, then the advice from the people here who automate things for a living may be relevant to you. Of course, we're all going off of extremely limited information; for all I know your problem does in fact require human judgment, though that set of problems does seem to be dwindling.

The next time you're in town I'd be happy to introduce you to my python, er, I mean Python.

I am a Perl programmer and absolutely love how I can make the language do anything for me, but if I needed to check my email I would just use my Cingular 8125 (has a web browser). Sometimes it's easier to look to another technologies rather than to compilicate things with programming languages.

Jens - yes, you are right. Writing the program is fun. But I just wanted to complain about the fact that he was rubbing his programming knowledge in my face by pulling out the computer and trying to RACE me doing the puzzle! Face it, the guy's a JERK.

Oh, I agree. No language arguments --- just trying to correct a mischaracterization before it took root.

dubin: That's true of course, sometimes people fiddle with automated tasks far longer than it is justified. On the other hand, it sometimes works the other way. I once wrote a short script in less than 5 minutes that reproduced 2 1/2 weeks of a grad students work in a couple of minutes. She hadn't even realized it was possible. Since she had another month or so to go at that rate, she was pretty happy about it.

As far as Megas actual problem goes, if it's Excel, VB is the obvious first cut at it. Eventually you'll hit diminishing returns with something like Excel, but no sense going elsewhere until it's actually needed. You'll have to ask yourself, Megan, if it is worth the week or so of work to get the basics down.

That's hardly fair s. The week or so of work? Learning VB will take a few days, sure, but it's a one time cost. Then, once you know one language, they're all the same, you can pick others up quickly, generally.

And, once she knows VB well, anytime something needs automated she'll knock it out much more quickly, maybe in a matter of hours, saving days of work.

Ok, ok, guys. I'm kidding. I respect that TJ wrote the Sudoku program. I was just trying to tell a funny story. I was kidding about him being a JERK. In reality, I HEART him. Even when he's trying to get my competitive up by asking me to race a computer, which is just mean. :)

And for the record I like the idea of automating things but I haven't taken a computer science course since high school. (Megan, did you take Dr. Paschul's Pascal class? Remember how he wore that fanny pack every day and tried to get us to enter all those competitions, which we did enter and thus contributed to his glory as a fanny-pack-wearing highschool computer science teacher? That class was pretty fun, actually.)

Justin: Not really unfair at all. She has to decide whether she does enough of this sort of thing to spend the time, or if that time is better spent elsewhere (or if hiring out is best). It isn't always a net gain to learn these things. It's also worth noting that the few days to a few weeks isn't only spent once, if you don't keep using it to some degree. If coding becomes part of your daily grind, it will stick. Otherwise, 6mo later it will almost be back to square one. Nitpicking now, but the language transference you speak of doesn't hold --- coding paradigms transfer, but languages don't.

Without knowing a lot more detail about what she is actually doing we can't say much with certainty. Bob_v is sort of on the right track there, except nobody was really talking about data formats and you can say some useful things about language design without knowing the problem domain.

Dude! The Teej is totally funny, and saying that his program is going to race you in Sudoku is totally attitude and teasing you. Props to him for fronting you in a clever way. I HEART him too, in an appropriate way from a friend.