Saturday, April 30, 2011

The first part is a journal of thoughts as they were happening, so try not to laugh at me too hard. The second part is a bit of reflection on the project so far. This is a summary/record of my week participating in the 2011 Spring Lisp Game Jam. Hopefully, it's useful to someone (the very act of writing it was very useful to me).

April 23

I did the first public facing push of this code-base at about 4:00 am my time today, and then curled up into bed. When I got up at 10:30, there was already a bug report waiting for me. Not off to a spectacular start...

The problem turned out to be how cl-css handles file compilation. Specifically, it fails if the directory it's trying to compile to doesn't exist. At this point, I'm ready to load up cl-fad to whip up a simple fix, except that cl-fad doesn't have a function to create directories. Okay... maybe it's a Lisp primitive then? A quick search for "directory" over at the hyperspec symbol index turns up nothing. So...huh? I spend 10 minutes checking the cookbook and googling fruitlessly[1]. That doesn't discourage me though. I commonly run into situations where I have a naive mental model of a given problem that some Lisper has already built an elegant solution to. So I google the more general case common lisp making directory, which turns up a lispworks doc page abut ensure-directories-exist.

This is what I meant. When I'm making a directory, I really just want a place to put some specific file. Abstracting the basic directory creation means that I don't have to special-case the situation where multiple nested directories are required, and I also don't need to worry about checking whether the directory exists before creating it. Elegant. Maybe this is a good start. Day 1 and I'm already learning something.

April 24

I went pretty batshit on the code-base today, removing anything that looked like an unnecessary additional step. I'd been keeping the model and view as separate as possible because of an assumption that there would be some sort of SQL database involved in the game eventually. I may still need to switch to one at some point, so the move might come back to bite me in the ass, but the barriers got broken down. Mostly it was intermediate functions all over the place that slightly simplified the next set of intermediate functions[2]. Things like planet-info and inventory which took model structs and returned plists instead. To be fair, that actually did help at some point; since there doesn't seem to be a way to map over a struct, it was very easy to dump an entire plist into the view with a loop.

Anyway, I made the decision during my shower that I'm keeping the struct approach since I think it might help with the concurrency issues I'll be facing next. There's also one decision from earlier that's sort of coming back to haunt me there. The way that a given captain's current planet is stored is as a string of that planets' name. That meant that I needed a function called planet-name->planet early on, and it meant that any manipulation of the current planet happened directly on the *galaxy*. It's fine for a single player, but more would cause some odd errors that wouldn't be particularly fun. The way I'm thinking of solving that is by having each captain copy out the current planet, and keep track of transactions they conduct. The total changes would then get applied when they travel. A given market could still dip into the negatives in some circumstances, but it should serve to make it much harder to force an error on an opponent sharing your planet.

Finally, markets behave differently now. Instead of static prices, each market generates a new price from the *tradegoods* and averages (By which I mean "gets the arithmetic mean of") that with its current price. It just makes sure that markets tend to stabilize over time, while hopefully keeping enough fluctuations to make profit possible.

April 25

Changed up a lot more of the system. Today, I focused on the purchase!/convey! end of things rather than the basic model. Basically, captains now have a copy of the planet they take with them and merge transactions back later. As I mentioned yesterday, this isn't a measure to make sure that no more product leaves a planet than absolutely should. In fact, I'm tempted to make deficit exports an explicit mechanic, upping the price of a good fairly substantially if it slides into the negatives. That would also play off a tweak I want to make to the market-produce! function. Namely, generating (+ 4d20-30 (/ productivity tech-level)) instead of the current (+ 2d20 (/ productivity tech-level)) to allow goods to be consumed as well as created by planets[3].

Today's lesson[4] was that state is fucking hard. Introducing side-effects to a primarily functional system played more hell with my code than I thought it would. I still don't know if I squashed every possible bug with that transaction system. It's enough to make me go paranoid and start sprinkling asserts in the vicinity of all setfs and !s. I won't yet; I'll wait 'till something unexpected blows up, but I can see an argument for it here. The end result is that this game should now be playable by more than one human at a time. There's no explicit goals yet, I'm going to save that for later, I think, but you can still get a few friends together on a lan[5] and play economic hell with a small galaxy with the game as it stands.

[Break time to remodel my kitchen. Did a bit of light view coding, but nothing interesting to report.]

April 28

Spent today mainly on the UI end. Making it pretty-ish and squashing a few display bugs I came across. Still not happy with the default theme, but it's all I'll have time for in a week. I did make an effort to completely disconnect it from all other code, so others can theoretically just push a folder of images+theme.css to add a new one. The entire experience was a bit surreal; in the middle of using GIMP to do some graphic design work, I found myself embroiled in a conversation on Reddit about how GIMP is not good enough to do design work. Since the best argument I read was "But, I like Photoshop!" I'll stick with the free[6] version, thank you.

Part of that UI tweaking mentioned above was adding jQuery and jQuery UI to the codebase. I didn't want to have to tell people to go download it themselves, and there's really nothing that needs to be done other than putting them in the correct directories. The downside is that this is now considered a Javascript project by GitHub. Which, ok, I guess is true by character count, but it still feels inaccurate.

April 29

I realized earlier today that my time's officially up for this little project[7]. Technically, time ran out about an hour ago, but I kinda got into the swing of things and decided to finish one last TODO before doing a final check-in for the event and collecting my thoughts. That ended up taking more time than I thought it would, but it really was my own fault for giving into indecision for so long. The game looks passable, it plays nicely, doesn't seem to blow up[8] and is actually pretty fun[9]. I really shouldn't have picked out something this ambitious for a week I knew I'd be occupied, but hey. Such is life, I guess. I'll keep working on it for as long as I can stand to look at the code-base, but for the moment, it's done.

Anyway, lets get to the juice; here's the distillation of what I learned in trying to put together a small-ish game in Lisp over the course of a week.

Incremental beats planned...

It was honestly surprising how true this turned out to be. I started out trying to plan out as much stuff as possible into the future, going so far as to put several levels of indirection into the model in case of a switch to a relational database later. That... was a tremendous waste of time. Both in the sense that useless code was produced as a result, and that I then had to spend time going through call-trees to figure out what a given function was actually doing. I got fed up with that fairly early on in the week and went on an already noted rampage through the model, deleting everything that I wasn't actively calling right now, preferably in multiple places.

Overall, the time spent just diving in and writing code resulted in a lot more of the final code-base than carefully plotting out where I needed to go[10]. That's probably partly a result of the plans not being comprehensive enough, and partly a result of the plans changing mid-way. I'd argue that's healthy though; there are some specific interaction points that I'd have been hard pressed to predict in advance, but that became perfectly obvious when I tried to implement them properly.

Don't read this as "don't plan", because it does save a bit of time, but not as much as I thought[11]. My best guess is that there's some point of diminishing returns with planning. You want to do about that much because doing less causes some pretty serious headaches in terms of direction, but if you do more, you're not helping (and are likely hurting). That's just a hypothesis based on my experience, of course, I'd love to see some experimental data about it.

...but remember to fix it up

The other side of the incremental approach is that it ends up producing a lot more code. The main catch is that you need to remember to stay flexible, since you might be changing large portions of the system at any given point. Even so, prospective coding still turned out to be a very powerful tool. Producing more code does mean that you have to periodically measure and cut again. There are already two places where I can see a lack of planning costing me efficiency, but I'm not sure I would have been able to predict that in advance.

Distractions hurt. A lot.

That really shouldn't even need saying, but there you have it. Trying to code up a storm doesn't work out very well when you're also re-modeling your kitchen. I'll know for next time, I guess. The one upshot is that it resulted in a system that's easy to fit in my head basically by necessity, because I had no hope whatsoever of getting an entire 8 hour stretch of time devoted to it.

Deadlines help. A lot. No, more than that.

I've had this project on the back-burner for close to four months. In those four months, I've managed to put together a half-way decent name generator and comb over the code-base for Elite for Emacs a dozen times or so. In the past five days or so, I did the rest of it[12]. The deadline helped get me in gear where I probably would have sat indecisively for a further few months. Even with all the distractions, I easily did four times as much work since the beginning of the game jam as I did since starting on this.

Even more than the deadline, the fact that I had to look at this from different perspectives helped a lot. It wasn't just writing code, there was also a lot of brain power spent on thinking about how I would explain the code, what it meant in the grander scheme of the project, and whether there was a better way of doing it. It's a bit counter-intuitive, but it seems that reflection (thinking about things you've just done) is a lot more useful and a lot more productive than planning (thinking about things you're going to be doing soon). I didn't end up explaining it very well regardless, but the process of thinking up said explanations seemed to help in the construction regardless.

I don't know enough

Probably the biggest one. I'm not sure if any hypothetical reader can apply this, but I sure as hell will. I've got a lot of learning left to go in pretty much every direction. First, I get the feeling that most of the model could have been put together much better using CLOS than my current approach of structs and functions. For example, planet-produce!, add-to-market!, banned? and local?[13] would have made more sense as methods. I can say that without actually knowing much about CLOS other than the name, and I'll be going through documentation and tutorials as soon as I can squeeze it into my day. The 3D math is also something I'm rusty on. I could swear I used to know this stuff back in high school, but got surprisingly little use out of it since. It would have helped a lot. Specifically, having a clearer understanding of 3D transformation would have let me create an actual 3D interface instead of faking it poorly with layers. I've ... gotta work on that. Lastly[14], I still have no clue how to use loop properly. I gather I'm not alone since it gets its own chapter in both PCL and the CL Cookbook, but it still caught me by surprise how deep the construct is.

So there. I knew I knew little, but it turns out I knew less than I knew.

That's certainly something to fix for next time.

Footnotes

1 - [back] - "I guess Lisp can't create directories." Is what I would have thought about two years ago

2 - [back] - And by "slightly", I mean "almost not at all"; the functions that did do necessary abstraction are still there

3 - [back] - Produced, on balance, but it opens up that scarcity mechanic

Wednesday, April 20, 2011

It seems that I only ever get around to working on this pet project when I'm sick (which I was earlier this week). It's taken almost 5 months at this point, but the hours counter is really closer to ~15, which means that I could have done the work during a single, particularly slow, weekend.

Anyway, moving on, I've been plaing around with the codebase for Elite for Emacs (and there's a post around here somewhere that details some of the blunders it contains). Today, I'm dealing with the next level up; not pointing out where primitives are being misused, but pointing out needless patterns where they don't belong and showing one way of composing them properly. Actually, now that I look at it, I'd better take a single pattern out and deconstruct it lest I bore the ever-living shit out of everyone, including me. I'm also not eliding anything this time, this is going to deal with specifics from the Elite for Emacs 0.1 codebase and how I'm thinking about re-implementing them.

Describing Things

Actually, before I get to that one,

Random Numbers

At a cursory examination, I've found myrand, randbyte, rand1andgen_rnd_number (and no uses of the the built-in rand function). They may or may not do similar things. The author also insists on tracking his own random number seed in a global variable (and re-generating it with a function named mysrand). Here's a sample

I'm not sure why Lisp coders get stick for re-implementing infrastructure if this is reasonably common in the outside world. Building your own byte-oriented random number generator is something a Lisp can do, but[1] you really shouldn't. If you were in the middle of writing your own implementation of rand in Elisp, Common Lisp or Scheme before you started reading this, please just do us both a favor and stop.

What it does in context, basically, is take the string "\x8F is \x97." and expand it out recursively until all the "byte" references are gone and it ends up with a little semi-sensical, explanatory description like "The planet is reasonably famous for its inhabitants' ingrained shyness but scourged by deadly edible wolfs." or "The planet is famous for its pink parking meters.".

The non-lisp thing I hinted at last time, and wanted to discuss this thime out, is this idea of byte-orientation. This is an architecture built by someone used to assembly or C, that then tried to shoehorn the same way of looking at the world into Lisp. I wouldn't mind so much, but it's far too easy to imagine someone hacking together a system like this and thinking to themselves "Wow, this really sucks. I could have done it MUCH more efficiently in C, and I wouldn't have had to deal with all this 'list' and 'setq' nonsense. I guess Lisp is just a language for masochists...". Going against the grain of any language creates the impression that it's less powerful than it really is, and this is a prime example.

I took a minute out at the beginning of this post to point out how this codebase re-implements random number generation at a very low level. Well, the reason I consider it an egregious mistake here is that the main place I found that particular generator used is in goat_soup above. In other words, the coder set up a list (and associated various bytes with each element), then used gen_rnd_number to generate an appropriate byte and then pick out an element from the resulting list. The really funny part is that I could see this being implemented as a performance optimization in a C version of this game, but when you're dealing with string representations of bytes that you have to split and convert before operating on, any performance gains fly directly out the window.

It's beside the point, though. Remember, Lisp is a symbolic language. So here's a Lispier way of generating some planet descriptions.

That's the data, at any rate. The above uses all the original words and combinations from the Elisp codebase (I did add "Australian, indoor-rules" to the activity adjective list, but that's all), so the descriptions popping out of it should be the same as those coming out of goat_soup and friends. Note that the * surrounding the variable name denote a global variable[3]. Note also that instead of using byte relations, the plist approach lets me avoid splitting strings in intermediate steps. A terminal is a string and a non-terminal is an atom. The way to unfold these is

All I have to make sure is that these get displayed along with the planet name and we're golden. This is the sort of elegance that Lisp is capable of when you go with the grain. ~140 lines of flaming death and side effects replaced by two recursive functions and a plist that succinctly and accurately signal the intent of the programmer. I wouldn't be particularly surprised if there's an even simpler way to accomplish the same thing, actually.

Naming Things

Planet names seem like they should be implemented the same way, given what they really are. In fact, I did just implement them as another grammar that depends on the same functions to unfold, but the original takes a different approach.

s is passed as an argument to the planet-generator, the rest of which I won't inflict upon you, I think you may have gotten the idea already. In other words, planets are restricted to 4 syllable names put together by side-effect in the procedure that creates planets. They're put together by using byte operations on a string that represents the valid pair combinations contained in a planet name.

Well, seeing as I already put together a convention for unfolding components to strings by using a recursive function and a plist, I figured I'd do the same for this. Code reuse is good, I hear.

string-capitalize just makes sure it looks like a proper name (it's a lisp primitive, so I won't define it here). The important part, which I'll likely cover in a future post, is that making things functional aids in composeability. The setq sequence from the original codebase has no hope of being reused anywhere because it intentionally grubs about in the surrounding state. If nothing else, the expand-production approach ensures that if I ever need a planet name in some other context, I can easily generate it. Also, as we've seen already, abstracting out the general pattern of "compose strings from a given pattern of components" easily pays for itself with even one instance of reuse.

My Sinister Purpose

The reason I've been picking away at this codebase isn't idle fancy[5], or an intense hatred of Sami Salkosuo[6]. It's that I've been putting together a little web game based on it. It's still not done, mind you, but I'm going to try to put something together fairly soon for you to poke at (even if it's ugly as sin from the visual perspective to start with). If nothing else, I'm tossing the ported Common Lisp codebase up onto my GitHub this weekend so that some other pedantic bore can pick apart a project I was just doing in my spare time. After two articles full of bitching about poor style and inelegant expression, it seems like it's only fair.

Footnotes

2 - [back] - Which weren't particularly horrible, most of them were upholding the time-honored tradition of testing by printf

3 - [back] - It's not enforced except in one or two places, but it is a style convention. SBCL will bitch at you if you try to do something like (let ((*foo* '(bar baz))) ...), for example. Not refuse to run it of course, but it'll warn you that it's using a lexical binding for *foo*.

4 - [back] - That's the first time that my blog-mode highligher chugged for a second before coming back with the result, by the by, take a look at the source code and see if you can see why.

Saturday, April 9, 2011

This won't be a long entry, but I still have to get it down out of my head before long.

I had this 24 inch iMac around for a few years. Got it back when 2GB of ram was a lot and a 2.4 Ghz single core was blazingly fast. Played around with it for a while and it served well for various design tasks. When I switched gears to more development than design a couple of years ago, I also put together a modest Linux machine. It actually started out with worse specs than the Mac. A much smaller hard drive, 1GB of ram and I forget how fast the processor was. Definitely not very; I remember getting the cheapest AMD I could find. It could grow though. I've always liked tinkering and building machines, but having used Macs through my university years made me forget how much fun it was for a little while. Before long, I was tuning and tweaking again. A slightly better processor here, a bit more ram there. Eventually, SSDs got cheap enough that I could afford a small one. The Linux machine began to rival, and eventually surpass the shiny giant.

I kept the Mac as a design machine, just to run Photoshop/Illustrator and (occasionally) Flash. After about half a year of this, I realized that my time was spent primarily in Emacs on the Linux machine and secondarily in a browser/terminal (on either machine). The miniscule remainder was actually using Photoshop/Illustrator. I had also taken up GIMP for smaller jobs, just so I wouldn't have to switch back and forth between computers. It became obvious that as shiny as it was, the Apple desktop wasn't doing much for me, so I gave it to my fiancee. She fell in love with it, not that she would admit that as a former Windows user. And I mean "user" in the sense of "end user". A computer is a tool that lets her do the stuff she's really interested in. She doesn't care how it functions on the inside, as long as it does what she wants it to. In any case, she got quite comfortable with the Apple setup in short order.

Well, earlier today, the Mac died.

She was doing something random with her Kindle when it shut down randomly. It wouldn't come back up, or respond to any of the start-up keys, and it wouldn't boot from its installation DVD either. So we were staring down the OS X equivalent of the blue screen of death. The problem potentially wasn't as simple as the hard drive getting borked. Given that the mere process of replacing a hard drive for one of these units involved supplies we'd have to go shopping for, she decided she'd just switch[1]. She wouldn't go back to Windows, so I got her onto Ubuntu without much convincing.

Ok, that was the background. Here's the story.

I mentioned being a graphic designer. Actually that should probably say "Graphic Designer" because I have the degree to back up those capitals. So when I say that I underestimated the importance of UI on the decision making of end-users, understand the implications.

She was intensely disappointed by pretty much everything she's seen of other Linux machines, so I more or less gave up on turning this into a house that respects the four freedoms [2]. The thing that ended up placating her is, and I shit you not, cairo-dock set to auto-start at login with the spaces widget removed and the trash icon enabled. OpenOffice is close enough to MS Word for the stuff she does, Rhythmbox syncs with her iPod and her browser of choice is available. She still wants Photoshop, so I may need to resort to some VirtualBox shenanigans, but I'm hoping to get her using GIMP instead. Those are nitpicks though. It turns out that dock and the iPod syncing were the deal-breakers.

This isn't meant to be funny, by the way. It's here to serve as a reminder of how small a change it takes for an end user to willingly switch.

Footnotes

1 - [back] - I've got some extra supplies lying around. Nothing terribly impressive, but still enough to put together a half-way decent backup machine in an hour or so if I need to.

2 - [back] - It's still not incidentally. As I said, she owns a Kindle, and an iPod from a while back, but our desktops are now all open source software and commodity hardware.

Ruby and Erlang each come with their own modes, and recent Emacs versions ship with a built-in Python mode and shell. Smalltalk uses its own environment (though GNU Smalltalk does have its own mode), and I'd really rather not talk about PHP. If you're writing in it, chances are you're using Eclipse or an IDE anyway.