Epoch Status Update

It's been a fairly long time since I've posted here, for a number of reasons; but I felt like it was time to post some of my recent thoughts on the Epoch language project and what's been going on there.

Things as they stand nowThere have been only two notable bursts of activity on the project in the past year. One was at the end of 2010 when I put a lot of work into core language features; unfortunately this then again languished until July when I shipped Release 11. Following that update, it was painfully clear that the compiler for Epoch was prohibitively slow. In early July 2011 I started taking a hard look at how to optimize the compiler and get compilation times down to something reasonable; feature progress essentially halted as I hacked through a bunch of the parser and more or less decided to rewrite it entirely.

In the process of redoing the parser, I bit the bullet and also rearchitected the entire compilation pipeline, including the back end code generator for the VM. While the bytecode format remains unchanged and the VM itself is virtually untouched at this point, there's still a huge amount of churn involved in replacing that much logic. The code has been poked and prodded a few times since July, but is largely sitting around to rot right now.

At this point, there are four major tasks left in the optimization process that started over six months ago:

Finish semantic checking system

Implement a large part of the code generator which is still incomplete

Profile and optimize the new compiler architecture

Examine the VM itself for possible low-hanging performance gains

All of this represents a tremendous amount of effort and investment. I'm convinced that it will be worth it, mainly because the 1000-fold increase in parser speed I managed earlier this year gives me a lot of hope for similar gains in the remaining code. Epoch has always been targeted at interactive (if not truly realtime) computational processes, and having a slow tool pipeline just plain runs contrary to this goal. Iteration speed for Epoch programmers is also a serious concern; I'd eventually like full REPL support in the IDE and a minimal-overhead compilation model, both of which will require the language tools to be far more efficient than they have been in the past.

Things as they will be, going forwardClearly, there is much left to be done before Release 12 will be ready to ship. Just as clearly, work is moving at what could generously be described as a snail's pace.

At this exact moment, I'm a bit torn on how to proceed with the project as a whole. Enthusiasm for the concept has more or less evaporated in the five and a half years since the original thread that spawned the project (note: please forgive the formatting, which seems to have been eaten in one of the forum transitions since the original post). There has never been any substantial external support for even the idea of the language, let alone any other contributors to the effort at large.

It's painfully tempting at times like this to just chuck in the towel and call it a day; I'm amply busy with other interests and my full-time occupation. This fact is primarily to blame for the lack of forward progress on Epoch in the last several months. I still strongly believe that a language like Epoch is critical to advancing the state of software development as we hurtle ever further into the vast expanse of multicore computing.

What I'm not so sure about... well, two things.

First: I don't know if the world is ready for this yet. It would be nice to have something old and mature enough to step up to the plate when we finally realize, as an industry, that languages like C++ just aren't going to hack it anymore. I would very much like to be able to switch to programming in Epoch when that day comes, and that obviously necessitates having a rich and ready-to-go ecosystem to switch to. But for now, the pain of multicore seems bearable - even if I personally feel that the survivability is illusory at best.

Secondly: I'm not sure if I personally have the motivation to be involved in the push for such a technology, at least at this particular point in time. In an idealistic world, I'd love to see this come to fruition; if I could just press a button and have a finished Epoch implementation ready to go by tomorrow, I'd be hammering on that sucker like nobody's business. There are those who would argue that this is enough of an indicator that the goal is worth pursuing; at other times in life, I'd be inclined to agree.

Right now, though, it's depressingly hard to even want to continue the project.

It's been a fun ride, and I don't regret any of it. To be sure, the effort has forced me to learn more about programming technologies than I ever realized there was to know. I can even credit a lot of my career success to the lessons I learned in the course of building what little slice of Epoch has been completed to date. And there are certainly many more things to learn and discover. I daresay that on many frontiers Epoch would even be leading the cutting edge of programming systems if it had the chance to develop that far - so the discoveries are not limited to only things that are already known, but could well lead to all new discoveries entirely. Whether that research and progress is my own work or simply inspired by it is mostly immaterial to me, but either way, the untapped potential is certainly alluring.

Perhaps in another few months I'll find the energy and willpower to dig back into the code base and start hacking on things again. Right now pretty much everything is blocked on the fact that the code generator just doesn't exist in any usable form, and the semantic checker is crippled at best. It's discouraging to look at how badly dismantled the compiler has become, and look at how much had been working previously. There is certainly no small temptation to shelve the optimization project and return to feature development, although that would almost undoubtedly be doomed to failure, as developing features in a compiler that slow is just unbearably aggravating.

There's also a temptation to start looking for other contributors to the project, but if I'm completely honest with myself, I wouldn't want anyone else mucking with it at the moment anyways. I have too many clearly defined yet unrealized ideas about how the language needs to work to start adding chefs to the kitchen. I suspect that when the time comes to start recruiting other core project developers, it will be mainly for things like library implementation and tools support, rather than core compiler architecture and language design.

All things considered, I'm really not sure what to do with the thing at this point. Perhaps I just need some external voice of interest to rekindle my own desire to work on it. Perhaps I need to walk away for a while, maybe indefinitely, to recharge and decide just how committed I am to completing the mammoth task of building a complete programming language from scratch. And perhaps there are other solutions to the dilemma which simply fail to come to mind at 3 in the morning.

There has been scattered idle interest in the language here in the past, although it has waned to virtually nonexistent over the five years since the pragmatic language thread died down. Outside of this community, though, I've found almost no footholds whatsoever. A few people I've talked to about it in various venues have vaguely agreed that it's a nice idea, but there's certainly nothing close to an eager audience waiting impatiently for updates. All of that makes it difficult to gauge just how effective the effort really might be, and whether or not to continue at all.

Anyways... I had hoped that by blathering all this out I would reach some kind of conclusion for myself, but it seems that all I've managed to do is wrangle a bunch of words and more or less just whine about my existential crisis.

Wow...this was kind of depressing... I myself am trying my hands just now at developing a language and a compiler and I'm coming across the same difficulties you metion here: scarce free-time to work on the project, not finding an audience who are interested in helping or just giving feedback, and the worst of all: the nearly uncomprehensible work that has to be done (language spec., compiler, standard library, other tools...). How did you manage to keep working on Epoch for over 5 years? P.S.: Don't give up!

Likewise, though I had less interest and less clear success. Again, my career has given me plenty of interestingly challenging projects that just occupy my time and focus when I could be working on Tangent. Thankfully I treat it more of a hobby. I think it would be useful if a language like it existed, but I'm confident in my own inability to accomplish anything approaching what I'd consider practical; and certainly not anything the PLT community would respect.

Im sure it must feel like a chore to pick up working on a massive beast of a project like this with no light at the end of the tunnel, but what about a change? Maybe a smaller, more concrete language that will still aid in the overall development of Epoch without all the boring stuff. Pick a few features and only work on those and throw away any expectations that may or may not result from it.

If you dont feel like working on the compiler, you could always write articles about the process. I have been thinking of a bare bones stack based VM that I could implement over the next few weeks, but good articles that come from people with experience on such subjects are hard to find. I would almost pay you a subscription for weekly write-ups about various aspects of compiler design!

Actually, writing some articles is a great idea. It might be just the ticket to refreshing my perspective and enthusiasm for the possibilities of what Epoch could become. Has the great benefit of crystallizing some of my existing thought processes and documenting how things work, and might just be beneficial as an educational resource to boot.