Archive for the ‘Emacs’ Category

I started hacking on firefox recently. And, of course, I’ve configured emacs a bit to make hacking on it more pleasant.

The first thing I did was create a .dir-locals.el file with some customizations. Most of the tree has local variable settings in the source files — but some are missing and it is useful to set some globally. (Whether they are universally correct is another matter…)

Also, I like to use bug-reference-url-mode. What this does is automatically highlight references to bugs in the source code. That is, if you see “bug #1050501″, it will be buttonized and you can click (or C-RET) and open the bug in the browser. (The default regexp doesn’t capture quite enough references so my settings hack this too; but I filed an Emacs bug for it.)

I put my .dir-locals.el just above my git checkout, so I don’t end up deleting it by mistake. It should probably just go directly in-tree, but I haven’t tried to do that yet. Here’s that code:

In programming modes I enable bug-reference-prog-mode. This enables highlighting only in comments and strings. This would easily be done from prog-mode-hook, but I made my choice of minor modes depend on the major mode via find-file-hook.

I’ve also found that it is nice to enable this minor mode in diff-mode and log-view-mode. This way you get bug references in diffs and when viewing git logs. The code ends up like:

Suddenly it didn’t seem all that hard to implement something along these lines, and after a bit of poking around I wrote emacs-module.

The basic idea is to continue to follow the Emacs approach of prefixing symbol names — but not to require you to actually write out the full names of everything. Instead, the module system intercepts load and friends to rewrite symbol names as lisp is loaded.

The symbol renaming is done in a simple way, following existing Emacs conventions. This gives the nice result that existing code doesn’t need to be updated to use the module system directly. That is, the module system recognizes name prefixes as “implicit” modules, based purely on the module name.

I’d say this is still a proof-of-concept. I haven’t tried hairier cases, like defclass, and at least declare-function does not work but should.

The “notification area” is also called the “status icon area” or the “systray” — it is a spot that holds some icons that are under control of various applications.

I was a fan of the notification area since it first showed up in Gnome. I recognized it instantly as the thing I wanted that I hadn’t realized I wanted.

Now, as you know, the notification area has fallen on hard times. It’s been removed in Gnome 3… I searched a bit for the rationale for this deletion, which as far as I can tell is just that some applications abused it, whatever that means; or that it was used inconsistently, which I think the web has conclusively proven is fine by users. Coming from the Emacs perspective, where one can customize the somewhat-equivalent of the status area (see those recent posts on diminishing minor-mode lighters in the mode line…), and where a certain amount of per-mode idiosyncrasy is the norm, these seem like an inadequate reasons.

However, the reason doesn’t really matter. I love the notification area! When I moved more of my daily desktop use back into Emacs (the tides are strong but slow, and take years to come in or go out), I hooked Emacs up to it, and made it a part of my basic configuration.

It’s indispensable now. What I particularly like about it is that it is both noticeable and unobtrusive — the former because I can have the icons blink on important events, and the latter because the icons don’t move around or obscure other windows.

Ok! You should use it! And I totally plan to tell you how, but first some boring history.

My original post relied on a hacked version of the Gnome zenity utility. This turned out to be a real pain over time. I had to rebuild it periodically, adding hacks (once removing chunks), etc. Sharing it with others was hard. And, for whatever reason, the patches in Gnome bugzilla were completely ignored. Bah.

A bit later I wrote a big patch to Emacs to put all this into the core. That patch was rejected, more or less. Bah two.

Then even later I flirted with KDE for a bit. Yes. KDE had the nice idea to expose the notification area via dbus, and Emacs could talk dbus… so I did the obvious thing in elisp. However the KDE notification area was pretty buggy and in the end I had to abandon it as well.

So, it was back to zenity… until this week, during my funemployment. I rewrote my hacks in Python. This was so easy I wish I’d done it years and years ago.

I’m not sure what the moral of this story is. Maybe that my obsession is your gain. Or maybe that I have trouble letting go.

Anyway, the result is here, on github, or in marmalade. You’ll need Python and the new (introspection-based) Python Gtk interfaces. This of course is no trouble to install. The package includes the base status icon API, plus basic UIs for ERC and EMMS. Try it out and let me know what you think.

While streamlining my mode line, I wrote another little mode-line feature that I thought of ages ago — using the background of the mode-line to indicate the current position in the buffer. I didn’t like this enough to use it, but I thought I’d post it since it was a fun hack.

First, make sure the current mode line is kept:

(defvar tromey-real-mode-line-format mode-line-format)

Now, make a little function that format the mode line using the standard rules and then applies a property depending on the current position in the buffer:

We have to do this funny wrapping and “%”-quoting business here because the :eval form returns a mode line format — not just text — and because the otherwise appealing :propertize form doesn’t allow computations.

Also, I’ve never understood why mapconcat can’t handle a character result from the map function. Anybody?

Now set this to be the mode line:

(setq-default mode-line-format '((:eval (tromey-compute-mode-line))))

The function above changes the background of the mode line corresponding to the current window’s start and end positions. So, for example, here we are in the middle of a buffer that is bigger than the window:

I left this on for a bit but found it too distracting. If you like it, use it. You might like to remove the mode-line-position stuff from the mode line, as it seems redundant with the visual display.

At least, it looks sort of like this if you ignore the lamenesses in the screenshot. If you’re like me you probably don’t remember what all these things mean, at least not without looking them up. What’s that “U”? Why the “:” or why three hyphens?

At a local Emacs meetup with Damon Haley and Greg Pfeil, Greg mentioned that he’d done some experiments on using unicode characters in his mode-line. I decided to give it a try.

I took a good look at the above. I rarely use any of it — I normally don’t care about the coding system or the line ending style. I can’t remember the last time I had a buffer that was both read-only and modified. The VC information, when it appears, is generally too verbose and doesn’t show me the one thing I need to know (see below). And, though I do like to see the name of the major mode, I don’t really need to see most minor mode names; furthermore I like to have a bit of extra space so that I can use other modes that display information that I do want to see in the mode line.

What’s that VC thing? Well, ordinarily you may see something like Git-master in the mode line. But, usually I already know the version control system being used — or even if I don’t know, I probably don’t care if I am using VC. By default the branch name is in there too. This can be quite long and seems to get stale when I switch branches; and anyway because I do a lot of work via vc-dir, I don’t really need this in every buffer anyway.

However, what is missing is that the mode-line won’t tell me if a buffer should be registered with version control but is not. This is a pretty common source of errors.

So, first the code to deal with the VC state. We need a bit more code than you might think, because the information we need isn’t already computed, and my tries to compute it while updating the mode line caused weird behavior. Our rule for “should be registered” is “a VC back end claims this file, but the file isn’t actually registered”:

We’ll use that final function in the mode line. Note the odd character in there — my choice was U+26C3 (BLACK DRAUGHTS KING), since I thought it looked disk-drive-like — but you can easily replace it with something else. (Also note the weirdness of using string rather than a string constant. This is just for WordPress’ benefit as its editor kept mangling the actual character.)

To deal with minor modes, I used diminish. This made it easy to remove any display of some modes that I don’t care to know about, and replace the name of some others with a single character:

The thesis that underlies my project to translate the Emacs C code to Common Lisp is that Emacs Lisp is close enough to Common Lisp that the parts of the Emacs C code that implement Lisp can be dropped in favor of the generally superior CL implementation. This is generally true, but there are a few difficult bits.

Symbols

The primary problem is the translation of symbols when used as variable references. Consider this code:

First, Emacs Lisp uses dynamic binding by default (optional lexical binding is a new feature in Emacs 24). This applies to function arguments as well as other bindings. So, you might think you could translate this straightforwardly to:

But not so fast! Emacs also has buffer-local variables. These are variables where the value is associated with the current buffer; switching buffers makes a different binding visible to Lisp. These require no special syntax, and a variable can be made buffer-local at any time. So, we can break the above translation simply by evaluating:

(make-local-variable 'global)
(setq global 0)

Whoops! Now the function will return the wrong result — the translation will have no way to know that is should refer to the buffer-local value. (Well, ok, pretend that the setq magically worked somehow…)

My idea for implementing this is pretty convoluted. Actually I have two ideas, one “user” and one “kernel”:

User

I think it is possible to use define-symbol-macro on all symbols that come from Elisp, so that we can tell the CL compiler about the real implementation. However, a symbol can either be treated as a variable, or it can be treated as a symbol-macro — not both at the same time. So, we will need a second location of some kind to store the real value. Right now I’m thinking a symbol in another package, but maybe a cons or some other object would work better. In either case, we’d need a macro, a setf method for its expansion, and some extra-tricky redefinitions of let and defun to account for this change.

This would look something like:

(define-symbol-macro global (elisp:get-elisp-value 'global))
(defsetf elisp:get-elisp-value elisp:set-elisp-value))
;; Details left as an exercise for the reader.

This solution then has to be applied to buffer-, keyboard-, and frame-local variables.

Kernel

The kernel method is a lot simpler to explain: hack a Common Lisp implementation to directly know about buffer-locals. SMOP! But on the whole I think this approach is to be less preferred.

Other Problems

Emacs Lisp also freely extends other typical data types with custom attributes. I consider this part of the genius of Emacs; a more ordinary program would work within the strictures of some defined, external language, but Emacs is not so cautious or constrained. (Emacs is sort of a case study in breaking generally accepted rules of programming; which makes one wonder whether those rules are any good at all.)

So, for example, strings in Emacs have properties as a built-in component. The solution here is simple — we will just translate the Emacs string data type as a whole, something we probably have to do anyway, because Emacs also has its own idiosyncratic approach to different encodings.

In elisp, aref can be used to access elements of other vector-like objects, not just arrays; there are some other odd little cases like this. This is also easily handled; but it left me wondering why things like aref aren’t generic methods in CL. It often seems to me that a simpler, more orthogonal language lies inside of CL, struggling to get free. I try not to think these thoughts, though, as that way lies Scheme and the ridiculous fragmentation that has left Lisp unpopular.

This is a followup to my earlier post on converting the Emacs C code into Common Lisp. This one is a bit more technical, diving into some specifics of the conversion process.

Basics

One important fact is that we do not need to convert an arbitrary C program to Common Lisp. This might or might not be efficiently possible — but we do not care. We only need to convert Emacs. This is simpler for two reasons. First, we can just ignore any C construct that Emacs does not use. If the translator barfs after some new update, we can fix it then. Second, Emacs itself is already written in a relatively Lispy style, being a Lisp implementation itself. We further exploit this by allowing the translator to know some details about Emacs. As a trivial example, all the Smumble globals created by the DEFUN marco need not be translated into Common Lisp as structure constants — they are an artifact of the implementation, and will show up directly in the generated defuns instead.

What to ignore

A good portion of Emacs is simply redundant in the CL world. There are a few types (cons, vector, integers, functions) that are shareable — in fact, sharing these is part of the goal of this effort. There are also a number of functions which are effectively identical. There are also entire redundant modules, like the garbage collector, or the bytecode interpreter.

The question is how to have the translator differentiate between what is useful and what is not, without breaking builds of future versions of Emacs.

I don’t currently think there is a high road to solving this problem. For modules like the GC, I plan to have ad hoc translator rules for the particular source files. For functions and data types, I’m adding new GCC attributes that I can use to mark the ignorable definitions.

Types

There are two type-related issues that arise when translating the source.

First, how should Emacs-specific types be represented? Primarily these types are structures, like struct buffer or struct string (we cannot use the CL string type, because Emacs adds properties directly to the string, and Emacs has its own idiosyncratic character handling). My answer here is to just straightforwardly translate them to defstruct.

The other question is when translating a C function, what do we do with the types of local variables? For the most part I am pretending that they don’t exist. This works fine except for local arrays and structures, but these are easily handled by initializing variables properly. My rationale is that while this is slower, it lets me get something working more quickly, and we can always update the translator to emit CL type declarations later on.

This simple approach doesn’t actually cover all the needed cases. For example, there is code in Emacs that takes the address of a local variable and passes it somewhere. This is easy to deal with; much of the remaining work is just digging through the code looking for special cases to clean up.

I’m similarly omitting type declarations from the generated structures. One possible nice side effect of this approach is that it will make it easier to lift Emacs’ file-size restrictions, because there will no longer be any code assuming that the size is a fixnum.

Macros

Many low-level details of the Emacs implementation are hidden in macros. For example, Emacs stuffs some type information into the low-order bits of pointers. It uses macros to add or remove this information. For this build, I redefine these macros to do nothing. This makes the GCC Gimple representation much closer to the abstract meaning of the program, and thus simpler to translate.

There are also some macros that are useful to redefine so that we can more easily hook into them from the translator. For example, Emacs has a C macro INTEGERP that is used to check whether its argument is an integer. Normally this macro uses bit twiddling to get its answer, but I redefine it like so:

The output looks pretty weird, because the translator works after GCC’s CFG is built, and so the most straightforward translation is to use this mess with tagbody. I doubt this matters much, but in any case the translator is readily hackable — it is still less than 400 lines of Python, including comments.

One thing to note is the translation of “PT“. This is actually a macro that refers to the current buffer:

#define PT (current_buffer->pt + 0)

The translator properly turns this into a reference to “buffer-pt“.

Another detail is the handling of packages. My plan is to put the Emacs implementation into one package, and then any elisp into a second package called “elisp“. A DEFUN in the C code will actually generate two functions: the internal one, and the elisp-visible one; hence the “elisp:” in the translation.

Next Steps

There’s still a good amount of work to be done. The converter punts on various constructs; type translation is implemented but not actually wired up to anything; the translator should emit definitions for alien functions; and plenty more.

First, why rebase? Performance is the biggest reason. Emacs Lisp is a very basic lisp implementation. It has a primitive garbage collector and basic execution model, and due to how it is written, it is quite hard to improve this in place.

Seccond, why Common Lisp? Two reasons: first, Emacs Lisp resembles Common Lisp in many ways; elisp is like CL’s baby brother. Second, all of the hard problems in Lisp execution have already been solved excellently by existing, free-software CL implementations. In particular, the good CL implementations have much better garbage collectors, native compilation, threads, and FFI; we could expose the latter two to elisp in a straightforward way.

By “rebase” I mean something quite ambitious — rewrite the C source of Emacs into Common Lisp. I think this can largely be automated via a GCC plugin (e.g., written using David Malcolm’s Python plugin). Full automation would let the CL port be just another way to build Emacs, following the upstream development directly until all the Emacs maintainers can be convinced to drop C entirely (cough, cough).

Part of the rewrite would be dropping code that can be shared with CL. For example, we don’t need to translate the Emacs implementation of “cons“, we can just use the CL one.

Some CL glue would be needed to make this all work properly. These days it can’t be quite as small as elisp.lisp, but it still would not need to be very big. The trickiest problem is dealing with buffer-local variables; but I think that can be done by judicious use of define-symbol-macro in the elisp reader.

Emacs might be the only program in the world that would see a performance improvement from rewriting in CL :-). The reason for this is simple: Emacs’ performance is largely related to how well it executes lisp code, and how well the GC works.

I’ve been running an Emacs built from bzr for a while now. I did this so I could try a newer version of Semantic; the one in Emacs 23 is just too broken to use.

Semantic, in case you haven’t heard of it, is an ambitious project to turn Emacs into an IDE. Really it is quite a crazy project in some ways — it includes its own implementation of CLOS (which opens a strange Emacs maintenance debate: how can CLOS be ok but the CL package not be?) and a port of Bison to elisp (but again, strange: the port is really a pure port, it does not use sexps as the input — bizarre).

Semantic is now usable in Emacs — I’ve found a few buglets, but nothing serious. In fact, now I find it indispensible.

I have it configured in the most Emacsy way of all: I didn’t make any changes, I just enabled it with M-x semantic-mode. Then I visited a bunch of gdb source files. Semantic started indexing them in the background.

Now when I want to jump to a declaration or definition of a function, I use C-c , J. The key binding is nuts, of course, but I’ve been too lazy to rebind it yet. Anyway, this acts basically like M-., except you don’t have to ever run etags. Wonderful.

Semantic has some other nice features, too, but I haven’t really used them yet. If you’re using it I’d love to hear what you do with it.

I wrote a while ago about my blog-reading woes. Those woes are now over!

Lars Magne Ingebrigtsen, of Gnus and gmane fame, has now brought us gwene — an RSS to NNTP gateway. You enter the feeds you want to read, and soon they show up as newsgroups in gmane. Thanks also should go to Ted Zlanatov, for bringing this up on the gmane discussion list, and thus getting it all rolling.

I haven’t quite retired my rss2email cron job, but that is mostly out of laziness. Any day now.

Normally I am unhappy about the whole SaaS trend, but gmane gets a pass. I am not sure if it is because Lars seems trustworthy, or because NNTP is so obviously a fringe interest, or because gmane is at least theoretically replaceable in the event of the worst.