JavaScript is about run-time. Run-time is great and all, but especially when dealing with the browser, and how your Web page has to bootstrap the entire world on every load, Ajax developers have to think about issues that other people don’t. Problems that others can compile away, or know that “that happens once when I start up the puppy on the server” are here for us to stay.

This often seems to mean that we have to deal with writing our applications in ways that aren’t as clean as we may like. We run across problems where for the Nth time someone was bitten as somewhere code did a “for in” that wasn’t guarded with hasOwnProperty and then someone throws up there arms. Never Again. Dojo does a lot of things out of this experience. It is out of real-world pain that choices were made in the toolkit. One of these is how they are very careful not to pollute the global namespace. This is great in that you don’t run into collisions, especially in a world where code is being sucked in from who knows where (e.g. some Ad code is sucking in things). As the author of some JavaScript code, you don’t actually know what else may get into your global area when running, so you need to guard against it.

The problem is that this means that you can lose out. Prototype feels so right to me in many ways as it is less of a “JavaScript library” than a “way in which JavaScript should have evolved”. We have seen some of its goodness get into ES 3.1 (e.g. bind()) but at that rate of progress we will get four more methods in 20 years ;)

I have to take a second to tease here. Did you notice that there are two trim methods in Dojo? dojo.trim and dojo.string.trim. This is a good example of the crazy things that we have to think about. It appears that dojo.trim is lean code, so it gets into base, but dojo.string.trim is more code, but it runs faster. Wow :)

Once you get a lot of code going, you suddenly realise that the word “dojo” appears 50 times per screen of code. It makes me want to create a Bespin plugin that change the shade of that one word. It feels like Java. Unlike Java, you can’t import away some of the pain. You can try to var foo = some.really.big.package; but that gets annoying quickly.

So, to the point of the blog post. When I saw Alex Russell at a nice wine bar in San Francisco last week, I told him how I would die for the ability to have the best of both worlds of Dojo and Prototype:

I want to be able to write code the way that feels right, without the verboseness, but ALSO not run into the scoping issues. I would even take a performance hit for this. What if I could have:

runSomeCodeWithMagic(function(){// in here I am in the lovely land where "some content ".trim() works// and I can just forEach instead of dojo.forEach// but the outside world isn't affected});

Well, Alex listened to me blather on, probably thinking I was a total idiot, and then went on to quickly indulge me by giving me a little of my wish by giving me dojo.runWith:

Note/warning about the runtime cost: If your browser parses things fully in it’s JS engine up front, this function may hurt. A lot. It de-compiles a function using the toString() method, meaning that it does an uneval + eval + string concat + with() call. Each of these operations alone might be painful in a slow engine. Together they could be fatal. On the other hand, if you’re using these functions in, e.g., Chrome/V8, this could turn out to be relatively cheap, particularly as this is run-once kinda thing. The runtime cost involves namespace misses on locals, and that can be significant. I dunno. You’ll have to test to find out.

Note that you won’t easily be able to define globals from here by dropping a “var”. This might be a feature or a bug, depending on how you think about it.

What about getting the magic on the core objects? Again, only within the magically land of that scope do we want String to have the trim method…. and code outside of it shouldn’t see it. Can we swap onto the objects and their prototypes? Alex has some thoughts here too:

I think I can proxy intrinsics at some additional cost. I’d like to make it a protocol, though, so that you might be able to have a list or function that handles your extensions. E.g.:

This would give us a way to un-install them later, but I don’t have a solution for aync code as we do with with(){ … } which actually binds definitions at declaration time.

I might try to proxy intrinsic prototypes somehow, but I need to spend more time thinking about how to get that to work well. I’d like these things not to place big constraints on how you think about or use this system.

Wicked. This also ties into Pete Higgins and some of the very interesting work he is doing in similar but different veins with his dojotype and plugd which munge Dojo into interesting forms.

Since Dojo has pretty much everything that any other library has, you can start thinking about how you can bend it to look and feel like others, take on their APIs when it makes sense, and bend it to your whim.

With research like Alex’s we could see an interesting view when you can create worlds which don’t affect each other, let you have a view that makes sense for your code, but doesn’t affect others.

Even if you poo-poo some of it for the performance aspects, this is why it is incredibly exciting to see the latest JavaScript Vm work. With these guys running, they can optimize a lot of this a way, and things that used to be bottlenecks in the code will cease to be.

Thanks to Alex and Pete for indulging me, and taking the time to listen and produce really interesting solutions!

Updated: Using the Dojo Loader

James Burke has a very cool follow up on how to give a solution to this kind of problem using the Dojo Loader itself:

When Ben and I first started to use Bespin, at the time that we developed Bespin we would argue, or “discuss” subtle nuance of the coding experience. This inspired us more in the creation of the project, as we want to create an editor that Web developers can tweak and change in the language, platform, and tools that they use in their own lives.

Making the time to trick out and customize your developer tool is crucial. We spend day after day in our tools, and if you don’t take the time to learn them and trick them out, you aren’t going to be as efficient and productive as you can be.

Developer tools are often expert systems, and so we want to make Bespin as extensible as possible. Common attributes are wrapped in settings, and one of them is called strictlines. It is one of the places that Ben and I differ on how we like our editors to work (and disagreeing is fine because we get to choose for ourselves!)

The default behavior of the Bespin editor right now is to have this setting turned off, and that means:

You can click the mouse on ANY area that you can see in the editor and the cursor will go there and allow you to start typing.

If you position yourself at the beginning of the line and hit the left arrow, you will not budge.

If you position yourself at the end of the line and hit the right arrow, you will keep moving right

This is how IntelliJ IDEA works by default.

Contrast this to strictlines mode (where you have to explicitly set strictlines on). Now:

If you click off to the right you will be placed at the last column of the row

If you position yourself at the beginning of the line and hit the left arrow, you will go up a row and placed at the last character (so backwards).

If you position yourself at the end of the line and hit the right arrow, you will move to the start of the next line (so forewards).

I prefer strictlines, and most editors seem to come with that mode on. I don’t like the feeling of being able to click anywhere and not knowing if there is space in the file over there or not. I like being able to hold down left and go through the lines backwards, and vice versa for heading right.

Below is a video showing the two modes so you can see them in action:

What do you think? What would you like to see as default? To vote, comment on the blog or get on Twitter and “@bespin set strictlines on|off”.

I was excited to hear about the Canvas 3D effort that Mozilla, Google, and Khronos are engaged in (and others can too of course).

Khronos is the group being OpenGL, and thus a good set of folks to be involved in the Canvas 3D approach that is in the mould of “OpenGL ES like for the Web” in that it is a low level API that others can build on top of. Others have played with higher level “Games” APIs, or virtual worlds, and this is not the same. It is a primitive that will enable people to do interesting things that sit on top.

So it’s unfortunate to see that even the browser vendors have given up on moving the open web forward through standards. Whether it’s the WHATWG versus the W3C or the trials and tribulations of actually implementing HTML5, things are very broken and everyone is moving on regardless. I don’t blame any of them, but it doesn’t seem like it’s good for web developers.

I already talked about how many of the leaps on the Web haven’t started in the W3C (and rarely start inside standards orgs first) and rather come out in browser implementations that are then shared. Think XMLHttpRequest. Think Canvas itself from Apple! Do something well, see people use it and get excited about it, and then get multiple implementations and standards. Everyone wins.

John’s wording is interesting:

“But Mozilla’s proposal relies upon further proprietary extensions to the experimental CANVAS tag”

“And you’d lose the moral fulsomeness of the ‘Web Standards for The Open Web!’ pitch when focusing on your own proprietary alternatives to existing standards.”

Look at how browsers have done some things recently. Take some of the new CSS work that Apple started out. When the Mozilla community liked what they saw, and had developers demanding, they went and implemented it too. When you see WebKit and Gecko doing this kind of work it is particularly Open because the projects are open source and you can check them out (well, if you are allowed ;) How great is that, to iterate nicely in the open…. and then when ready we can drive into the standards bodies.

Back to the Canvas 3D work. Having Mozilla, Google, and Khronos work on this in the open seems pretty darn good to me. This won’t be hidden behind a proprietary binary that no-one can see. There will be some work in marrying the world of OpenGL ES and JavaScript as nicely as possible, and there will be plenty of room for the jQuery/Dojo/Prototype/YUI/…. of the world to do nice abstractions on top, but this is good stuff. This is more than just throwing out an API on top of a proprietary system, and I can’t wait to see what comes of it all. Want to get involved? You can in this world.

In former times (I love how Germans say that) I spent time working on event driven systems and busses. Those unfortunate enough to work with Enterprise JavaBeans were happy when Message Driven beans were announced as they could tell the pointy hair chaps that they were indeed doing EJB, and would continue working in their MOM architecture.

Now I fast forward to my days with JavaScript, and I remember getting excited to see the common JavaScript libraries support custom events. I now had a very nice way to work with my library to use event driven architecture on the client side, which is actually quite natural to do too. We see this pattern in desktop APIs such as Cocoa, and it helps enforce the practice of responsive applications for end users.

I may have taken this notion to the extreme a little in Bespin.

We have a general event system to put general work and you will also find that most of the components of Bespin also have their own events.js. A component will often load up the events by doing something like this:

// from within the editor itself, pass into the event system so the events are in scopenew bespin.editor.Events(this);

The event class will take in that component, put it in scope, and then an event can just use it. Note the use of editor here:

As new events come into play for the editor, this is where they will naturally be placed.

There have been some really nice fall outs of this approach. It has been really nice to have the notion of “this just happened, if anyone is interested, here is some info.” There is no need to find an object and call a method on it…. the event bus just takes care of it all for you.

There has been a nice side effect in the case of sharing commands across different parts of the application. Take the dashboard vs. the editor itself. Both have a command called newfile that looks a bit like this:

bespin.cmd.commands.add({name:'newfile',
takes:['filename','project'],
preview:'create a new buffer for file',
completeText:'optionally, name the new filename first, and then the name of the project second',
withKey:"CTRL SHIFT N",
execute:function(self, args){if(args.filename){
args.newfilename= args.filename;delete args.filename;}
bespin.publish("editor:newfile", args ||{});}});

The core piece is that the command basically just sends an event of type editor:newfile. On the other side of the coin, who is listening to this event? Well, in the case of the dashboard, it changes the URL to the editor itself, but if you are already in the editor itself, you don’t need to do that, and can just clear the editor buffer and file state.

Having commands at this high level means that they are easy to share.

We do need to do some more thinking about when it makes sense to promote something to be a full fledged event. In general I like the idea that people can write plugins and extension code that only requires them to grok some events and they are in and out in a jiffy.

However, some team members have brought me down to Earth and I realize that if we go too far here we will be reinventing method dispatch itself, and in a way that is only asynchronous :)

We don’t want to go that far. The no-brainer use cases are for typical call backs such as “a file has opened, care?” Something that an event asking for something to be done (e.g. newfile above) is a more subtle item and shouldn’t be over used.

Another side effect from using this model is that I remember exactly how much I like named parameters, which is why you will see the pattern is publish(eventName, argumentsHash). Dojo allows you to pass in multiple arguments, and in fact makes it harder to only send in one object as you have to dojo.publish(eventName, [ one ]). The [ ... ] looked ugly, so we wrapped the mechanism as bespin.publish(...). We also did this because we wanted to be a layer above the implementation. The Prototype version used Element.observe() and Element.bind() and we had to change all of that work in the port. Hopefully we won’t have to do this if a major change happens in the future.

Fun times! I will also write-up the effects of this, and how extensibility kicks in with Bespin on commands, your configuration, and plugins.

Do you remember when Web applications were CGI scripts? You had that cheeky bit of Perl that would be exec’d away. It was a haven for the lazy among us. What if you didn’t close the DB connection down perfectly? Didn’t matter. Use too much memory? Who cares. The process would die in seconds!

I then remember the real Web 2.0, when mod_perl and the like came about (yah yah there was NSAPI/ISAPI which I had to deal with too) and you realized that all of that bad programming now came at a price. The applications would be sitting on the server for a long time, so you could share a lot, but if you were sloppy you could cause bad things. Now you have to audit the code and gets thing right (as well as working around mod_perl bugs, but that is another story!)

The same has happened with the world of Ajax. Applications that used to be pages that would be refreshed are now replaced by sites like Gmail that sit in a browser tab all day long. If your application does something a little bad now, it can be noticed.

This is where the post that Ben just made on a newmemory tool for the Web comes in. As we write Bespin and other Web applications we feel the pain where we wish certain items weren’t as much of a black box as they are now in the browser. A first bite is memory, and being able to get some visibility into the heap, or see when the garbage collector is kicking in and how long it is taking.

What other tools like this would you like to see as you write long living Ajax applications in the browser? We are all ears!

Going from hacking away on Bespin before our launch, and now watching it 100% out in the open thanks to open source has been a fascinating transformation. Building a community is so much harder than hacking on code, and very different constraints appear. The past of “get a feature done” is changed to be “make it easy to get features done”. We still have a lot of work to do on extensibility, but it has been fun to see what people have already been able to do.

We have had experienced and junior folks pick up Bespin and help out, and I am trying very hard to strengthen a tough skill… delegation. Instead of picking off some bugs, I try to document them better and explain them so anyone in the community can pick them off. Sometimes it would be easier to hack up a quick fix compared to walking someone through a set of patches. However, that doesn’t meet the goal of getting people fishing away on our code and scratching their itches. Ben, myself, and our team doesn’t scale out to the size of developer tools groups at other huge companies, so we need to do what Mozilla does best….. build honest community. I am having a great time doing just that! The early contributors have been amazing already.

There is one recent experiment that I wanted to share. I was thinking about hacking on a key feature…. the ability to quickly search for and open files. This is the Apple-T feature in Textmate. I use it in the same way that I use Alt/Apple-Tab, or Apple-~ to move around. It is a core way in which I move around my projects. Instead of going right into code, we put together a mockup of how the feature could look:

Then, I spent a bit of time on the general design document itself to explain the feature, both from a use case / design angle, and on the “high level” coding side. I tried hard to give enough detail to explain the feature, while still allowing an implementor the ability to be creative and come up with their own ideas.

A few days later, Julian Viereck (a contributor who has already been incredibly helpful and generous with code) stepped up to the plate to say he would implement this, and in short order with the help of Kevin Dangoor building the server side infrastructure (index to search to find out the file names in this case), they had built a solid first version of the functionality.

It’s phenomenal, and I am so grateful to Julian for putting in time to make it work so well. Not only did he write the feature, but he also create a new Thunderhead component head to allow for moveable windows. Very cool indeed. Here are his thoughts on the implementation:

As described in the DesignDoc Quickopen is a window popping up in the editor to let you choose a file you want to jump to and perform some work on with the editor. This allows you to open a other file without going to the dashboard and back again to the editor => you can stay longer in the editor and have not to reload the whole page just for changing the current file ;) This is a quite important feature when it comes up to work on a “real” project in Bespin as you really stay on the work itself :)

To open Quickopen press ALT + O in the editor.

Quickopen: How it works?

When the user fires up Quickopen the first time it shows a list of the current opened files in the project (quite the same as the Open Session thing in dashboard, but just for the current porject). When tipping a searchkey a request is send to the server and a result list back to the user and displayed.

Kevin did the backend stuff. He says “the server is using a *really* stupid file cache to make searches zippy”. Well, it’s really zippy ;) But here is a list of things I would think that could be improved:

a) the seachindex is not updated at the moment. So once deleted a file, it is still in the seachindex. Try to open this file with Quickopen will cause a strange behavoir. The best solution would be, to have the searchindex in sync with the filesystem, but well, thats maybe a bigger deal. For the moment it would be great to have a command for paver like “paver updateSearch -u <username>”.

b) the search results list also files that cannot be opened by Bespin (e.g. image files…). I would not like to see the backend making a choice which files the user should be able to seen in this result list and which not by certain rules. But I think let the user make that choice is a good point. I’m for example not interessted in the image files BUT also not interessted in all these .py files and these stuff. There should be a new user setting to make this adjustments. But at the moment I have no cloue how this setting has to look like? A regex, or something like “excludeFiles= *\.js|.html|.css” for excluding all files except js-, html- and css-files?

c) the search should remember how often the user picked a certain file from the list and put this file more above in the resultlist.

Other ideas? These is work to be done on the backend. I never wrote one line of python and have not looked really at the backend stuff, so maybe someone else should take over this part :)

But when making so often a switch between the files it also comes up to add other things: the editor should remember the mouse position on the files as the user jumps between them. This makes it easier to continue working on the files. For this I thought about adding a new kind of “settings” that stores such data like mouse positions, window positions and such stuff, but I cannot come up with a name for it.

What do you think?

BUT: There is even more new stuff: th.window!

When implementing the Quickopen window I was thinking: “why is there no such class in th”? Well, there it is: th.window!

th.window brings up a window in the browser, with the same border and window bar as the one used by Quickopen (well, Quickopen uses th.window already, so the Quickopen stuff in quickopen.js is a good starting point to see how to use th.window ;)). Having a th.window class was something Malte asked for and I hope other stuff will provide from this new class as well :)

When creating a new th.window object, a new <div> with a <canvas> is insert. The canvas is used for the th.window::scene, to which is the th.WindowBar added automaticaly as well as the user panel; the place to put in the things that should stand within the window. Some basic functions are added to the window: you can drag them around on the screne, there is a close button within the WindowBar, the window closes automatically the user clicks outside the window (is this prefered in all way, I’m not sure) or press the ESCAPE key, there is a toggle, move and center function. Just the basics, but a good point to build on!

Thanks so much to Julian and the other bright sparks that have made Bespin a fun project to work on. There is so much to be done, but hacking on a tool that you actually use is compelling, so I can’t wait to see more!

From the get go, the Bespin project means a few different things. One of the components is the Bespin Editor component itself. We have already seen people taking that piece and plugging it into their system. For example, the XWiki integration.

The problem is that we (Bespin team) haven’t done a good job at making this reuse as easy as it should be. That has now changed with the edition of the bespin.editor.Component class that tries to wrap up the various parts and pieces that the editor can tie into (settings, toolbars, command lines, server and file access) so you don’t have to think about them.

A common use case will be embedding the editor itself, and having it load up some content, maybe from a container div itself.

First we read in the embed wrapper code, which relies on Dojo (so Dojo has to be loaded first).

Then we create a component passing in the HTML tag to inject into, and options which in this case tell it to use JavaScript syntax highlighting, and then load up the editor using the value in the div that we are injecting into.

At this point the editor is ready to go. You can focus on the puppy and start typing, but chances are you want to access the editor text at some point (for example, read from it and post it up to a form).

To mimic this, we have a textarea that we can copy contents into (editor.getContent()), and then send it back to the editor (editor.setContent(contents)):

The example also shows how you can change settings for the editor via editor.set(key, value).

There are more features we should probably put into the editor, such as automatically syncing to a hidden textarea with an id that you specify (so then a form can just be submitted and the backend gets the right stuf).

We have plans for integrating rich documentation and resources with Bespin. One quick hack that I have been meaning to do for awhile just got done finally today.

I created a simple command that lets me type doctype DivElement to get an inline popup with details on a <div>. It looks like this:

Right now you have to know what sections are available, which is laid out for you. There are common patterns in there too, such as: NameOfTag + “Element” (e.g. the DivElement) and the same for *Entity and *Attribute.

In the future I would love to be able to tie search into the system so it is more forgiving, and also other features such as getting this data from a context menu, and more. What would you like to see?

The command is documented here, and to use it in your Bespin, you will need the latest version (as of March 12th) and then you can cmdedit doctype and paste it in:

The solution in place at that time worked well in Safari, with the on[cut|copy|paste] and unfortunately not great with Firefox due to one crucial big of data being omitted.

Then Tom Robinson and a couple of others made the great point that I could get ALMOST everything I wanted with just the hidden textaraea trick. Instead of tying the trick to the on[cut|copy|paste] events, I can just manually grab the Cmd/Ctrl-C X V commands. The only downside to this is that if the user goes to the Edit menu and chooses something, it won’t work. Annoying, but that’s the world of the hacky Web sometimes.

I added in this new tactic, and copy and paste works OK for Firefox in the latest version of Bespin in code (not deployed to bespin.mozilla.com at the time of this writing):

Because of the issues, I took out the UI buttons for cut/copy/paste, and am in fact wondering if the editor needs that row at all. I wonder if we can consolidate the header to one line, giving us more vertical space. A code editor for developers is not like Google Docs for average Joe users, so having the visual cues probably doesn’t matter in the same way for items like copy.

There are a few subtle annoyances such as running an action like killLine (Ctrl-K) which cuts the selection but has to do so in a non-work with the clipboard way.

End result: getting there, but still need to work on making this generally viable for any application on the Open Web Platform. What do you think?

UPDATE: Used the copynpaster variable throughout, and added a setTimeout around the paste operation as I found some people needed to hit the paste key twice as the focus()/select() was taking a little too long.

Roberto Saccon has given the Bespin project a nice gift, by porting the Bespin codebase to Dojo. This was a lot of work and is very much appreciated. The obvious question is going to be “Why did you decide to change from Prototype to Dojo?”

Firstly, Ben and I really like Prototype. I am a Ruby guy, so it feels good to me. I talked about it a bit here, and I will use Prototype in many projects in the future.

The state of Ajax frameworks is quite interesting. On the one hand, many of the top dogs have actively learned from each other which has lead to many of them offering similar functionality. For example, Dojo and Prototype can do a good job with DOM selectors, which jQuery is known for.

Although you can do a good job in any of the top libraries, they still differ in scope, and are optimized for certain use cases, project sizes, and functionality wants.

jQuery offers a phenomenal API for doing stuff with the DOM. It feels right. It is also trivial to extend this world, which has lead to the huge number of plugins to do just that. Because of the clean, simple API, we have seen a huge surge in jQuery usage and interest. I would say that it is optimal for designers and beginner JavaScript folk. This is not to say that it isn’t also great for experts. There is large support from folks like Simon Willison. John Resig did something amazing. If you think functional, this will be your cup of tea even above and beyond.

Prototype goes a little further in that it offers more sugar on top of the language itself. Some find the strapping on of methods on core Objects as obscene. Personally, I love it. It just feels so much nicer for me to say array.last() or array.include(thing), compared to dojo.indexOf(array, item) > -1 for example. In fact, looking at a bunch of code has me feeling a touch nuts with all of the “dojo.” items in it. My brain is starting to ignore them.

That being said, Dojo’s purity gives you a lot. It may be more verbose to dojo.byId than $, but we have no namespace pollution.

The community quickly wanted us to package up Bespin in a way that we could share things. We wanted the same, and have natural components (e.g. the Editor, Thunderhead itself), and some didn’t like the leakage of the Editor component requiring Prototype which in turn meant pollution into the global namespaces. With Dojo we can hide away nicely, and can even run a build to become foo instead of dojo if we wanted.

Since the community really wanted it, and someone stepped up and actually did it, we accepted the change. On our codebase it had some interesting side effects. For example, before hand we had Prototype, Script.aculo.us, and a slew of third party libraries to give us a bit of this, that, and the other. With Dojo, they had everything we were needing and more, so we could hg rm external/ and be done. We are also doing Comet work and the like, and it fits in nicely.

The first change here was to get Bespin working in Dojo, but now we have more work to make sure that:

The codebase feels Dojo-y

Use more of the Dojo features (e.g. dojo.keys for key handling, CSS store, Comet, …)

Clean up and package our stuff nicely (e.g. bespin vs. editor vs. th)

If you are a Dojo fan, or fancy getting into it anyway, please join in! There are a few Prototype things left around, so some of the spirit is there.

A bit of an aside…

A big pain with Ajax and components, is the whole “I really like that jQuery UI component, but I am using Prototype already…. grrr”. Simon Kaegi of IBM has been putting together some thoughts and code around a JavaScript module system that would enable you to say “I want service X which happens to depend on jQuery, and service Y which depends on Prototype.” I am very interested to see where this goes. We sorely need it! The annoying problem on the client is that having multiple libraries is not cheap. On the server though? Not as big a deal potentially.