Regrettable Hacks

Thursday, January 31, 2013

As a programmer coming to JavaScript from the Java world (arguably a bad influence) and the ActionScript world (undoubtedly an even worse one), learning how JavaScript works has been a gradual and error-prone process. I've recently been working on a Coding conventions page on the Orion wiki, and a big chunk of that page is devoted to avoiding the kind of common JavaScript traps that I'm always falling into.

Today I wrote a section about creating classes, taking care to point out a mistake that I'd made dozens of times in the past. This post is adapted from there, so go read the original writeup if you're not interested in the extra verbiage.

First, here's how we tend to create classes in Orion's JavaScript code. I won't claim this is the One True Way, just an easy and straightforward convention that we've settled on.

Everything here is reasonable except the XXX'd line, SeaDuck.prototype = new Duck(). This part is wrong. (Unfortunately, many occurrences of this pattern remain in the Orion source code, lots of them written by me, which still need to be cleaned up.) So what's wrong with it?

It's inefficient, since SeaDuck.prototype has fields created by Duck that are never used (like name, for example). Any work done in the Duck constructor is useless to us here.

It's fragile, since the proper operation of this code relies on Duck not validating its input parameters. If anyone ever changes Duck to assert that it receives a valid name, our SeaDuck code will blow up. (And sure: we could pass in a fake name to satisfy it, but that clutters up our code with even more useless data).

For our SeaDuck.prototype, what we really want is not to call the Duck constructor, but just to create a new object whose prototype is set to Duck.prototype. In other words, we only want steps #1, #2, and #4 of the new algorithm, not #3.

Object.create

For a long time JavaScript provided no way to decouple object creation and prototype-setting from initialization. ECMAScript 5 finally fixed this problem by introducing Object.create. Among other things, Object.create allows you to build a new object and tell the JS engine exactly what its prototype should be. Just pass the desired prototype object as the first argument. Easy!

SeaDuck.protoype = Object.create(Duck.prototype);

And that's exactly what we wanted. In fact, we can use Object.create to replace even legitimate uses of the new operator. Instead of this:

var robert = new Duck("Robert");

…We can write this, effectively re-implementing the new algorithm by hand:

beget avoids the problem I pointed out in the previous section by creating a new BogusConstructor every time it's called, which does no initialization work and only exists to achieve point #3 of the new algorithm.

You could also turn beget into a partial shim for Object.create. I say partial because Object.create also deals with property descriptors, which are impossible to shim.

The prototype, "prototype", [[Prototype]], and __proto__ mess

Looking back, a big source of my confusion as a learner was in understanding how the prototype property of functions relates to an "object's prototype" and how that in turn affects property lookup. You can easily see that regular objects don't have a prototype property: try evaluating ({ }).prototype in your debugger. So what's this "prototype" thing everyone keeps talking about on objects? Well, here's my attempt to clarify things, in point form:

When we say "an object's prototype", we're referring to an internal property of the object, which the ECMAScript spec calls [[Prototype]]. Being an internal property, [[Property]] is not observable from regular ECMAScript code.

An object's [[Prototype]] is consulted to resolve property names when the dot . and array index [] operators are used on the object. If the desired property name is not found in [[Prototype]], then the [[Prototype]]'s [[Prototype]] is consulted, and so on. When people talk about the prototype chain, this is what they mean.

The prototype property can be set on a function. When some function F is invoked as a constructor through the new operator, the value of F.prototype becomes the [[Prototype]] property of the newly-constructed object.

To create a new object whose [[Prototype]] is some existing object P, use Object.create(P).

I lied a bit in point (i). Most JavaScript engines provide a non-standard alias for the internal [[Prototype]] property. It's called __proto__. (ES5 has defined a standardized Object.getPrototypeOf, so use that instead of __proto__!)

While you can't rely on __proto__ in production code, it's great for debugging, and for fixing your mental model of how the JS engine works. Here's what a SeaDuck's __proto__ looks like in the JS console:

Note how you can expand the __proto__ chain all the way up to Object.prototype.

Wednesday, October 24, 2012

Things are pretty wild in the Orion world right now. We're coming up fast on our 1.0 release (**balloons fall from ceiling**). So given how busy we all are, this seems like a great time to stop fixing important last-minute bugs and instead write a leisurely blog post.

Here I'll explain a new feature that landed in Orion 1.0M1: plugin settings. Plugin settings provide a way for your Orion plugin to contribute settings to Orion's Settings page. You tell Orion about your setting, and it generates a user interface for manipulating your setting's value. That user interface lives on Orion's Settings page. To receive updates about the value of a setting, your plugin registers a service that will be invoked by the Orion platform to deliver notifications.

To show how this looks from a user's point of view, check out this predefined setting that ships with Orion 1.0:

JSLint validation (click for larger).

This setting lets you customize the options passed to the JSLint validator that checks your JavaScript files for problems. You can turn off annoying validation behavior globally by putting your preferred flags in here — for example, filling in eqeqeq:false will tell JSLint to tolerate JavaScript's type-coercing == and != operators without complaining*.

Naturally, this setting is contributed by Orion's JSLint Plugin itself, which ships bundled with Orion. This is where things get a little more interesting: if you went ahead and hacked your Orion environment to remove the JSLint Plugin, this setting would disappear from the UI as well. Makes sense, right? This also emphasizes a fact that I think will become increasingly important as Orion matures: plugins are not one-trick ponies (or one-service ponies, to be exact). Rather, they are ponies that can include a whole package of related functionality supporting a given task or feature set.

A plugin.

OK, back to settings. Here's the second setting I mentioned: the New Tab/Same Tab option.

New Tab/Same Tab (click for larger).

This thing controls whether the links in the Navigator (and a few other places) open in a new browser tab. You may recall a similar option from Orion 0.5, but back then it was implemented in a rather ad hoc way, and now it's a plugin setting, which is much cooler. Note how this setting gets a drop-down menu: this is how our generated UI presents a setting whose value is restricted to an enumeration of discrete options.

Show me the code, already

Note: I've omitted a few lines of boilerplate plugin code in these examples. If you're unfamiliar with writing Orion plugins, check out the Simple Plugin Example from the Orion Developer Guide.

So how do you, the plugin author — (if you're not a plugin author, pretend you are) — how do you write one of these plugin settings? Like much of Orion, the heavy lifting happens declaratively through service properties. To contribute a setting, we register a service with some properties telling the Orion Settings page about it. Here's what the code for that service registration looks like:

Staying up to date

Shoving your setting into the UI is all well and good, but it's not very useful if nobody cares when its value gets changed. For a setting to be useful, it has to affect the world somehow. That's where the orion.cm.managedserviceextension point comes in. Your plugin implements one of these Managed Services in order to receive updates from Orion about the setting.

The API for receiving an update is very simple. It looks like this:

updated: function(properties)

Interface of orion.cm.managedservice.

That's it: just one method with the signature updated(properties). The framework calls this method to notify your service of a setting change, passing the values of all your setting's properties in the properties object. Like all service calls in Orion, this happens asynchronously. In addition to that method, your service registration must have a property named 'pid', giving the PID (that is, the identifier) of the setting that it's interested in. Here's what a minimal ManagedService implementation looks like:

However, the real power of this stuff comes in when your Managed Service plays several roles. In Orion's service framework, a single service can be registered under several service names — or in other words, a single service may implement several extension points. Think of orion.cm.managedservice, then, as an optional interface that can be implemented by an existing service to make it configurable by the framework.

For example, in the case of the "JSLint validation options" setting I mentioned earlier, we initially started out with just a validation service — ie. a service with the name "orion.edit.validator". When I wanted to make it configurable via the setting, I added the service name "orion.cm.managedservice", along with the pid property and the updated() method to fulfill the ManagedService contract. The final result ends up looking like this:

As an added bonus, the framework guarantees that, when your service is "managed" (ie. when it implements orion.cm.managedservice), its updated() method will be called to deliver the setting's value before any of the service's other methods are called. So in the code above, our validator knows that myOptions will be set before checkSyntax is invoked to validate a file. This is particularly nice because most Orion plugins are activated lazily (that is, only when one of their services needs to be called), and plugin authors would otherwise have deal with the possibility that one of their service methods might be invoked before the service had received its configuration. As is, however, the framework saves you from having to worry about that.

Under the hood

(This section talks about internal details, which aren't necessary to use the Settings API. Skip it if you're not interested.)

If you've worked with OSGi frameworks before, you may be familiar with the OSGi concepts of managed services and metatypes. Orion has both Managed Services and Metatypes, which are similar to OSGi's (albeit greatly simplified), and they form the low-level building blocks upon which the Orion Settings API is implemented.

Orion manages Managed Services using a service called ConfigAdmin (another borrowed OSGi-ism). The ConfigAdmin is provided by the Orion platform (platform being a convenient word to describe the low-level plugin and service plumbing). The ConfigAdmin starts up very early in the page's lifecycle, and from then on, monitors the registration of Managed Services and calls their updated() methods when necessary.

A Metatype describes the shape of an object: that is, what properties it can have, their data types, and default values. If this sounds similar to what goes into a setting, it is: in fact the set of properties within a single setting actually comprise a Metatype. Every setting defined through the orion.core.setting API is internally translated into a Metatype definition that describes the setting's properties. A setting, then, is basically just a thin layer on top of a Metatype, combining the Metatype with a PID, and some additional information telling the Settings page how to categorize the setting (the category field).

Far from being just an identifier for settings, a PID provides the crucial linkage between a Managed Service and Metatypes. A Metatype can be "designated" (associated) to a PID. This association informs the framework of the data shape (properties) that a Managed Service having that same PID expects to be configured with.

In summary, Managed Services are the services that expect to be configured with properties, Metatypes describe what properties they receive, the PID is what connects them, and Settings provide a convenient intersection between all three concepts.

Thursday, April 5, 2012

Amazon S3 is a popular cloud file hosting service. For months people have been complaining that there was no way to store your Orion work — that is to say, your files and folders — in S3 so you'd have a fast, reliable, place to store your data in the cloud. (Well OK, these complaints were all from one particular Orion committer, but whatever.)

In any case, I finally opted to write a plugin for Orion that lets you use S3 as a file store. This post talks about my experiences. If you'd rather skip right to the code, go here:

The S3 Worldview

In the S3 storage model, an object is the basic unit of storage. Objects are resources that you can refer to and manipulate using a REST API. Objects live in buckets, which are flat containers that can hold many objects (sometimes millions). As an S3 user, you'll generally create one or more buckets to hold your data. S3 also supports powerful security policies for granting bucket access to other users, but that doesn't concern us here.

Some S3 concepts.

The Orion File Model

Orion's file support is based on the concept of filesystems. The model here is decidedly more traditional than S3's:

Orion file concepts.

As the diagram tries to show, a filesystem is a place accessible to Orion where files can live. At the top level of a filesystem is the workspace. Within a workspace are folders and files. Naturally, folders can contain files and other folders, nested to an arbitrary depth.

From S3 to Orion

The important thing to note about S3 is that the bucket is the only unit of containment. Every object is contained by one and only one bucket. There is no containment relationship between objects. Initially this might seem like it limits us to a filesystem that is simply a huge, flat list of files. But that's not the case: a hierarchical view can be imposed on a properly-structured bucket. I won't go into detail about how this is done (read the AWS documentation if you're interested). The upshot is that we can indeed implement the Orion filesystem concepts on top of S3 — although some operations will necessarily be more complex (and likely slower) than if we were connecting to a true hierarchical back end.

Extending Orion's File Support

To connect S3 as an Orion filesystem, we need to write a plugin that contributes an implementation of a service named orion.core.file. Various parts of the Orion client UI delegate to implementors of this service. For example, the Orion navigator relies on the fetchChildren() operation, so implementing this function immediately allows the Orion navigator to descend into your filesystem and display child items as nodes in the navigation tree. Implementing read() and write() allows the Orion editor to open and save files that live in your file system. There's also file and folder creation, move, copy, rename, delete, etc. The more of these features you implement, the smoother the integration into Orion will be.

Domain Topology

For maximum portability, we want our implementation of the FileClient API to live purely on the client side: no server-side proxies or other hacks allowed. This means all our logic for talking to S3 will run as client-side JavaScript in a browser. Our code will make Ajax requests using the S3 REST API to manipulate S3 objects.

This imposes one important limitation: the web browser's same origin policy. Essentially the plugin (web page) hosting our S3 file service must have the same origin (protocol + host + port) as the target of all its Ajax requests. So does S3 let us configure a bucket to satisfy this limitation? The answer appears to be no. You can't have, say, a particular path prefix in your bucket configured to host a static website, while other prefixes respond to S3 API calls. A bucket can be configured either to host a static website, or as an S3 endpoint, but not both.

OK. Can we instead use two separate buckets to overcome this limitation? Yes! Conveniently, S3 allows you to access objects using 3 different URL styles:

http://s3.amazonaws.com/bucket/key

http://bucket.s3.amazonaws.com/key

http://bucket/key(where bucket is a DNS CNAME record pointing to bucket.s3.amazonaws.com)

Notice that while URLs #2 and #3 have the bucket name as part of the host name (hence tying the origin to a single bucket), URL #1 does not. Using the style of URL #1, our plugin can be hosted from one bucket (configured as a publicly-readable static website), and still be free to blast Ajax REST requests at a second bucket (configured as a restricted-access S3 endpoint). Both buckets are accessed from the same domain, s3.amazonaws.com, so the browser won't complain about same-origin violations. Below is a diagram showing how this arrangement works out:

High-level view of the architecture.

In the diagram above, Bucket 1 is publicly-readable. Thus the Orion plugin registry can access and load the plugin without worrying about how to authenticate itself. By contrast, we've restricted access to Bucket 2 to just our S3 user account, thus the Ajax requests that are sent its way must be authenticated. The next section explains further. (Also note: I've drawn the Orion client UI in red since access it usually requires some kind of authentication to access. That's not a requirement, however: you can deploy Orion in all sorts of less-secure ways.)

Security

Keeping all the code on the client is convenient, but presents some security challenges. To modify the contents of Bucket 2, all our REST requests must be properly authenticated. Authentication requires three pieces of information: the request, our Access Key (public key), and our Secret Access Key (private key).

So the plugin needs to know how to authenticate requests, and to authenticate a request it needs your private key. This exposes a limitation in the existing Orion plugin/service framework, which is that there's no secure way to do this. (More generally, there's no way for a plugin to request access to a user setting from the Orion core… but that's a larger issue.)

My somewhat-inadvisable temporary solution is this: when the plugin requires your keys, it simply opens a JavaScript prompt asking you to paste them in. It also asks for a "passphrase", which it uses to encrypt the keys. The encrypted keys are kept in localStorage, which saves you the trouble of typing them in every time. The passphrase is not persisted, and gets squirreled away inside a closure variable, which vanishes when you close the browser tab.

The upshot is this: if you close and reopen your browser, the plugin will find the cached keys in local storage, and prompt only for the passphrase so it can decrypt the keys and carry on.

Now, obviously asking people to paste secret information into a web page is quite dubious. On top of that, storing sensitive information (albeit encrypted) in a localStorage database for a domain that you don't fully control (here, s3.amazonaws.com) is also a very bad idea. Finally, the usability suffers because the keys are tied to your browser, not your Orion user settings. (Hence, if you use a different web browser, you'll have to paste in your keys all over again). So please regard this as a starting point for future work around security and authentication in Orion, not a realistic long-term solution!

The End Result

If the disclaimer in the previous section hasn't scared you off, you can set up the plugin (see its GitHub readme for installation instructions). It lets you do basic file operations against your S3 bucket, and performs pretty well.

Future Directions

Storing any sensitive data in localStorage is not a good approach. The naive solution might be to move your private credentials into the Orion preference store, where access to them could at least be managed somehow (perhaps on a per-plugin basis). But while security is one of Orion's concerns, becoming a secure-storage provider is not. In fact, it's probably better to compartmentalize sensitive information so that the Orion core never actually sees your private credentials.

So how do we accomplish that? One approach would be for the Orion core to simply establish a channel between an signing requester and a signing provider. The requester in this case would be our S3 plugin, and the provider could be any Orion-connected entity who possesses your S3 credentials and knows how to use them to sign a given request. (We can imagine a provider linked to a secure, trusted third-party service running a cloud deployment of something like KeePass, or even a private provider that is only accessible from your local network.)

Once the provider–requester channel is established, these pieces talk only to each other. Most importantly, your private key never has to leave the signing provider. This all sounds a bit hand-wavy right now, but remember Orion lives on the web where all kind of crazy ideas are possible.

What you need

Language Grammars

Orion implements a subset of the language grammar format used by TextMate,
a popular text editor for Mac OS X. A grammar gives a declarative description of the structure of files of a certain type (for example, a
.java or .php file). This "structure" can be limited to simple keyword recognition using regular expressions, but the format
provides enough power to recognize more complex language constructs like classes or function definitions. Sometimes a carefully-written grammar can even detect
common syntax errors, such as a missing comma in a list of items.

The grammar provides rules that tell the underlying TextView what CSS class a portion of text should have. The mapping of CSS classes to rules
(font-color, font-weight, etc.) is ultimately determined by the stylesheet applied to the TextView.

Finding a Grammar

We're going to use a JSON grammar, which its authors have made available under a BSD license. Not every grammar you come across in the wild will allow redistribution, so take note.

From PList to JavaScript

If you open the JSON.tmLanguage file you'll see that it's written in a strange XML dialect. It's actually a Property List
file, which is a common data storage format used in OS X. TextMate's grammars are written as property lists. This poses a problem for all faithful JavaScript programmers, who avoid
XML like the plague.

But don't whip out your validator just yet. It turns out that property lists convert fairly well to JavaScript object literals, and we've written a tool that will do it automatically:

The tool only accepts XML property lists. If you come across a property list in an ASCII or binary representation, use OS X's plutil command to convert it to XML first.

Running the converter

Go to the grammar converter in your web browser. Find the folder on your computer where you saved the JSON.tmLanguage and drag it onto the textarea shown in the converter. Immediately, the result should appear below it. Click the Clean up button and the converted JavaScript object will be formatted a little more nicely.

Copy and paste the JavaScript object from the text field into a new file somewhere. This is our grammar.

Packaging the Grammar as a Plugin

Now we have to write some code that registers our grammar with the Orion editor. Orion allows clients to contribute language grammars using the orion.edit.highlighter service.

First grab a copy of plugin.js, which provides the Orion plugin API. Put it in a new folder called JsonProject in your Orion workspace. In the same folder, create a new file called jsonPlugin.html with the following content:

The first registerServiceProvider call is the crucial part: it contributes our grammar and tells the Orion editor to use it for .json files.

The second registerServiceProvider call associates the Orion editor with .json files so you can click them from the Navigator view to open them in the editor.
This is not strictly necessary to get syntax highlighting, but it's usually what you want when you're adding support for a new file type.

Installing and Testing

To install your plugin, you'll need to host it at a web URL. You can do this by launching the JsonProject folder as a standalone site from within Orion — see
this page for instructions. Or if you have a personal
web server, you can copy the project contents there instead.

Once you have a URL to jsonPlugin.html, visit the Plugins page in Orion. Type the URL into the box and click Install.

Reload the Navigator page so that the plugin we just installed takes effect. Create a new .json file. Click on it and the editor should open. At this point you should
be able to see it recognizing JSON styling as you type:

Conclusion

We've seen how to take an existing .tmLanguage file and use it to provide syntax highlighting in Orion.

See Also

jsonPlugin.html — A pre-built version of the final result, just in case you want to cheat