Objects are the main characters

This little post is to add a summary of what I’ll be showing in the upcoming ESUG conference. I’ll present a paper about my work in the workshop and present the recovery tools of Oz in the innovation awards :).

For the paper

I’ll present the ideas and implementation of Oz object spaces. Especially the metacircular problems it solves, and a bit on the vision we (I and my supervisors have on this line of work). If you want to read the paper, you can go to:

Reﬂective architectures are a powerful solution for code browsing, debugging or in-language process handling. However, these reﬂective architectures show some limitations in edge cases of self-modiﬁcation and self-monitoring. Modifying the modiﬁer process or monitoring the monitor process in a reﬂective system alters the system itself, leading to the impossibility to perform some of those tasks properly. In this paper we analyze the problems of reﬂective architectures in the context of image based object-oriented languages and solve them by providing a ﬁrst-class representation of an image: a virtualized image.
We present Oz, our virtual image solution. In Oz, a virtual image is represented by an object space. Through an object space, an image can manipulate the internal structure and control the execution of other images. An Oz object space allows one to introspect and modify execution information such as processes, contexts, existing classes and objects. We show how Oz solves the edge cases of reﬂective architectures by adding a third participant, and thus, removing the selfmodiﬁcation and self-observation constraints.

For the innovation awards

I’ve prepared a demo and with that I recorded a little video showing it. You can see it in:

While hacking the VM the other day, I went into a very special situation: I got an image that crashed on startup. And I had stuff without commit!! So I started the crusade to recover it.

I know that some time ago, people like Sean De Nigris had the same problem and ended up hacking the VM to solve their problem. You can have an idea of what exactly happened by reading the original mailing list post.

But doing that required debugging the vm at the bytecode level (that is, debugging each bytecode internally) until I reach my point and solve my problem.

The concrete situation

The last thing I executed in my broken image was the following two statements in one doit:

So when my image was started, it executed all the startup code, returned to the context of the code above, and tried to send the doSomethingThatCrashesTheVM message to someObject, causing a crash.

I knew that Oz could do it, so I went into it.

The recovery environment

I needed to put my image in a place where I could fix it. And that place is an object space, which I am implementing in Oz. The idea is to put the broken image inside another image with an object space, so we can manipulate it freely, fix it, and restart it. With the Oz library and VM, creating the object space is as easy as:

objectSpace := Pharo20 loadFrom: 'broken.image'.

The diagnostic

Once we have the objectSpace object, we need to understand how to solve our problem. So I wondered around by getting the scheduler of the loaded image, and looking at the active process.

The medicine

There are many ways to manipulate the contexts and and processes to solve that. But there is also one that is really easy and simple.

We only need to change the someObject reference to nil. That way, we avoid the crash in the startup and obtain a debugger instead with a:

“Undefined Object does not understand doSomethingThatCrashesTheVM”

Nice! So, in order to do that, I needed to understand a bit how the code was compiled. I did it by sending a couple of messages to the compiled method, to understand its structure. Finally, I got the following conclusion: someObject was the first literal in the compiled method. And I fixed it:

I restarted the image in the object space (with the so far ad-hoc method):

[[ objectSpace giveChanceToRun. true ] whileTrue: [ ]] forkAt: 80.

So you make a loop and in each iteration you give the object space a window of time to run. And after that the broken image will appear before you (if you fixed it reasonably well) :).Then, I saved it with another name, restored manually the changes file (because my object spaces solution does not handle yet :)) and voilá. I had my image back again, with all my objects there. I and could commit :).

As you may already know, ZeroConf scripts are bash scripts that ease the installation of a Pharo environment. A funny thing about these ZeroConf scripts is that they are seen as bash scripts by a bash terminal, and as simple and minimal html pages by a web browser. These scripts are extensively used to simplify the configuration of Pharo CI jobs. They allow you to easily download many versions of the Pharo image and VM.

As I’m working for my Phd, and have a custom version of my virtual machine and image, and also want to make use of the advantages our CI server provides, I wanted to build my own ZeroConf scripts specialized for my needs. I also heard recently on the pharo mailing list that there was some work on customizing ZeroConf scripts[1] for Moose[2]. So I wanted to do it as well for my project :).

Downloading the ZeroConf package

The ZeroConf scripts are generated automatically by the ZeroConf Pharo package. You can find this package in [2]. To download the current version, you just have to execute the following piece of code in a workspace:

Getting what’s inside ZeroConf

The ZeroConf package is pretty small and simple. There is an abstract class AbstractZeroConfBashScript implementing most of the script generation and bash writing utils. Its subclasses will implement the concrete script generation. Current implementation includes three main classes below the AbstractZeroConfBashScript hierarchy, implementing a composite pattern:

Finally, I created a convenience method for creating a script corresponding to the 1.0 version of my custom image.

OzZeroConfImageScript class>>oz10
^self new
release: '1.0';
yourself

Now you can try generating your script in a workspace,

OzZeroConfImageScript oz10 generate

and see the generated results in your working directory!

Customizing a vm script

A custom vm script is defined by a subclass of ZeroConfVMScript. ZeroConfVMScript defines, as its image friend, some vm common behavior, such as the release number and virtual machine type(i.e., if it is a jitted vm or not), which we will use in our script.

Integrating everything and automating generation

As I already showed you, every script understands the message #generate to generate itself. However, we may want to generate many scripts, and combine them. The ZeroConf infrastructure already provides for that the ZeroConfCommandLineHandler. The ZeroConfCommandLineHandler is a command line handler that knows which are the scripts we want to generate, combines them appropriately and generates them. So we will subclass from ZeroConfCommandLineHandler and specialize it to fulfill our needs.

In the recent times, there appeared many many ways to leverage the installation and deploy of Pharo applications. These installation approaches enhance significantly the experience of using Pharo, by simplifying either dependency management with OS libraries, enabling to write deploy bash scripts or loading prebuilt images for any (and many) taste(s).

However, if you are not in the Pharo mailing lists, you probably have not heard about many of these installation mechanisms, and therefore, you cannot enjoy them. So, let’s summarize a bit some of these mechanisms, at least the ones I know. If you know some more, contact me so we can include it.

Manual download from the webpage

Downloading Pharo manually is the easiest but more primitive approach. Proceed to the download page [1] and download the flavor of Pharo you like the most. You will find in here the 1.3, 1.4 and 2.0 releases, plus the option to load the latest (still in development) version of Pharo 3.0.

Focusing on what is available for Pharo 2.0, you can either install

under the category “Pharo Installers”: a package specific for your operative system containing both virtual machine and the image with the runtime and development environment

under the category “Custom Downloads”: the possibility to download them by separate. This option is useful if you already have a virtual machine and only want a new image to play with.

Virtual Machine PPA for Ubuntu linux

There is a PPA available for Ubuntu users (probably it works also for any distribution using apt-get package manager) which is in charge of downloading the virtual machine and its dependencies, simplifying its installation and deploy. We thank Damien Cassou for taking finally the initiative of creating the PPA!

ZeroConf scripts

The ZeroConf scripts[3] are already built bash scripts easing the download and installation of pharo. They are scripts served by get.pharo.org which can be parameterized for getting the pair vm/image you want.

Their usage, as written in the ZeroConf webpage can be resumed as

curl url | bash
#or if curl is not available:
wget -O- url | bash

where url is replaced by the formula vmVersion|imageVersion|vmVersion+imageVersion

The Pharo launcher is an initiative of Erwan Douaille and Damien Cassou. And of course, you can contribute to it. In their release notes they added some points they would like to enhance in this project:

In the last time I was working on the system bootstrap again, trying to enhance the process, lower the imposed limitations… And I have a pretty new version. I’ll present here a summary of what I’ve learnt in the last time:

A review on the process steps, with exemplar code snippets

Some of the new key ideas and new infrastructure

And improvements over the last version :)

What are we going to bootstrap

For the sake of simplicity, we will not bootstrap a full Pharo (because that would include preparing for example Morphic to be bootstrapped, or to be loaded/unloaded). Instead, we will bootstrap an adapted version of MicroSqueak from John Maloney, which we re-baptized as Pharo Candle. Pharo Candle has only 50 classes, as you can see in the Github repo, and limited features. All classes from Pharo Candle has a PC prefix, which is not important for the bootstrap, since it will not let be name collisions.

To understand the rest of the post, the following is important: we will take a textual and static definition of the system (methods, classes…), and load it into our image. For that we will parse the files and include them as definition objects. These definition objects will ease the access to the data needed instead of accessing lots of lookup and symbol tables and stuff in a procedural way…

Some infrastructure basics

The last bootstrap implementation was very coupled to Pharo and its internal implementation. To create objects, to initialize classes, we relied on the existing system, on sending messages, on the VirtualMachine interpreter and infrastructure. So the first step for this new version was to decouple that. Decoupling the existing system from the new system that is about to be created. To do that we encapsulate the state of the new system inside an object, which we call an object space.

An object space reifies the new system. It is an object that understand messages such as “create an object”, “translate this string to a string of the new system”, “register this object as a class in yourself” or even “execute this piece of code”. This object space lets us structure the creation of the new system in a more comfortable way.

Our next concern, is not installing objects from the existing system into the new system. We want the new system to be transitively closed. So, the best is not to have direct references from the existing system to the new one. The best is to control those references carefully. Then, every time an object space gives you a reference to one of his objects, he really gives you a proxy. That proxy allows you to manipulate, at some level, the object inside the object space. A proxy can give and revoke permissions on the inner object, and make transformations or validations when necessary to keep the model consistent. Since these proxys may perform meta-operations on the new system objects, we end up calling them mirrors at some point.

Finally there are VirtualMachine limitations when executing code on the new space. We introduced a new piece in the puzzle to overcome them: our own language-level interpreter. In this particular case, we are using an AST interpreter, but we could, if available, use any other kind of interpreter. The important thing about using our own interpreter is controlling the semantics of the new system, and leverage the VM’s limitations.

Where do we start?

When the bootstrap starts, there is nothing. There are no objects, no structures, nothing. So we have to build everything from the start. And the question is… where do we start?

Our system is composed by objects, and thus, we need to create objects. And every new object may have pointers, which are initially pointing to nil. But what if there is no nil in our system? So, let’s build a first nil, so all objects created later can point to this nil object.

Creating the first object

As we decided before, we create our first nil object. However, there is a question that arises when we try to create nil. How can we create an object without a class? The answer is, so far, that until we have classes we will create objects without classes. We will not care about their classes and we will solve that later, once classes are created. Fortunately, this problem is present with very few objects.

Since we have no class to create nil, we have to specify the format of this object. That is, the amount of memory to be allocated for it, the amount of slots, and if they are pointers or bytes or what. The format is known by the definition of undefined object.

Fortunately, nil has no pointers to other objects, except for his class, simplifying the process. We will set nil’s class once we create it later.

We create the classes

Creating a class is a complex operation. A class has a metaclass. And it has a name which is a symbol (unique in the system). And it has a superclass, which may have not been created yet. And it has a dictionary of class variables. And… a lot of stuff.

Our objective is to keep this bootstrap the simplest. And for that, we will delay all the complex operations to the moment when they are not so complex. In this step we create empty classes. We only initialize their format with a SmallInteger, and we let the rest of their pointers pointing to nil.

The first step for creating a class, is to create its metaclass. And to create a metaclass, we need the class Metaclass. This Metaclass, in a ST-80 like model, follows the Metaclass<->Metaclass class loop. That is, Metaclass is an instance of Metaclass class, and Metaclass class is an instance of Metaclass, as shown in blue in the following figure.

We create the first Metaclass and Metaclass class as objects without class, and then we make each one an instance of the other.

After this initial initialization is performed for every class, we finish by initializing the class variables. Class variables are represented by a Dictionary object. A dictionary object is an object with a complex structure, and the way to manipulate it depends on the nature of the system we are bootstrapping. The solution, so far, is to delegate the initialization of the dictionary to the dictionary itself.

For that we use a combination of the code of the dictionary and an AST interpreter. An AST interpreter needs to be initialized before its usage so later, all class variables can be initialized. As you can see in the code below, the AST interpreter usage is hidden inside the mirror implementation :).

Install methods

Now we have all classes of the new system created and initialized. We can start installing all their methods. Before, we should declare all global variables of the system, so the compiler knows how to bind them correctly. After that, we take the source code of all methods from the system description and compile them. The compilation gives us as result the bytecode of the method + the literals. This method is then translated to a method in the new world, and installed into the new system.

Then, for each class, we have the following code to create and install the methods:

"build the methods as instances of this system"
newMethods := aMethodBuilder
methodsForBehavior: mirror
fromDefinition: aBehaviorDefinition.
"create a method dictionary of the new system"
newMethodDict := objectSpace createMethodDictionary: newMethods size.
newMethods do: [ :m |
"install a method from this system to the other"
"the translation to a method to the other side is made inside"
newMethodDict installMethod: m
].
"we set the method dictionary to our class"
mirror methodDictionary: newMethodDict.

Initialize the system state

Finally, we initialize the system state, with the aid of the AST interpreter. This last step consists mainly in:

This last week I’ve been involved somehow in several discussions which, although not explicitly, talked about optimizations. In particular, premature optimizations. Of course, we all know they are evil. Do we? I’ll not discuss today on optimization techniques, but on what should an IT professional think about when thinking about optimizations in his programs.

The main arguments I heard this week of people supporting premature organizations were:

A guy using some technology X, should know the underlying details of X, or he will fail. Let’s say, if you are a Java programmer, you must not only know there is a GC, but also how it works.

A tech guy should be always conscious of the resources used. i.e., not to store a lot of objects in caches because memory is a finite resource.

Assumptions on what should be faster. Using or not a macro in C for example.

And these arguments are not even totally wrong. But they are not so totally true as they were stated.

Premature optimization is the root of all evil

You know Donald Knuth? This phrase (attributed to C.A.R. Hoare, btw) became famous because of this paper he authored. The interesting thing is that this phrase, when used, is taken out from context. The original phrase is:

Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.

Being optimal is not all that matters

When writing software, running optimal is not the only variable to analyze. We want programs to be debuggeable, understandable, extendable, maintainable. And sometimes, optimal code can be ugly, and can be hard to change, or to fix bugs on it.

So, let’s look at this piece of code written for Pharo. This piece of code takes a string, splits the substrings taking the space character as separator, excludes the substrings that do not exist as symbols in the system, and then it converts them to symbols.

splitOn: creates a new (and temporary) collection. It also creates strings for each substrings and copy all the contents into those strings;

reject: iterates over the result of #splitOn:, creates a new (temporary again) collection;

collect: iterates over the result of #reject:, creates a new collection to put the results of #asSymbol

Finally, there are two intermediate collections that are discarded, some substrings are created by copying all the contents and finally discarded (cause we only care about the symbols). Yes, that is inefficient: lots of temporary allocations that could launch the GC, several iterations over collections… we could do better. Let’s see an alternative version Camillo Bruni (Rmod, Inria) suggested to improve in terms of speed and memory usage:

This new version, which btw ends up with the same result, is pretty much more efficient:

Streaming on the result causes only one collection allocation without temporal ones;

Some special methods introduced into String to avoid extra collection allocation, and substring copies;

One collection means only one iteration :)

But wow, the code became much more complicated (given the simplicity of the example), and less object oriented. We do not manipule so easily the substrings by sending messages to them, we have instead the indices into the source string. Our code is much more aware of the problems we stated before, and recurring to lower level APIs to avoid them.

Now, extend these ideas to a whole large application. Hundreds or thousands of classes written this way. We write methods of tens (or hundreds, why not?) of lines of code to avoid message sends (and therefore method lookups), we avoid at the maximum object allocation and go for an if based solution… and soon we will have lots of duplicated code, stringy code everywhere… And yet I can tell you (just guessing :^) your program will not be tons more optimal. What? Now my code is so hard to maintain and not very much faster? Not cool…

Being optimal when optimal matters

So let’s say we have this function that takes 100.000 database rows, makes some calculations, and show a simple result to a user. It takes 1 second, which is a lot for a nowadays machine. But this function is used once per hour…

Now take the code that evaluates the bytecode that access an object’s field. It gets executed maybe some lots of thousands of times per second? So, if this operation starts to take 1 second… :)

Or take this application that stores data on background, but when restoring wants to be as fast as possible to give a really good user experience. Will you care how much it takes the storing operation?

Do we really have to spend a lot of time optimizing code that is almost not used? Or code that does not need to run that fast? Wait! My application runs ok, do I really have to optimize something?

As Knuth says, 97% of the code is not critical. Only 3% deserves to be optimized.

Understand when and where optimal matters

So now you know the key point (optimizing when it matters), and you understood it mattered in your case. Time to find that 3%. And it may be not so obvious…

Thanks engineers invented profiling! Just look a bit around, there are tons of tools to help you understand what you’re doing wrong: where is memory allocated?, and of which type?, is the GC launched so often? is a time consuming function executed too many times? Profiling is a technique that should be on every software engineer tool-case.

The rules of optimization

As a conclusion, today I found this link I want to share about the rules of optimization. And I think they are a pretty good guideline. When you are thinking on making an optimization:

First time: Don’t do it!

Second time: Don’t do it yet!

Third time: Ok, but you first profile and measure, and then optimize

There is much to lose when only thinking on the optimal solution to a problem in terms of machine resources. Remember people’s time to understand the written code, to adapt it to new situations and to fix bugs on it is also a valuable resource.

Pharo 2.0 release includes the Keymappings library. Keymappings is a library for configuring shortcuts for the current UI library (Morphic). It models concepts like: shortcuts, key combinations, event bubbling. It is a very simple library which I’ll introduce gradually in this post.

Key combinations

Keymappings main task is it’s ability to associate a key combination to an action. So we have to build up those key combinations. The simplest key combination is the one that gets activated when a single key is pressed. We call these combinations single key combinations:

Although, usually key combinations get a bit more complex. It is very common to combine single keys with meta keys or modifiers. These meta keys or modifiers are the well known ctrl, shift, alt and command keys. To build a modified key combination we can do as follows:

It is important to notice that all key combinations are not case sensitive. It takes a and A characters as the same, since they are the same key.

Have you ever used emacs, Eclipse or Visual Studio? Then you probably know sequences of key combinations that launch one only action. Like Alt+Shift+X, T (to run JUnit tests in eclipse)? So keymappings can do that too:

$a command shift, $b shift. -> "key sequence (Cmd+Shift+A, Shift+B)"

Sometimes, you want to configure an action to be activated in two different cases. Those are Keymapping options, and get activated when one of the options gets activated:

$a command | $b command. -> "key combination (Cmd+A or Cmd+B)"

Finally, since Pharo is a cross platform system and it is important to provide a good user experience by with the most suitable shortcut layout, keymapping implements platform specific shortcuts, which get activated only when running in the specific platform:

Shortcut configurations

Now you know how to build key combinations for your purposes, you probably want to go to the action. Map those combinations to actions and make them work!

Single shortcut configuration

The simplest way to attach a shortcut to a morph is by sending him the #on:do: message. The first argument expected is a key combination and the second one is an action. In the example below, a workspace is created with two shortcuts:

when Cmd+Shift+A is pressed, the workspace is deleted

when Cmd+Shift+D is pressed, an information growl should appear yelling ‘this shortcut works!’

Shortcut categories

Sometimes you want to group and organize shortcuts in a meaningful way and apply them all together on a morph. Sometimes you want some morphs from different hierarchies to share the same group of shortcuts easily. Those groups of shortcuts are what keymapping calls Categories. A category is a group of shortcuts, so far (will change in the future) defined statically by using a keymap pragma on class side:

Shortcuts defined through the builder specify the name of the category they belong to, a default key combination, an action, and a description. All this metadata is there to be used as settings in the future.

Finally in order to get your morph handle those shortcuts you can use the #attachKeymapCategory: message as in:

Bubbling

Keymappings’ shortcuts bubble to their parent if not handled, up until the main world morph. That has two main consequences:

Shortcuts for your application can be designed in a hierarchical way and;

Every time a shortcut does not work for you, it means that a morph below you has handled it ;) (be careful with text editors that handle loooots of key combinations)

Future work

So far, so good, but there is some plan on Keymappings for Pharo 3.0 development, which I can anticipate:

Some API changes: #on:do: can be confused with exception or announcement handling. #asShortcut will probably be properly renamed as #asKeyCombination. There is an inconsistency between the #command and #ctrl messages…