The Essence of OOP: It's all messages, all the time.

Category Archives: Metaprogramming

What is an Object Space?

An object space is an object that encapsulates the execution context of an Essence# program. It is also responsible for initializing and hosting the Essence# run time system, including the dynamic binding subsystem that animates/reifies the meta-object protocol of Microsoft’s Dynamic Language Runtime (DLR.)

Any number of different object spaces may be active at the same time. Each one creates and encapsulates its own, independent execution context. The compiler and the library loader operate on and in a specific object space.Blocks and methods execute in the context of a specific object space. Essence# classes, traits and namespaces are bound to a specific object space. Even when a class, trait or namespace is defined in the same class library and the same containing namespace, they are independent and separate from any that might have the same qualified names that are bound to a different object space.

In spite of that, it is quite possible for an object bound to one object space to send messages to an object bound to a different object space. One way to do that would be to use the DLR’s hosting protocol. That’s because an Essence# object space is the Essence#-specific object that actually implements the bulk of the behavior required by a DLR language context, which is an architectural object of the DLR’s hosting protocol.

The C# class EssenceSharp.ClientServices.ESLanguageContext subclasses the DLR class Microsoft.Scripting.Runtime.LanguageContext, and thereby is enabled to interoperate with the DLR’s hosting protocol. But an instance of EssenceSharp.ClientServices.ESLanguageContext’s only real job is to serve as a facade over instances of the C# class EssenceSharp.Runtime.ESObjectSpace. And EssenceSharp.Runtime.ESObjectSpace is the class that reifies an Essence# object space.

So, if you are only interested in using Essence#, and have no interest in using other dynamic languages hosted on the DLR, there is no need to use a DLR language context in order to invoke the Essence# compiler and run time system from your own C#, F# or Visual Basic code. You can use instances of EssenceSharp.Runtime.ESObjectSpace directly. The only disadvantage of that would be that using other DLR-hosted languages would then require a completely different API (e.g, using an IronPython library from Essence# code requires using the DLR hosting protocol, and hence requires using a DLR language context).

The advantages of using instances of EssenceSharp.Runtime.ESObjectSpace directly would be a much richer API that is far more specific to Essence#.

You can get the object space for the current execution environment by sending the message #objectSpace to any Essence# class (even to those that represent CLR types.) And the Essence# Standard Library includes a definition for an Essence# class that represents the Essence#-specific behavior of instances of the C# class EssenceSharp.Runtime.ESObjectSpace. It’s in the namespace CLR.EssenceSharp.Runtime, and so can be found at %EssenceSharpPath%\Source\Libraries\Standard.lib\CLR\EssenceSharp\Runtime\ObjectSpace.

There are many ways that Essence# object spaces might be useful. One example would be to use one object space to host programming tools such as as browsers, inspectors and debuggers, but to have the applications on which those tools operate be in their own objects spaces. That architecture would isolate the programming tools from any misbehavior of the applications on which they operate–and vice versa.

Just because there is as yet no Essence# GUI library, and therefore no native Essence# code browsing tools, doesn’t mean that the intrinsic reflecting capabilities of Essence# can’t be used. In fact, scripts are provided in the shared scripts folder that provide at least some of the functionality traditionally provided by code browsers:

ShowAllMethods: The ShowAllMethods.es script can be used to print out the names and declaring class or trait of all the methods of a class or trait. The subject class or trait must be passed in as an argument, as in the following example which will print out the names and declaring class or trait of all the methods of class Array to the Transcript:

es ShowAllMethods -a Array | more

ShowAllMessagesSent: The ShowAllMessagesSent.es script can be used to print out all the messages sent by each method of a class or trait. The output is cross-referenced by the sending methods, and each such method specifies the class or trait that declares it. The subject class or trait must be passed in as an argument, as in the following example which will print out the names of all the messages sent by each method of class Array to the Transcript:

es ShowAllMessagesSent -a Array | more

ShowAllSenders: The ShowAllSenders.es script can be used to print out the names and declaring class or trait of all the methods in the object space that send a specified message. The subject message selector must be passed in as an argument, as in the following example which will print out to the Transcript the names and declaring class or trait of all the methods in the object space that send the message do:

es ShowAllSenders -a #do: | more

ShowAllSendersInHierarchy: The ShowAllSendersInHierarchy.es script can be used to print out the names and declaring class or trait of all the methods of a specified class that send a specified message. The subject message selector and the subject class or trait must both be passed in as arguments, as in the following example which will print out to the Transcript the names and declaring class or trait of all the methods of OrderedCollection that send the message do:

es ShowAllSendersInHierarchy -a #do: -a OrderedCollection | more

ShowUnimplementedMessages: The ShowUnimplementedMessages.es script can be used to print out the names of all the messages sent by the methods of a specified class or trait to the pseudo-variable self for which the specified class or trait has no implementing methods. The subject class or trait must be passed in as an argument, as in the following example which will print out to the Transcript the names of any messages sent to self by the class OrderedCollection that send messages for which OrderedCollection has no implementing methods:

es ShowUnimplementedMessages -a OrderedCollection | more

Note: Classes that represent CLR types typically have virtual Essence# methods that don’t need to be formally declared, because the Essence# dynamic binding system will automatically bind to and invoke the methods of a CLR type, provided those methods have less than two parameters. Messages sent in order to invoke such methods of CLR types will unavoidably show up as “unimplemented messages” when using the ShowUnimplementedMessages.es script.

ShowTraitUsageConflicts: The ShowTraitUsageConflicts.es script can be used to print out the name and declaring trait of all methods which were excluded from a trait usage expression due to the fact that methods with the same selectors were declared by two or more of the traits combined in a trait usage expression. The subject class or trait must be passed in as an argument, as in the following example which will print out to the Transcript methods excluded from the trait usage of ReadStream because two or more of the traits used by ReadStream had the same method selector:

There are three issues that you will encounter when using Essence# that involve CLR types:

Obtaining a CLR type as a value that can be stored in a variable, passed as a parameter or sent messages;

Converting a value from one CLR type to another; and

Creating instances of CLR generic types.

Obtaining A CLR Type As An Essence# Value

The Essence# class System.Type (in the Essence# namespace CLR.System, which corresponds to the .Net namespace System) has many class messages that answer commonly-used CLR types: For example, the expression Type string evaluates to the .Net object that represents the .Net type System.String and the expression Type timeSpan evaluates to the .Net object that represents the .Net type System.TimeSpan. [Note: Those messages are unique to Essence#; the actual .Net class System.Type doesn’t support them. You can add Essence#-specific instance methods or class methods to any .Net class or struct; they just won’t be available outside of Essence#.]

Of course, if you have an instance of the type, you can just send it the message #getType (which is a message to which all .Net objects will respond.) Another way to get a CLR type is to send the message #instanceType to the Essence# class that represents CLR values having that type.

If the CLR type can’t be obtained using one of the techniques explained above, the fallback is to encode the assembly-qualified name of the type in a string, and then send the string the message #asHostSystemType, as shown in the following example:

In most cases (but not all,) the #asHostSystemType message will fail if the fully-qualified name of the assembly is not provided. Two exceptions to that would be when the type is in the mscorlib assembly or in the EssenceSharp assembly.

CLR Type Conversion

Formally–in other words, in theory–Essence# is dynamically typed: The “type” of an expression is simply the powerset of all messages that can be sent to the expression without causing a method-binding error. And that is also true in practice, in that the compiler permits any message to be sent to any expression, and in the fact that the compiler permits any expression to be assigned to any variable or passed as any argument in any message, and in the fact that typing errors are only discovered and reported at run time.

But that does not mean that any expression can be passed as any argument in any message to any receiver without causing any run-time type errors, even in cases where the only messages that will be sent to the value of an argument won’t result in a method binding error. That constraint does hold in most other Smalltalk implementations, and it does hold when the method that will be invoked by the message is an Essence# method. But it does not hold when the “method” that will be invoked is a CLR method.

In Essence#, a CLR method is just a “user primitive”–although the “primitive” does not always need to be formally defined as a method in the Essence# source code, because the dynamic binding subsystem automatically binds Essence# messages to CLR methods that require less than 2 arguments, to CLR type constructors that require no arguments, and to any non-private properties or fields of a CLR type. And it is generally the case in all Smalltalk implementations that “primitive methods” will fail if the types of their arguments are not what they require–even if the primitive could have performed its intended operation simply by sending (using Smalltalk dynamic message dispatch) the right messages to the argument. Primitives generally don’t send messages to their arguments using Smalltalk dynamic message dispatch. And the methods of CLR types certainly do not.

Fortunately, the Essence# dynamic binding system will–if it can–automatically do the required type conversions for you, such as number type conversions, String/Symbol conversions, and any conversions defined by implicit or explicit conversion operators defined by a CLR type. It will even convert Essence# blocks into CLR functions with the required parameter type signature.

But the Essence# dynamic binding system isn’t magic. It can’t handle all cases. It’s simply not possible to convert any type into any other type. And even when it is, the run time system may not have the information required to do it correctly. And in other cases, the logic to do the required conversion simply hasn’t been implemented, for one reason or another.

CLR Array Types

One common case where a conversion might be possible, but the dynamic binding system doesn’t attempt to do it, involves arrays. From the point of view of the CLR, an Essence# array is an array whose elements have the type System.Object (an instance of the Essence# class Array,) or the type System.Byte (a ByteArray,) or the type System.UInt16 (a HalfWordArray,) or the type System.UInt32 (a WordArray,) or the type System.UInt64 (a LongWordArray,) or the type System.Single (a FloatArray,) or the type System.Double (a DoubleArray,) or the type System.Decimal (a QuadArray) or the type String (a Pathname.)

Unfortunately, there are many methods of CLR types that require an array having elements of some type other than the ones supported by Essence#. If the CLR method parameter requires an array whose element type is not the same as that of the corresponding argument at run time, then the binding to the method will fail.

Fortunately, if the conversion of the array to one with the required type is possible (because all the elements can be converted to the required CLR type,) you can code that conversion yourself in Essence#. The following example shows how (this is not new functionality; it just hasn’t been documented/explained):

CLR Generic Types

The CLR has three types of “generic type”: Open generic types, partially-open generic types and closed generic types. An open generic type is also called a generic type definition. An open generic type is one where all of its generic type parameters remain “open” because none of them have been bound to a specific type argument. A partially-open generic type is one where some, but not all, of the type’s generic type parameters have been bound to specific type arguments. A closed generic type is one where all of the type’s generic type parameters have been bound to specific type arguments. For the most part, a partially-open generic type is essentially the same as an open generic type. The distinction that really matters is between closed generic types and ones that have generic type parameters that aren’t bound to a specific type.

It’s possible to create instances of closed generic types, but it is not possible to create instances of open or partially-open generic types. And that can be a problem, because .Net class libraries typically only provide generic type definitions that define open generic types. So, although you can define an Essence# class that represents a .Net type that is an open generic type definition, you won’t be able to use that Essence# class to create instances of the type by sending it the normal instance creation messages (e.g, #new or #new:). Creating instances of such a type requires providing one or more types that will be used as the type arguments to construct a closed generic type from the open generic type defined by the .Net class library.

Fortunately, you can use Essence# to construct a closed generic type from an open generic type definition, as illustrated in the following example:

One good way to handle the case where the .Net type you want to use is an open generic type definition would be to define a subclass of an Essence# class that represents that type, and then use one of the messages above to construct a closed generic type that will be the instance type of the subclass.

Another good way to do it is simply to use the CLR’s syntax for closed generic types when specifying the instance type of the subclass. The following examples show both approaches:

Not so much: Smalltalk programmers have been coding with analogous capabilities since before Steve Jobs ever saw his now famous demo at Xerox PARC.

The Smalltalk development environment now has close to 40 years of maturity and experience behind it. And its cool features from 1979 still haven’t all been added to the IDEs of other languages, to say nothing of the ones added since then.

Essence# currently provides no support for any sort of GUI. Nor does it provide any code editing or code browsing tools. You might think that that means that the only way to develop code in Essence# is to use generic text editors, but such is not the case.

You could develop–and debug–your code using, for example, Squeak, Pharo or VisualWorks. That can be done because, unless your development environment’s Smalltalk compiler accepts (or worse, requires) non-standard syntax, the Essence# compiler can and will compile it. It will also accept/compile many of the common non-standard syntax extensions, such as qualified (“dotted”) names (VisualWorks, Smalltalk-X,) dynamic array literals (Squeak, Pharo) or even dynamic dictionary literals (Amber.)

So the heart of the issue is the differences in the standard libraries supported by other Smalltalk implementations. But that issue can be addressed by adding Essence# compatibility classes and methods to the environment where you develop your code, and by adding foreign-environment compatibility classes and methods to your suite of Essence# libraries/namespaces when you port your code to Essence#.

I expect a large body of such inter-Smalltalk compatibility code to be developed over time–some by me, and some of it contributed by others. In Essence# such special-purpose code modules don’t have to be installed by default. They can be partitioned into separate libraries that are only loaded when and as needed. So if you don’t need either the VASmalltalk or the Dolphin compatibility library, just don’t load them.

However, there is one obstacle in the path of this strategy: The file-out formats of other Smalltalk development environments are not at all the same as the Essence# class library format. Given that, it would actually be a lot of work to port code from VisualWorks, Squeak or Pharo to Essence#.

Or more correctly, it used to be a lot of work, because I’m using this post to publish and explain a solution to the problem: Code that will file out a class or class category from VisualWorks, Pharo or Squeak into Essence# class library format. The code can be obtained from the Essence# Tools Repository on GitHub. Although at present only those three development platforms are supported, it should not be too hard to figure out how to port one of them to another Smalltalk development platform. If you do so, please contribute it to the community.

The version of the code export tool for each of the three development environments is named EssenceClassLibraryExporter, and each has the same external API–which will be explained below. The one for Squeak is located here, the one for Pharo is located here, and the one for VisualWorks is located here. Warning: Don’t export code that you do not own, or that is not open source, unless you have the appropriate permission(s.) The VisualWorks class libraries published by Cincom are not open source, although the “Contributed” code may be (don’t assume; verify.)

After you’ve filed in EssenceClassLibraryExporter.st into Pharo, VisualWorks or Squeak, you use it by writing code in a workspace and executing it as a “do It.” Here’s what you need to know in order to do it correctly:

There are two key issues: 1) What code should be exported, and 2) what Essence# class library and namespace should it be exported to.

Because of the way the EssenceClassLibraryExporter works, it’s best to answer the second question first.

When exporting from VisualWorks, you probably should use the same namespace name in Essence# that you use in VisualWorks. But when exporting from a Smalltalk development environment that doesn’t have (or doesn’t typically use) namespaces, you’re faced with an architectural decision that you didn’t have to make until such time as you needed to export your code to Essence#. You can, of course, simply export your code to the Smalltalk namespace. But best practice would be to define and use a namespace in Essence# that’s specific to your project (unless you are deliberately extending a namespace authored by someone else.)

Namespaces in Smalltalk have always been controversial. There is a vocal and influential subset of the Smalltalk community that opposes them. So I am going to address that issue upfront:

First, I must point out that all ST80-compliant Smaltalk implementations that support “shared pools” actually do have and use namespaces. That’s precisely what a “shared pool” is: a namespace. Shared pools could be used to do all the things that namespaces do, but for the fact that the ST80-style compilers and code browsers generally used (with the notable exceptions of VisualWorks and Smalltalk-X, and soon Essence#) haven’t been designed to make such usage elegant and easy. But they could be.

And Smalltalk classes also act as namespaces, in that they can define “class variables” that can be used to change the meaning of references to variables defined in the SystemDictionary. And “shared pools” can have the same effect. Consequently, having formal support for modern, first-class namespaces adds no code comprehension or interpretation issues that were not already present even in Smalltalk-80 ca. 1983, as distributed by XSIS.

But the bottom line is that computer science long ago learned that different name binding scopes are necessary–and become ever more necessary as the size of the code base increases, and as the number on independent contributors increases. That’s why objects provide their own “namespaces” for their instance variables, so as to keep each object’s set of named instance variables separate and distinct from that of all other objects, even when the names of the variables are the same. It’s why each method defines and uses its own “namespace” of parameters and local variables–and why blocks do the same. The bottom line is that Essence# (and VisualWorks, and Smalltalk-X, and .Net, and most modern languages) provide namespaces that localize and encapsulate references to non-local variables for analogous reasons, to solve problems that are essentially the same, using a very similar solution, justified by issues and logic that are also largely analogous.

To the extent that multiple name binding scopes cause problems of comprehension or interpretation with respect to the meaning of named variable references, they do so universally for named variable references of all types and in all contexts: Block and method parameters, block and method local variables, function parameters and local variables, the fields of records/structs, the named instance variables of objects, the class variables and shared pools of ST80, the static variables of C++, C# and Java, and modules/assemblies/programs that contain multiple classes and/or functions.

If, on balance, programmers and the code bases they write get more benefit than harm from having separate name binding scopes for class variables, instance variables, parameters and local variables, why would that not also be the case for classes? Is that not one of the key benefits provided by having separate program files in the case of file-based languages, and separate images in the case of image-based Smalltalk systems? If separate name binding scopes are bad when provided by formal, first-class namespaces, why are they not also bad when implemented by having different images, or by having different program files?

For those reasons, opposition to formal namespaces as first-class objects has always seemed to me to be the equivalent of opposition to compilers by old-school assembly-language coders, opposition to structured-programming by old-school FORTRAN and COBOL coders, opposition to OO by old-school procedural programmers and opposition to dynamic typing by those who think a variable is semantically equivalent to the object it references, and/or that a class is semantically equivalent to the type (concept) it implements.

I’ve used VisualWorks with namespaces for a decade and a half, and have never encountered the issues that opponents of namespaces say they fear. Not once, not ever. That’s one reason it reminds me so much of opposition to dynamic typing.

Get over it. Move on: Formal namespaces as first class objects solve more problems than they cause. The proof is in the experience of those who’ve been using them for one or more decades now, and in the ever-increasing adoption of them in modern programming systems and languages.

Conceptually, an Essence# class library is an independently-loadable code module. It roughly corresponds to what other languages/platforms call a “package,” “module,” “parcel” or “assembly” (and these, too, were historically opposed by some; thankfully, they “got over it.”)

It’s important to note that an Essence# class library can contain one or more Essence# namespaces, that the same Essence# namespace can occur (be defined and/or modified by) more than one Essence# class library, that namespaces are an essential and foundational aspect of the .Net platform, and that Essence# namespaces were designed to deal with the fact that namespaces are so central to the way that the .Net framework operates.

You partition your code base into one or more class libraries so that your code modules can be loaded when and as needed as logically-consistent and cohesive units of functionality. You partition your code into one or more namespaces in order to isolate the named entities in your code base from unwanted name clashes with other parts of your own code base, or from unwanted name clashes with code from third parties (of which you may have no knowledge or even interest.) That’s two separate concerns solved by two separate mechanisms (“objects,”) which is proper OO design.

Key point: You probably should not export your code to the Essence# Standard Library, for the reasons explained in the Essence# documentation section on CodePlex, under the heading Handling multiple releases of Essence#. For one thing, it puts your work at risk of being overwritten when you merge with a new release of Essence# using the Essence# installation program. Using your own private Essence# class library avoids all risk of that happening, and provides several other important benefits that I won’t go into here.

Using the EssenceClassLibraryExporter

The following code snippet will export the class named MyClass to the Essence# class library MyClassLibrary in the Essence# namespace MyNamespace, assuming that MyClass is locally defined in the namespace Smalltalk (i.e., that’s the namespace in which the class is defined in the Pharo, Squeak or VisualWorks image you’re using):

By default, the exported code will be placed in the current working directory of your Smalltalk image. So in the case of the example above, the folder representing/implementing the target Essence# class library would therefore be the current working directory of your Smalltalk image, and the target namespace would be implemented/represented by the folder MyNamespace, which would be created as a direct subfolder in the image’s current working directory.

To use a different folder as the target Essence# class library, send the message #libraryPathPrefix: aPathnameString to the class EssenceClassLibraryExporter before exporting your code, as in the following example:

To export to a nested Essence# namespace, use a qualified named (“dotted notation”) for the target Essence# namespace name, as in the following example, which would export to the nested namespace MyRootNamespace.MyNestedNamesapce, at the absolute path ‘/Users/myUserName/Documents/Developer/EssenceSharp/Source/Libraries/MyClassLibrary/MyNamespace/MyNestedNamespace/’:

To export all the classes in a class category, replace the message #exportClass: aClass as in the examples above with the message #exportClassCategory: aSymbolNamingAClassCategory, as in the following example:

I haven’t discussed or explained much about the metaprogramming capabilities of Essence#. So let me do that now. First, I will present some example code, and then I will explain what it does, and how it does it:

What the example code does

The example code shows how to define instance-specific behavior for an object.

In this specific example, the array #(5 8 13)–which is an array of the three integers 5, 8 and 13–is made to also have the behavior defined by the trait Examples.TDebutant, and it is also bound to a method defined dynamically by the code of the example–one that no other array instance in the whole universe will have.

Because of the new behavior added uniquely to this one particular array object, it (and it alone) will understand the messages #introduceYourself (imported from the trait Examples.TDebutant) and #whatAreYou (dynamically defined and bound solely to the one array instance by the code of the example.)

So when the array instance #(5 8 13) is sent the message #introduceYourself, it responds by writing the text ‘Hello, world! I”m a Debutant!’ to the Console. And when it is sent the message #whatAreYou, it responds by writing the name of the superclass of its class to the Console. However, were either of those messages sent to any other array objects, the result would be a MessageNotUnderstood exception (that’s true even if any of the other arrays also contained the same three elements–only the object identity of a message receiver matters when one engages in this sort of metaprogramming.)

The point of the final statement of the example is to demonstrate that the array instance whose behavior was modified by the code of the example nevertheless continues to behave like a normal array object in all other respects. The final statement evaluates to the first element of the array that, when divided by 2, has a remainder (technically, a residual) of zero. So it evaluates to 8, which is the same result that one would obtain by sending the same message to any other array instance whose first even-numbered element was the integer 8 (assuming all of its elements were numbers.)

How the example code works

The code in the example does its magic by creating a new class (technically, a new Behavior, which you can think of as a “lightweight class” or “proto-class”), sending the new class the message #uses: Examples.TDebutant so that will import the methods defined by the trait Examples.TDebutant, sending the new class the message #addMethod: with a method literal object as the message argument, and sending the new class the message #superclass: with the class of an empty array object as the message argument (which will, of course, be the same class as that of the array #(5 8 13)). But all of that is only the appetizer; by itself, it would not be sufficient to modify the behavior of the array object #(5 8 13).

The master spell that takes us to the next level is performed by the magic message #changeClassToThatOf:, whose receiver can be any Essence# object whose object state architecture is compatible with the instance architecture of the class of the object that is the argument to the message. As you might infer from the name of this spell…uh, message, the effect of uttering (sending) it is to change the class of the receiver. In this case, it changes the class of the array object #(5 8 13) to be the new class we’ve created.

Obviously, the reason the array object #(8 5 13) acquires new behavior in the example code is because its class gets changed. And the reason it–and only it out of all other array objects–gets the new behavior is because we only changed the class of the one array object, and not the class of any other array. And the reason it gets the ability to respond to the messages #introduceYourself and #whatAreYou is because we specifically added methods with those names to the array object’s new class (note that the order in which we add the new behavior to the class and change the class of the array doesn’t matter, provided both of those changes are accomplished before we try to make use of them.)

And finally, the reason the array object continues to have array behavior after we’ve changed its class is because we made the array’s original class be the superclass of the new class we created. Since the array’s original class is the superclass of its new class, any messages sent to it will still end up being handled by the methods of its original class.