Thursday, 18 September 2014

Over the past few months I have been working on creating some resources to help
people learn to use core.async. My goal is make this the best resource
available to help people get started with core.async and to document best
practices for composing applications with core.async.

It is still a work in progress, but it currently includes the following:

An introduction to channels, buffers, and basic channel operations

A tutorial/workshop that introduces core.async using both Clojure and
ClojureScript

A reference section of the core.async API, including both the Clojure and
ClojureScript sources

Wednesday, 20 August 2014

I have released liberator-transit 0.3.0. The main new feature in this
release is the option to set write handlers for Transit. To do so, just add
them to the options using the :handlers key just as if you were calling
transit/writer. For more details, check out the README.

Sunday, 10 August 2014

It didn’t take long. I have released liberator-transit version 0.2.0,
which fixes one bug and adds some configurability. Now, the response will not
have a charset added to the Content-Type header. Furthermore, you can now
disable verbose JSON output, make verbose JSON the default, and change the size
of the initial buffer used in serialising the response. You can read the
details in the README. Most importantly, this work paves the way for
setting write handlers for the Transit output, currently planned for release
0.3.0.

Friday, 08 August 2014

As part of my prepartion for the core.async workshop I’ll be giving at Strange
Loop, I have come across the need for a RESTful API for a web application I am
creating. Naturally, I thought this would be a good chance to try out
Liberator and Transit. I was pleased to see how easy
Liberator is to use. Moreover, it was nearly trivial to add Transit
serialisation support to Liberator. Nonetheless, I thought it would be good to
wrap it all up in a library ready for reuse, which I have unimaginatively
called liberator-transit.

The default Transit/JSON encoding is the non-verbose format. By adding the
“verbose” to the “Accept” header, liberator-transit emits the verbose JSON
encoding.

Since the MessagePack encoding is a binary format, I pipe the output through
xxd. Otherwise, unprintable characters are output.

How it works

Liberator has a built-in mechanism for formatting output of sequences and maps
automatically depending on the headers in the request, as seen above.
Moreover, it’s possible to extend this easily by adding new methods to the
render-map-generic and render-seq-generic multimethods in
liberator.representation. These methods dispatch on the media type that has
been negotiated and take two arguments: the data to render and the context.

Getting the verbose output to work was just a touch trickier. The parameters
to the media type are stripped by Liberator. As a result, it is necessary to
actually examine the request headers that were placed in the Ring map.
Fortunately, this is easy to do as it is a part of the incoming context.

Further work

This library is really quite simple. I spent far more time creating
test.check generators for the various Transit types than I did on the library
itself. It mostly exists to provide an even easier way to add Transit to
Liberator.

Nonetheless, if other people find a need for it, there is possibly room for
improvement, such as:

Is there a better way to handle the request for verbose JSON?

Should the library be configurable? If so, what should be configured and
what is the best way to do it?

Monday, 04 August 2014

Over the past month or so, I have started dabbling a bit with writing code for
the web/front-end. In large part, I have done so in an effort to have a nice
interface for the workshop on core.async that gave at Lambda Jam and will give
again at Strange Loop. I am comfortable enough picking up what I need to know
for the back end, but how can I create a site that doesn’t look like it came
from 1997? I have used Twitter’s Bootstrap before, and it’s not a bad starting
point. However, this time I decided to try using Polymer.

In this entry, I briefly introduce the motivating principle behind Polymer, and
I share some of the lessons I learned in creating Polymer elements with
ClojureScript. Finally, I answer the question: Why use Polymer and not
something like Om/React?

A quick introduction to Polymer

Polymer is a technology developed by Google built atop Web Components, a group
of related standards such as templates, HTML imports, custom elements, and
shadow DOM. Ultimately, the end goal of this combine chunks of styling,
markup, and logic into discrete reusable components. For example, embedding a
Google map into a page is as simple as:

<google-map latitidute="…" longitude="…"></google-map>

It couldn’t get much simpler than that, right? Well, Polymer has a whole
series of UI components designed to help create web applications implementing
Google’s new material design. These looked nice, so I decided to put
them to use.

Using Polymer with ClojureScript

Overall, using Polymer has been a relatively smooth process. For my workshop,
I primarily just used the Polymer element libraries, but I did create a few. I
primarily used JavaScript to implement the logic behind my elements. However,
there is one component where I absolutely needed to use ClojureScript, as it
demonstrates the different types of buffers for core.async channels. Rather
than creating a simulation, I used core.async itself. Learning to use the two
technologies together correctly took a bit of experimentation, but here are a
few tips for you should you decide to go down the same path:

Tip 1: Beware of what you build in your prototype

The typical call to register a Polymer element with a few properties in
ClojureScript will look something like:

However, you may want to create an element with more complex properties such as
JavaScript arrays or objects or even ClojureScript objects such as a core.async
channel for managing callbacks. As such, you may be tempted to do something
like:

However, as the documentation will warn you, this is not the right way
to do it. The object created here is a prototype that will be used in each
instance of your element. As such, any complex types will not get deep copies
and essentially constitute global shared state. Instead, you need to
instantiate these in the created callback, as in:

Tip 2: Pass your element reference around

If you don’t grab onto the this reference inside the scope of your prototype,
it can be very hard to get it later. As such, when calling your own functions from
a callback, be sure to pass the reference in. For example, the previous example could
be rewritten as:

When using ClojureScript’s advanced compilation mode, it will munge the
property names if you use the first method. The result will be a broken
element featuring lots of null or undefined values. Unless you are sure
you will never use the advanced compilation mode, just use the second syntax.

Tip 4: Accept mutability

As a Clojure/ClojureScript programmer, it maybe very tempting to try to store
your application state in an atom and deal with it a nice, sane way—just the
way Om does it. Well, to do so in Polymer while supporting Polymer’s two-way
data binding is just a pain. It can certainly be done, but I am not sure it is
worth the effort. Let’s take a look at a simple example to see what I am talking
about:

On line 3, we set up the mutable cell that will be visible to Polymer.

On line 4, we set up the immutable cell for our Clojure code.

On line 5, we set up a handler so that we can update the Clojure atom when
the property is changed from Polymer. For example, this may occur if the
property was bound to the value of some user input.

On line 8, we create our atom.

On line 9, we set up a watcher on the atom so that if a change comes from
the Clojure side, i.e. from a swap!, the change will be pushed into the
mutable cell that Polymer will see.

Finally, on line 13, we add the atom to the element’s state.

This is a lot of overhead for just one property. You can try to manage this
more easily by just having one atom with all application state in some sort of
map/object. This does mean you have fewer atoms to watch, but otherwise it
doesn’t help too much.

The reasons for this are twofold:

Now your template code has to reference a bunch of nested properties. What
used to be {{property}} now looks like {{state.property}}.

You must now use an observe block to get notifications of
property changes, and you must specify each path through your state map.

This is an awful lot of boilerplate. I suppose it may be possible to automate
much of it using a macro, but the question that remains is what does this
additional complexity buy you? In my opinion, though it may be a bitter pill
to swallow, just accepting and using the mutability of Polymer element seems to
be the most pragmatic route. Doing so allows things like Polymer's two-way data
binding to just work.

Tip 5: Properly include the ClojureScript in development mode

There are three ways to register a Polymer element that has logic:

Inside the element definition in the body of a script tag,

Referencing an external script inside the element definition, and

Loading the external script before the element definition.

With ClojureScript, we can choose from the latter two. Most of the time, the
second form is the most convenient:

However, this only works when one of the ClojueScript optimizations levels is
used, i.e. whitespace, simple, or advanced. If you are in development mode,
you would might be tempted to do something like:

However, if you are importing your element, e.g. <link rel="import"
href="my-element.html">, you will run into problems with the Google Closure
library. It doesn’t like this since it will attempt to inject your script’s
dependencies after the main page has loaded. Instead, while you are in
development mode, place all of your script tags in your main HTML page.

Tip 6: Be wary of processing the component HTML

This isn’t ClojureScript-specific, but I thought I’d include it nonetheless.
During development, I use some Ring middleware that injects script tags into
the served HTML to connect to my browser REPL environment. This generally
works well, but sometimes I saw some really bizarre behaviour. For example a
paper-input component refused to take focus, or a core-submenu component
hid its icon.

It turns out that Polymer uses a conditional attribute syntax:

<span hidden?="{{isHidden}}">You can hide me!</span>

If the expression is false, then the attribute will be omitted from the markup.
Some HTML parsers such as Enlive or htmlmin cannot process that. At least in
the case of htmlmin, it caused a mysterious hang. I received no warning at all
from Enlive.

The bottom line is: don’t try to process a component’s HTML unless you are sure
your processor can handle it.

Why bother with Polymer?

So given the existence of ClojureScript-friendly frameworks like Om/React, why
bother trying to write Polymer elements in ClojureScript? That’s a great
question. Here’s my take on it:

The two are not mutually exclusive. You can easily use both Om/React
components and Polymer elements on a page.

However, nesting them in each other proves a lot more tricky. I don’t think
there is any reason why you couldn’t embed Om/React component inside of a
Polymer element. Unfortunately, I am not sure it’s possible to go the other
way around. As best as I can tell, React expects to be React all the way
down to the DOM.

With the core and paper elements libraries, Polymer offers a compelling case
for using it. As I am not a web front-end developer, the ability to easily
use nicely-styled widgets and declaratively define animations is
particularly nice. I am not an HTML/CSS/JavaScript wizard, and it would
take me a long time to implement what Polymer provides. Using Polymer, I
can instead spend my time working on my application.

Frankly, I don’t think that ClojureScript is the ideal fit for Polymer. If
you need to use ClojureScript, as in my case where I needed to use
core.async, it’s certainly an option. However, if ClojureScript isn’t
absolutely necessary, consider sticking to JavaScript. In the end, it’s all
about choosing the right tool for the job.

After working my way through all of this code, what’s my take on Dart and
ClojureScript? I’ll start with addressing errors from my previous post and
then compare Dart and ClojureScript in the following areas:

Standard libraries

Ecosystem

Tooling

Debugging

Documentation

Outside the browser

Integrating with Polymer

Integrating with Angular

Asynchronous programming support

Errata from ‘First impressions’

Before I get to the comparisons, I would like to correct some things I got
wrong last time.

Static typing

Dart supports a mix of dynamic and static typing. You can program in Dart
without ever declaring a type. However, without the static types, static
analysis tools available for Dart will be less effective. Nonetheless, it is a
choice you get to make. For example, take the following program:

An IDE or the dartanalyzer program will flag line 6 above and give you a
warning that A value of type 'int' cannot be assigned to a variable of type
'String'. Nonetheless, the program will run just fine and output 2 2.
However, running the program with ‘checked mode’ enabled (either as a runtime
option to the Dart VM or a compile-time option when compiling to JavaScript)
will produce an exception at line 6 with a message akin to type 'int' is not a
subtype of type 'String' of 'withType'.

There is one place where Dart’s type system does irk me: only false is false
and only true is true. In the following program, all of the print statements
will print that the value is false (in unchecked mode):

In this case, the static analyser does not find any problems with the program,
but running it in checked mode will produce a type error since the expression
for the if condition must be a boolean type.

I have grown used to things like null pointers or zero being false and non-null
pointer and non-zero integers being true. I find needing to explicitly make an
equality check annoying.

Clojure has core.typed, a library to add gradual typing to Clojure
programs. However, using it is not nearly as seamless as is choosing to use
static typing in Dart.

Serialisation

This is one area where I got a lot of feedback last time. First, a few points:

It is idiomatic Dart to serialise to JSON.

Dart’s JSON library can automatically handle num, String, bool, and
Null types. List and Map objects can also be automatically serialised
subject to a few constraints.

Dart also has a serialistion library that serialises objects
reflectively or non-reflectively. It is fairly powerful and highly
customisable.

What’s the fallout from the above with respect to ClojureScript? I have a
couple of thoughts:

ClojureScripts’s extensible data notation (EDN) is richer than JSON,
making it deal with complex data structures and retain semantic information.
For example, is a list of items intended to have constant random access (a
vector) or contain unique elements (a set)? Additionally, it is
extensible and allows you to add support for application-specific types.

ClojureScript’s data-centric approach (using the built-in data structures
rather creating new types) makes serialisation very easy. If you follow the
same approach in Dart, you can enjoy many of the same benefits. However, as
soon as you introduce new types in either language, the situation becomes
more difficult.

In conclusion, it seems like if you stick with built-in types, both languages
have a comparable serialisation story. Nonetheless, I think that the idiomatic
Clojure approach to data combined with the richness of EDN gives it an edge
over Dart.

Dart vs. ClojureScript

Now that I have spent some more time with both languages, I can make more
informed and helpful comparisons between the two. One thing to keep in mind is
that these experiences come from going through the Dart tutorials, so they may
not play to ClojureScript’s strengths.

Standard library

Dart takes a ‘batteries-included’ approach to its standard library, making it
easy to write both web and command-line applications without depending on
external libraries.

In comparison, ClojureScript is very much a hosted language. While it has
superior support for functional programming and manipulating data structures,
when the time comes to move on from solving koans to writing a real
application, you discover you have to learn your target platform and how to
interoperate with it.

I think of it this way:

Developer: Hi, I want to write a web application.

ClojureScript: Great!

Developer: Um… how do I get a reference to a DOM element?

ClojureScript: That depends.

Developer: On what?

ClojureScript: Well, are you going to use JavaScript directly, use
Google’s Closure library, or try a ClojureScript library like Domina or
Enfocus? There are also radical alternatives to manipulating the DOM
like Pedestal and Om.

Developer: Uh… I don’t know.

Developer spends the next half-day evaluating the ClojureScript options.

Some days later:

Developer: Well, now I need to do something different. I need to use
HTML5 IndexedDB.

ClojureScript: Great!

Developer: Is there a nice library for that?

ClojureScript: Sorry, you’ll need to stick to JavaScript or Google
Closure. I hope you love callbacks and have brushed up on your interop skills.

Developer groans.

Even later:

Developer: Now I’d like to write a command line utility. I love
Clojure, but its overhead is just too big. Can I use ClojureScript?

ClojureScript: Absolutely!

Developer: Great. How do I get started?

ClojureScript: Well, all you need to do is learn Node.js. You’ll find your interop and callback management skills handy.

Developer: I don’t suppose there are any ClojureScript libraries that will make this much easier?

ClojureScript: Nope, you’re on the wild frontier. There are some nice Node modules, though.

Developer considers using Python instead.

Ecosystem

Of course, there is a world beyond the standard library. I can’t account for
the quality of libraries, but I can comment on the quantity:

Repository

Number

Clojars (Clojure)

libraries with a dependency on ClojureScript

188

total number of libraries

8,270

CPAN (Perl)

129,130

Maven Central (Java)

70,518

NPM (JavaScript/Node.js)

57,443

Pub (Dart)

690

PyPI (Python)

39,573

RubyGems.org (Ruby)

69,863

Both ClojureScript and Dart have far fewer libraries than other, more
established, languages. It seems that Dart does have more native libraries
available than ClojureScript, and both can take advantage of JavaScript
libraries (like those from NPM). It’s hard to tell which language has true
edge in terms of ecosystem.

Tooling

Dart ships an all-in-one package, Dart Editor, that includes:

The Dart SDK,

A specialised build of Eclipse specialised for Dart (the Dart Editor), and

Dartium, a special build of Chrome that includes the Dart VM.

Additionally, there is Dart support as plug-ins for:

IntelliJ IDEA and WebStorm

Eclipse

Emacs

Sublime Text 2

Vim

I tried the IntelliJ IDEA plug-in, and it seems to be largely on par with Dart
Editor, including features like static analysis, code completion, refactoring,
and debugging. I also tried the Vim plug-in, but all it does is syntax
highlighting.

I believe the Eclipse plug-in is the same as what is bundled with Dart Editor.
I cannot speak for the Emacs or Sublime Text 2 support for Dart.

All in all, the tooling story for Dart is pretty solid. For a beginner, there
is one download that contains everything to get started. For experienced
developers, there is a good chance there is some level of Dart support in their
preferred tools.

ClojureScript shares much of the same tooling story as Clojure. I’m not sure
what the state of getting started on Clojure is these days, but it seems like
Light Table is quickly becoming a popular recommendation.

As an experienced Clojure developer with an established Clojure working
toolset, I still find working with ClojureScript more difficult
than it ought to be. vim-fireplace supports ClojureScript, but I could never
get a configuration that gave me an entirely satisfactory REPL experience.
Even when I did manage to connect to the browser, it didn’t seem like my
changes were having an effect. Also, features like documentation lookup no
longer worked. I’ll accept that this may all be my own fault, but in the end I
was back in an edit/compile/reload loop that at times seemed painfully slow (up
to about 30 seconds for a recompile).

I have used Light Table with the ClojureScript and
the Om tutorials. Undoubtedly, having the instant feedback
from using the REPL makes development a much more efficient and enjoyable
experience.

Debugging

Although this falls into tooling, I thought I’d draw special attention to
debugging. As I mentioned earlier, you can use the debugger in Dart Editor,
Eclipse, IDEA, or WebStorm together with Dartium and get a pretty good
experience. Dart also prepares source maps when compiling to JavaScript,
easing debugging on the browser.

One common complaint about Clojure is its poor error messages. I have never
felt that was the case, but I came to Clojure with a lot of JVM experience. I
think that ClojureScript definitely lends credence to the notion. It’s
possible to enable source map support for ClojureScript, and it helps.
However, that also significantly slows down compilation speed. Given how
difficult it is to figure out what actually went wrong (often just a mistyped
name) from a JavaScript stack trace, I began to really appreciate the static
analysis support for Dart.

Documentation

Dart has excellent documentation. There are many tutorials, articles, code
labs, and examples all on-line and accessible from Dart’s home page. Many of
the tutorials also don’t presume a lot of specific development experience. As
an example, this is very helpful to an experienced back-end developer who is
just getting started with developing web applications for the browser.

There are good resources for ClojureScript, but they are spread about the web,
primarily in different people’s blog posts. The wiki on ClojureScript’s GitHub
has some good links, but I found most of my resources through web searches.
Additionally, many resources presume that you’re already familiar with the
underlying platform, making it just a bit harder to understand what’s going on
and how to get started.

Outside the browser

One of the selling points of Dart is that in addition to being able to create
browser-based web applications, you can write the server in the same language.
This is great; you can reuse some of the same code seamlessly in both the
client and the server. Additionally, it is possible to write command line
utilities and even interact with native libraries written in languages like C
or C++.

This is also possible with ClojureScript. While it is possible to write
ClojureScript servers that run atop Node.js, it is much more common to write
the server side in Clojure. As with Dart, the primary benefit of this
arrangement is the ability to share code between the server and the client.
Additionally, the fact that both ends can easily speak EDN to each other helps.
The trickiest part of this combination is that there are subtle differences
between Clojure and ClojureScript, and you have to be careful to keep your
cross-language libraries within the intersection of the two.

Integrating with Polymer

Polymer is a library built by Google on top of Web Components designed to
make it easy to create and reuse bits of functionality in web pages using
custom elements. It is largely JavaScript-based, but there is a port of it to
Dart, Polymer.dart. I don’t have any previous experience with Polymer, but
I got a nice taste of it working through the tutorials.

Working with Polymer.dart was a relatively easy experience. It, along with
Polymer itself, is still in a pre-release state. It’s not quite
feature-complete compared to Polymer itself, but seemed pretty solid as a
whole. I felt the trickiest part of using Polymer.dart was ensuring that the
two-way data binding on nested data structures worked well.

There is no equivalent library for Polymer in ClojureScript, so it’s necessary
to use ClojureScript’s interop with Polymer. As a result, you can do it, but
getting the data binding to work with ClojureScript is an absolute pain. A
good library would go far in making working with Polymer more palatable.

Integrating with Angular

AngularJS is an MVC framework for building large web applications, and
AngularDart is said to be the future of the framework. AngularDart is not
a strict 1:1 port of AngularJS. It works a bit differently and has
been said to be ‘Angular reimagined’.

This is my first exposure to Angular of any flavour, and my impression is that
it is a fairly neat framework. I enjoyed working my way through the
AngularDart tutorial, but it is clear that it is still a pre-1.0 product. It’s
not so much that the library is buggy, but the developer documentation is
lacking compared to other parts of the Dart ecosystem.

I have not tried much Angular with ClojureScript; I simply haven’t had the
time. There are multipleefforts to make Angular and
ClojureScript work better together. Given the popularity of AngularJS, I
wouldn’t be surprised if a good AngularCLJS library comes about.

In conclusion, it’s still early for ports of Angular to other languages.
However, given that Google is behind AngularDart and pushing it forward, I
expect it to mature much more quickly.

Asynchronous programming

Dart has a couple of key features for facilitating asynchronous programming:

Dart’s futures are quite similar to Clojure’s, though a bit richer. For
example, there is built-in support for chaining futures and for propagating
errors through a chain of futures. Dart’s streams provide a way of acting on a
series of events, such as getting data from a socket or reacting to UI events.
Both of these features help ameliorate the ‘callback hell’ problem that’s
associated with JavaScript.

In comparison, ClojureScript has no native support for either one of these
mechanisms. However, there is core.async, a powerful Clojure and
ClojureScript library for asynchronous programming. With it, it is possible to
write highly asynchronous code in a fashion that reads as if it were
synchronous. This makes the code significantly easier to reason about. David
Nolen has written a good introductory article about the power of
core.async. The main downside to core.async I have run into is that
it makes debugging more difficult due to the immense transformation of the code
at compile time.

In the end, while I think Dart’s approach to handling asynchronous programming
is fairly decent, it doesn’t have the power of core.async.

Final thoughts

Dart was designed to be a language that is easy to learn and can scale to large
projects, and I think it has accomplished that goal. If someone with a
background in a language like Java or C++ asked me about a language for
developing web applications, I would definitely recommend that they consider
Dart. With Dart, as with ClojureScript, it is possible to write both the
client and the server in the same language reusing the same code. In fact,
it’s probably easier in Dart than a hybrid Clojure/ClojureScript application.

Does this mean I think Dart is better than ClojureScript? In a word, no. I
would still recommend ClojureScript to Lisp aficionados and adventurous
programmers. Most importantly, I believe ClojureScript’s Lisp roots make it a
playground for innovation. I do not think something like core.async’s go
macro is possible in a language like Dart. With a working browser REPL,
ClojureScript should have the same highly-interactive development experience
that Clojure provides, and that makes programming a much more enjoyable and
productive experience.

In the end both Dart and ClojureScript are great languages. Dart is probably
the more ‘practical’ of the two, and certainly the easiest to pick up.
However, ClojureScript is more powerful and, in my opinion, fun.

About the book

As you may infer from the word ‘cookbook’ in the title, this is not a
comprehensive Boost reference or a book you expect to read from cover to cover.
It consists of scores of recipes, all of which follow the same formula:

An introduction the problem to solve.

A brief statement of the prerequisite knowledge for the solution.

A step-by-step walkthrough on how to solve the problem.

A brief explanation as to how/why the solution works.

Discussion about comparable/related functionality in C++11 or in other parts
of Boost.

These recipes are all grouped together into chapters, with each chapter having
a general topic such as resource management or multithreading.

The intended audience for this book is experienced C++ developers who may not
be familiar with all of Boost’s functionality and how compares with C++11.

Highlights

There are a lot things that I like about this book, in particular its emphasis
on C++11 and the way it introduces a topic and then gives you to resources for
more in-depth learning.

Though it’s been out for a couple years now, C++11 is still a relatively new
standard and it takes time for programmers and programs to adopt it. Many of
the new library features in C++11 are based on Boost libraries, and some Boost
libraries exists to help ‘backport’ new language features to older versions of
C++. The Cookbook does a very good job of letting the reader know whether
C++11 has the same or similar features as Boost and how they differ.

The other thing I really enjoyed about this book is how it gently introduces
the reader to Boost. There are a lot of Boost libraries, and the quality of
the official documentation varies from very good to cryptic. In the past, I
have avoided some libraries simply because I could never figure out how to even
get started using them. This book can make some of these more accessible by
giving me a simple example from which I can get a toehold. From that point, I
can start to make sense of the documentation.

Room for improvement

No book is perfect, and there are a couple of ways in which this book could be
more useful:

It is not exhaustive. Granted, the number of Boost libraries is enormous,
and some of them have limited applicability. Fortunately, the Cookbook
does a good job of covering the most useful libraries.

It only barely touches on the situation when you have to do deal with
different Boost versions. Some Boost libraries have source and ABI
incompatibilities between versions, and it can sometimes be a bit of a
nightmare to write code that has to support different versions of Boost. It
would have been nice to see if the author had any insights on how to handle
that issue.

Concluding thoughts

Boost C++ Application Development Cookbook is definitely worth considering if
you are a C++ developer that uses or would like to use Boost. It’s a good
reference to have handy when you find yourself in a situation where you think:
'There has got to be a library for this.’

While it was obvious that the Clojure wasn't losing the metadata, now we can
actually see how it gets processed.

Even though the metadata is expanded three separate times, it only shows up
twice in the compiled result. Apparently, when compiling a particular
class, the compiler keeps tracks of what symbols are being used and
deduplicates them.

This last point got me thinking: how sensitive is the compiler to the symbols
and metadata it encounters?

Modifying the macro-expansion

To find out how sensitive the compiler is to symbols and their metadata, we can
replace the ns form above with its macro-expansion and modify it just
slightly:

The only change is in the comparison on line 6 where we have removed the
metadata from the greeter.hello symbol. Functionally, this has no effect as
metadata doesn't affect equality. However, does this change the generated
code?

Examining the impact

As a matter of fact, it does. We have been careful so that the change has only
affected the greeter.hello__init class. Just looking at the class signature,
we can see this change made an impact:

On lines 5-8, the order of the map metadata was changed. I don't think it's
a significant change, but it is a change nonetheless;

On line 11, const__2 now holds a version of the symbol greeter.hello
without the metadata; and

On line 12, const__3 holds the reference to clojure.core, which used to
be in const__2.

When we examine the decompiled output of load(), we see that, as expected,
the version of the greeter.hello symbol that has the metadata is used for the
in-ns call and the version without the metadata is used in the comparison to
clojure.core:

Closing thoughts

First, a big thanks to Stuart Sierra for the tip about *print-meta*, it has
been really helpful.

Second, examining the Compiler source code, it becomes a
bit clearer what's going on: as the compiler encounters constants, it stores
them in a vector to be emitted later. Additionally, it ensures that constants
are not duplicated by using an IdentityHashMap, which relies on identity rather
than equality. As such, we can see how the two symbols (with and without
metadata) would be considered different.

However, what's not entirely clear is how the compiler knows that in the
original macro-expansion that the two symbols with metadata are identical. I
spent some time studying the compiler source, but it's somewhat hard to follow.
I could probably use a debugger to trace its execution, but that's an exercise
for another day.

Well, that's interesting. It's not any different than what we had before.
Where did the metadata go? Is it possible that it's all lost? That's not
likely. Further macro-expansion won't help, so let's start decompiling.

As before, const__0 refers to the clojure.core/in-ns var and const__2
refers to the clojure.core symbol. The big difference here is that Clojure
is no longer storing the greeter.hello symbol it creates. Instead, it
creates that symbol, 'adds'² the metadata to the symbol, and stores the
result in const__1.

This explains, to some extent, where the metadata went. It has been preserved
by the compiler, but how can the Clojure runtime access the metadata? The
greeter.hello__init class doesn't implement IMeta. It seems unlikely that
the runtime would scour the class constants of loaded namespace classes looking
for metadata.

Clearly, there is more to investigate. Let's take a look at the
greeter.hello$loading__4910__auto__ class next.

greeter.hello$loading__4910__auto__

This is the class that implements (with-loading-context (refer
'clojure.core)). It hasn't changed as a result of the new metadata, so let's
move onto the last generated class.

hello$fn__17

This is the anonymous function class that registers the namespace with Clojure.
It effectively implements the following Clojure code:

(commute @#'clojure.core/*loaded-libs*
conj
'greeter.hello)

Decompiling the class, we see that it hasn't changed much. As with the
greeter.hello__init, the class signature is identical. In this case, the
implementation of the static initialiser differs:

As before, the first four class constants refer to the vars for
clojure.core/commute, clojure.core/deref, clojure.core/*loaded-libs*, and
clojure.core/conj. For the fifth class constant, instead of storing the
symbol greeter.hello directly, it adds the metadata to the symbol before
storing it in the class constant. So what are the consequences of this?

Well, when invoke() is called on this anonymous function, it ensures that
clojure.core/*loaded-libs* will contain the symbol that contains the
metadata. So, this must be where the namespace metadata comes from, right?

Digging deeper

At this point in my investigation, I was a little bit confused. At first, I
thought that the namespace metadata must come from the *loaded-libs* var, but
that's just a ref to a sorted set of symbols. However, if I want to get the
metadata from a namespace at the REPL, I use (meta (find-ns 'greeter.hello)),
and the type of the object returned by find-ns is a Namespace instance, not
a Symbol. This got me thinking: what is the purpose of *loaded-libs* and
where is the Namespace instance created?

The purpose of *loaded-libs*

*loaded-libs* is a private var declared in core.clj. You can get
its content, a sorted set of symbols, via the loaded-libs function. It is
used indirectly by require and use to keep track of what namespaces have
been loaded. For example, when you use require without :reload or
:reload-all, the presence of the namespace name symbol in *loaded-libs*
will keep the namespace from being reloaded.

When using :reload-all, Clojure uses an initially-empty, thread-local binding
of *loaded-libs*. This allows all dependencies of the desired library to be
reloaded once, and the resulting set of loaded namespace name symbols is added
to root binding of *loaded-libs*.

As a result, this means that the metadata used for the *loaded-libs* is not
the metadata we get from the namespace object. For that, we'll have to take a
closer look at the metadata attached to the symbol at
greeter.hello__init/const__1.

Another look at greeter.hello__init/load

Looking back at greeter.hello__init, the namespace name symbol with metadata
is stored in a class constant, const__1. The only place where this constant
is used is in the load() method, which is decompiled as follows:

Namespace contains a static member called namespaces, which is a
map³ of namespace name symbols to Namespace object instances. When
findOrCreate is called and there is no mapping for the symbol yet, a new
Namespace instance is created and inserted into the map.

The Namespace class extends clojure.lang.AReference, which holds metadata
and indirectly implements clojure.lang.IMeta. As such, the Namespace
constructor uses the metadata from the namespace name symbol as its
metadata.

At last, we now know how a namespace gets its metadata. Looking at the
implementation of find-ns, we see that it just calls
Namespace/find which merely does a lookup in the namespaces map.

Parting thoughts

If the purpose of *loaded-libs* is primarily to keep track of what
namespaces have been loaded, does it really need metadata? Metadata doesn't
affect the equality of symbols. Arguably, adding metadata to symbols in
*loaded-libs* is a waste of memory.

One interesting finding is that at the very heart of Clojure is a bit of
mutable state. Keeping track of loaded libraries uses Clojure's concurrency
utilities and persistent data structures, but namespaces rely on a Java
concurrent collection.

The only thing special about classes from java.lang is that their
mappings are hard-coded. If a new java.lang class were to be added to
Java, it won't get imported by default until RT.java is updated with a
new mapping.

Imports are mappings of symbols to class objects, and there is not a
separate set of mappings for Clojure vars.

Since Clojure keeps track of what's loaded in two different places, it's
possible mess up the environment in strange ways. In particular,
remove-ns does not clear a symbol from *loaded-libs*, meaning that it
would be possible to get Clojure into a state where it thinks a namespace is
loaded when actually it is not.

Footnotes

Note that the names
of the anonymous function classes can be different each time you compile.

Unsurprisingly,
symbols are immutable and annotating one with metadata generates a new symbol.

In particular, it is
a ConcurrentHashMap, a concurrency-friendly implementation of a map from
Java. It is not a persistent data structure, but it does concurrent reads
and limited concurrent writes.

Dart is a relatively new programming language created by Google for client-side
web development. I've started looking into it a bit as I am planning a local
Dart Flight School event as one of the organisers for GDG
Houston. In particular, I spent about an hour going throught the
Darrrt code lab. Having done this, I was curious to see how the same
application coded in ClojureScript would compare.

First impressions

Let me start by saying I haven't done much JavaScript programming since about
1996, and I am generally more comfortable with the server side of web
programming than the client side. This makes a big difference in how I
approach Dart and ClojureScript, as I am relatively unfamiliar with the
underlying platform. Someone with a lot of experience with JavaScript and
client-side web programming may approach these languages differently.

Dart

My first impression of Dart is that it is very Java-like. It's features
include:

static typing

object-oriented

Java-like syntax

generics

exceptions

One of the major ways in which Dart is different Java is that it has lexical
closures and first-class functions, which are welcome additions. Additionally,
I think Dart has pretty good documentation and a fairly decent standard
library.

ClojureScript

This isn't my first time using ClojureScript; nonetheless, it's still different
enough in tooling and language details that it takes me a little bit more time
to get up and running compared to a traditional Clojure project. However, the
biggest problem that I ran into isn't the language, but the library.

With Dart, things like making asynchronous HTTP requests and manipulating the
DOM are baked right into the standard library. With ClojureScript, it's not
quite as straightforward. Do I use JavaScript primitives, the Google Closure
library, or look for a ClojureScript library that wraps Closure or raw
JavaScript? In the end, I used ClojureScript libraries such as
Domina and storage-atom.

Some comparisons

Even within this short code lab, there are a few places where the differences
between the two languages was remarkable.

Serialisation

The code lab includes both storing bits of information in HTML5 local storage
and loading data from an external file. In the case of Dart, I found it to be
somewhat painful as it required transforming things into or from JSON.

For example, a Dart PirateName class requires the following JSON read/write
code:

Built-in functions

While Dart's library has a fairly decent set of functions for doing things like
interacting with the DOM, it doesn't have some of the functionaly that
ClojureScript has built-in.

For example, compare (rand-nth names) to:

final Random indexGen = new Random();
name = names[indexGen.nextInt(names.length)];

Asynchronous code

With ClojureScript, you can use core.asyncas described by David
Nolen to write asynchronous code that reads logically. Dart has no
comparable functionality (though it does have functionality to support
asynchronous code in general). While using core.async for this code lab is
probably unnecessary, I can definitely see how it could make a more complex
application easier to understand and maintain.

Final thoughts

I can definitely see how Dart is a big improvement over JavaScript.
Nonetheless, I think that ClojureScript is fundamentally a more powerful
language. It's shortcomings in comparison to Dart seem to generally lie in
terms of libraries, which can be easily written by the community. As such,
while I have found Dart interesting, and will continue to learn more about it,
at this time I would lean more towards using ClojureScript in a project.