NOTE: This page is no longer active. It was superseded by Reader Conditionals and included in Clojure 1.7 and ClojureScript.

Problem

There are Clojure dialects (Clojure, ClojureScript, ClojureCLR) hosted on several different platforms. We wish to write libraries that can share as much portable code as possible while leaving flexibility to provide platform-specific bits as needed, then have this code run on all of them.

Use cases for platform-specific functionality currently handled by cljx:

Use case

Example (cljx)

Example (feature expr)

Platform-specific require/import

In .cljx file which is preprocessed into .clj and .cljs files:

In .clj file loadable in both Clojure and ClojureScript:

Exception handling, catching "all" - see issue below on why this is not a complete solution in either case.

Proposal summary

The feature expression proposal includes the following items:

Where

Item

Description

Example

all

Adds *features* dynamic variable

Set of keywords indicating supported features. Can be changed in a binding around reading. The feature set is open and can be dynamically modified in binding or modified on startup. Non-namespaced features are reserved by the platform. User features should be namespaced.

all

Defines well-known platform feature names

Each platform guarantees that *features* includes the platform feature.

clj, cljs, clr

CLJ

Adds system property to set custom features at startup

Use system property clojure.features on JVM startup to specify additional custom features. The platform feature will always be included, regardless of this setting.

-Dclojure.features=arch/osx

CLJS

Adds build option to set custom features at build time

Use build option :features which should be a set of keywords. The platform feature will always be included, regardless of this setting.

:build #{:js/node}

all

New reader syntax for feature include or exclude

Both feature include (#+) and exclude (#-) include a conditional expression and a form. The conditional expression is made from features (boolean value based on membership in feature set), and, or, and not. This is the same syntax used in CL feature expressions and cljx.

(defn my-trim [s]
#+clj (.. s toString trim)
#+cljs (gstring/trim s))

all

Excluded tagged literals do not need to have a data reader

It is allowed for an excluded feature to use an undefined tagged literal. For example, the #js tagged literal is not defined in Clojure but may used in a #+cljs feature expression.

(def init
#+cljs #js {}
#+clj nil)

all

New .cljc extension indicating a portable source file

Clojure, ClojureScript, and ClojureCLR should load both .cljc (which may have feature expressions) and their own platform-specific files as source files.

Issues

Below are important issues considered and current status.

Name

Description

Proposed

Status

Version checks

Should feature expressions support broader kinds of expressions (JDK or Clojure version ranges, etc)?

Not included

ClojureCLR delimited symbols

ClojureCLR supports a wider set of symbols via a |-delimited extension which cannot be read if in feature expressions from the CLJ or CLJS readers.

Add support for |-delimited symbols in CLJ, CLJS. Issues involved relate to

Not included (reconsider in future)

Portable "catch all" errors

It is currently difficult even with feature expressions to easily "catch all" in CLJS (like catch Throwable).

Note: This is orthogonal to feature expressions but important to the greater goal of portable code.

Existing code that uses a particular ns for macros (.clj file) may have conflicts if they use the same .clj for Clojure code in the same library.

This scenario is not too common and must be resolved by using non-overlapping namespaces for macros and ClojureScript code in a namespace.

No change needed

Tooling on raw source of portable files is more complicated.

Editors and other tools that work on the raw source files cannot determine a specific dialect of Clojure based on the file extensions. This makes it more complicated to enable advanced editor features that require analyzing code. See Colin Fleming's comments (re Cursive).

???

OPEN

CLJS File Loading Issue

Adding .clj files as valid file extensions means that parts of ClojureScript that currently specify a directory of .cljs files will include any .clj macro files already in that source tree and interpret them as .clj source files.

Some alternatives and their impact to existing file types or code:

Alternative

Description

CLJS .cljs

Macro .clj

CLJS .clj

CLJS build

CLJS Tools

CLJ Tools

Separate macro path

Specify separate paths for ClojureScript code and macros. The .clj files in the code path are then unambiguously CLJS code.

-

Move to new macro dir

-

Pass macro path through stack

Need additional macro path

-

Exclude namespaces

Specify a build option to exclude a set of namespaces from compilation. Macros would be explicitly specified there and could then be intermingled but ignored when pulling CLJS source from a directory.

-

-

-

Pass excluded ns list through stack

Need list of excluded macro ns'es

-

Mark macro namespaces

Specify ns meta or other marker in .clj macro files so that they can be excluded based on file inspection.

-

Mark as macro ns

-

Needs to parse all .clj files to check if macro ns

-

-

Mark regular portable namespaces

Specify ns meta or other marker in .clj source files so that they can be included based on file inspection.

-

-

Mark as portable

Needs to parse all .clj file to check if portable ns

-

?

Specify a portable file extension

Define a new unambiguous file extension (.cljc) to indicate a portable source file to be read in multiple host envs. (The .cljx extension is somewhat analogous although it only exists during preprocessing.) Explored in more detail in next table.

-

-

New file extension .cljc

Pull in .cljc files as CLJS source

Look for .cljc files

Look for .cljc files

Portable file extension consequences

This table explores the consequences and changes required if we added a new portable file extension across Clojure and ClojureScript (a proposed solution for the prior issue CLJS File Loading).

Where

Description of Impact

Change

CLJS

ClojureScript finds CLJS source files in a dir by .cljs extension

Look for both .cljs and .cljc as source

CLJ

Clojure looks for CLJ source files by .clj extension

Look for both .clj and .cljc as source

CLJS and CLJ

If both .cljc and either .cljs or .clj exist, which will be loaded?

Should prefer platform-specific file to common file (.clj or .cljs over .cljc). This enables the open extension case described below in Portable Libraries

lein, mvn, other build tools

Find CLJ source files by .clj extension

Look for both .clj and .cljc as source

lein cljsbuild and other CLJS tools

Find CLJS source files in a dir by .cljs extension

Look for both .cljs and .cljc as source

CLJ or CLJS IDEs (CIDER, Fireplace, Light Table, Cursive, CCW, etc)

Need to recognize .cljc as potentially both .clj and .cljs files

Update editors and IDEs to be aware of portable Clojure files

CLJS repls (Austin, Weasel, etc)

Need to recognize .cljc as CLJS file

Update each REPL tool

Detailed proposal

Feature Sets

The platform feature will be one of: clj, cljs, or clr. Users may supply their own features, which should, by convention, use namespaces. Platform-specified features will not be namespaced.

In Clojure, the initial feature set may be specified at start time with the system property clojure.features, which is a comma-delimited list of symbols to add as features. The platform feature will always be added, regardless of whether it is set in the system property.

There may be tagged literals or classes that are not known or available on all platforms. The reader must be able to read but avoid constructing these entities when they are in excluded feature expressions.

Example:

The #js tagged literal is not known on the Clojure platform, but the reader should read and avoid trying to instantiate it.

ClojureScript File Loading

When code needs to be loaded based on a namespace, ClojureScript will first look for .cljs files, then for .clj files.

ClojureScript currently expects source files to end (only) in .cljs. Changes must be made in several places to allow reading .clj files:

Building Portable Libraries

Some libraries may provide independent implementations of a library for both CLJ and CLJS and be packaged in the same jar. In this case, namespaces should not overlap between CLJ and CLJS:

Case 2: Mixed portable and non-portable CLJS and CLJ packaged in the same jar

Case 3: Library designed for open extension

A library designed for open extension isolates platform-specific code in one or more namespaces. The library code may compile against (but not include) a stub namespace or use a .clj mixed mode file to contain a default library implementation. The non-isolated code should all be .clj files that are portable across platforms.

A separate jar would supply the platform-specific version of the ns:

Other additional jars would be used to supply other platform-specific versions of the ns. A consumer would use the first jar + the selected extension jar.

Alternate Approaches

Copy / paste

One approach is to maintain two versions of the same file that are largely the same but modify the platform-specific parts in each copy. This obviously works but is gross.

cljx

The most common use is to use clj and cljs tags and write .clj and .cljs files for consumption by other tools/compilers/etc

optionally applies the same transformation interactively via installation of a REPL extension

It has been used successfully by a number of projects (see the cljx README for a partial list). cljx's limitations include:

It does not address portability of macros at all; it is strictly a source-to-source transformation. Macros continue to be written in Clojure, and must be rewritten or implemented conditionally on the contents of &env.

It does not provide any runtime customization of the "features" you can use; these are set either within build configuration (via cljx' Leiningen integration), or via the configuration of the cljx REPL extension. The latter technically is available for modification, but is not in practical use.

The set of provided "features" is limited to one for Clojure (#+clj) and one for ClojureScript (#+cljs). Further discrimination based on target runtime (e.g. rhino vs. node vs. v8 vs. CLR) would be trivial, but has not been implemented to date.

cljx expressions are typically applied:

Inside ns macro

Top-level forms

Occasionally internal forms where it's concise

(.getTime #+clj (java.util.Date.) #+cljs (js/Date.))

lein-cljsbuild "crossovers"

lein-cljsbuild provides a (deprecated, to be removed) feature called "crossovers" that provides a very limited preprocessing of certain files during the cljsbuild build process; a special comment string is removed, allowing one to work around the -macros declarations required in ClojureScript ns forms. Crossover files must otherwise be fully portable. Language/runtime-specific code must be maintained in separate files. However, (my) experience shows that this can quickly lead to the situation where one has to think a lot about in which file to put a specific function, in order to go though the whole preprocessing machinery. Functions are split into namespaces because of conditional compilation, and not because they belong to the same part or module of the program.

on clojure you get the list (+ 1 2) and on clojurescript the list (+ 2 3) and there is no indication that a compile time conditional was read. this means the conditionals will not be visible to anything that operates on datastructures rather than text.

how will macros be shared? I guess they would check`*features*` directly to decide if they want to emit something different for clojure or clojurescript

another thing is there is no indication that the above is basically a cond, besides the forms being written next to each other. compare to something like:

(feature-cond clojure (+ 1 2) clojurescript (+ 2 3))

it is possible that we have to have conditionals at read time because some forms are readable on one host but another (vars? syntax quote in the reader does some class lookups I think, on the otherhand, syntax quote should be rewritten as a macro)

if issues with readable/unreadable forms are addressed between hosts, feature-cond above could be a macro that checks `*features*` so no change to the compiler or reader

At first, I wanted to agree with you that it would be nicer to just have a Macro rather than introducing a new feature into the reader. However, after more thought, I think that this absolutely needs to occur at read time. The main reason is that, if it's a macro, then it will be visible to outer macro forms when it shouldn't be. Consider some macro which expects a symbol, but instead gets a (feature-cond ...) form. It will (assert (symbol? ...)) and fail, even though the feature-cond form might expand to a symbol, the outer macro has no way of knowing.

It seems like the expected thing is for this to happen almost equivalent to the C preprocessor, so the reader is the right place to do that.

The complaint I have with this feature is that now all the separate implementations of a given expression are housed in a single file. This closes off the system, disallowing future expansion (without getting a patch submitted upstream to the library provider). This gets to be a bigger issue when we start porting libraries to other platforms.

What I'd like to see is some sort of "patch" based system. I'd like to see some way where we can put JVM specific code into a single source file, and then allow users of the library to replace those JVM versions with CLJS versions, CLR, Python, or C versions as needed.

The common lisp method shown here is simple, but I have to say, it feels like a hack.

That assumes you have a runtime compiler, which ClojureScript does not. You'd need to run load-file in the context of the compiler, but ClojureScript (currently) emits top-level forms as JavaScript, rather than running them in the compiler's environment.

Just an update re: cljx, which I've been maintaining for some months now. It was rewritten for the 0.3.0 release to perform all of its transformations statically, i.e. it does not use runtime metadata and does not use the Clojure reader (or tools.reader for that matter). (The previous usage of runtime metadata and Clojure reader necessitated some compromises in the quality of the transformed output and in preventing the use of some language features tied to the compiler state/runtime, e.g. alias-namespaced symbols and keywords.)

I've also added some REPL integration, so that namespaces defined in .cljx files are transformed and loaded/compiled as appropriate for the current REPL session (either Clojure or ClojureScript).

These enhancements have made cljx a workable (and pleasant, IMO) solution to the problem of targeting multiple languages/runtimes from a single Clojure codebase.

I'll update this wiki page to include a link to cljx under "current solutions".

> Are those keywords ok? Is :jvm for Clojure and :js for ClojureScript better?

> Should ClojureScript add something like :rhino, :v8 or :browser as well?

:jvm isn't a good idea for Clojure, since :rhino is also on the JVM. If you were writing some ClojureScript code, and wanted to test for a Java package, you'd want to check for :jvm. Similarly, you'd want a :gclosure member as well, if there was ever an implementation that targeted JS without it.

I'd imagine the Clojure set could look like #{:clojure :java :jvm}

And the ClojureScript one could be #{:clojurescript :js :gclosure} potentially conj :jvm for Rhino

Because it will soon be possible to have #{:clojurescript :lua}

This raises an interesting point: Why a set? Why not a map? Here's an example Clojure map:

This way the #+clojurescript and #-js things will still work like set memberships. We could come up with some way to include an arbitrary predicate against map values, or we could leave that to be feature not available to the reader, only to macros against *features*.

A few issues: Since this happens at read time, it may be too late to decide :v8 vs :rhino, etc. Although, some JavaScript shops/frameworks/teams/whatever do multi-compile sources for different browsers. ie. Do conditional logic by browser version, server side, rather than client side, serve up the browser-specific javascript. In that case, you even could support a :browser key with the various information there.

I feel like this needs some kind of cond macro. I should be able to check for "do I have a runtime that optimizes foo? if so, do this, otherwise, do it the slow way like this" I realize that I could use #+foo, then #-foo, but what if I had two possible optimizations? I'd need #+foo(x) #+bar(y) #-foo(#-bar(y)) ;; seems like I need some kind of "else" expression.

RE: read-time vs macro-expansion-time. Seems like the right thing for this is read-time because of symbol resolution, but we should also consider some compile-time macro support against the same dynamic var. In theory, it should be workable with all the standard macros and stuff, since it's just a set or map, but we should still explore it a bit, so we know we're not missing something.

Elias Mårtenson responded to one of my questions oncomp.alt.lisp. Here's what he wrote:

I can address one of the quest, whether they are better implemented as a macro. The answer to this is clearly no. In fact, it couldn't be implemented as a macro, because at the time the macro is evaluated, the symbols making up the expression has already been interned by the reader. Thus, if the form references any packages that does not exist in your environment, you would get an error in the reader, before your conditional expression has had the chance to run.

I would be careful conflating those issues with common lisp with clojure. clojure's reader is not the same as common lisp's and clojure's symbols are not the same as common lisp's. I think the only part of the clojure reader that has those kind of issues in syntax quote, which tries to resolve classes.

I would like to see the feature system being open to use by libraries. In pallet, we recently introduced features to be able to write providers that work with multiple versions of pallet, and I think it would make sense to replace this with whatever feature system ends up in clojure.

The current proposal seems to support this, but it would be good to clarify this as a requirement of the feature system.

Again, if 'features (or whatever it is called) is a map, then you might be able to do something like (load-file (str "foo/" (:dialect features) ".clj")) or something like that... That wouldn't be possible as a bit-flag set.

What if we provided read-time variants of unquote and unquote-splicing?

Here's a splicing example:

The code inside the #~ and #~@ forms would be run at read-time and subject to *read-eval*. This gives you the full power of Clojure for conditional compilation at read time. We could provide some standard function and macro utilities for conditional compilation that are available both at read and macro expansion time. Allowing for something like:

This proposal gives you the full power of Clojure for controlling conditional reading and compilation without adding dramatically new ideas (it's basically just syntax-quote at read-time).

have you looked at what it would take to allow the reader to splice in forms at any point? with syntax quote it is a little easier because the expression has to be synax quoted to begin with before you can splice in to it.

syntax quote at least can limit the scope of splicing and unsplicing to syntax quoted forms. if any recursive call to read can (some how?) returning multiple values I think the entire reader would be much more complex. I have no evidence of that, which is what I asked if you had looked in to what it would take.

I have no doubt it is possible to do, I just don't thing I'd care for the result

Those complications make me wonder if #~@ serves any conceivable purpose, that #~ together `(~@) can't (or shouldn't) serve.

pushing conditionals in to the reader does not save us from requiring the code to be readable on all platforms. it has to be readable for the reader to even recognize it as a form to throw away

True, this should be an intended design constraint. We want to have the same basic clojure syntax on every platform, i.e. a very small superset of edn, right?

The reason we even need conditionals in the reader is that the reader already does some possibly platform-dependent side-effects, i.e. reader tags, defrecord instances and friends.

So let's just KISS the syntactic considerations goodbye for a moment:

That gets us to the real meat of the problem:

Say we have:

Under current reader rules, would the #platform tag have a chance to remove the form before the reader tried to apply the #asm tag?

I suspect that's not the case, which makes me wonder if it's worth changing the reader to allow reader tags to omit reader tags from their child form. This would imply the ability to generate new reader tags (and other reader side effects) in code a reader tag emits.

I also suspect that if we try to just allocate a couple of new reader table characters and work downwards from there, we will end up with pretty much the same code it would take to apply reader tags in a second pass, except it would be less general purpose.

We still might not want to expose it through reader tags, since it would alter their semantics on the input side. Opinions on this one?

This proposal additionally adds a lot of complexity to tools which work on unexpanded source, such as Cursive. In particular, with complicated conditional feature expressions it may no longer be possible to accurately identify which language a particular form is written in. Admittedly this is already a problem with CLJS macros, the only way to tell if the body of a syntax-quote block in a macro is Clojure or CLJS is to look at where the macro is actually used - I'd definitely welcome any steps to make that easier to test.

This isn't just applicable to Cursive, any tool which interacts with a debugger and allows breakpoints to be set will need to know the language of a particular line, since Javascript breakpoints will almost certainly be managed differently to JVM ones.

The lack of an explicit extension for mixed-mode files means that all tooling will have to assume that all Clojure files might potentially be treated as CLJS or code for any other platform - this hugely complicates things like completion and documentation lookup since docstrings may differ for a particular form across platforms. Currently things like numeric operators are functions in Clojure, and both functions and macros in CLJS. Identifying which the user is referring to is already complicated but with this change will be considerably more difficult. I much prefer the patch style system proposed by Timothy Baldridge where code is contained in platform specific files and combined somehow - this does sound like it would create a lot more code duplication, though.

I'd also like to ensure that the implementation will still maintain line/column information accurately, as CLJX currently does. I assume that's fairly easy, at least with the read time proposal.