Jekyll2018-05-07T23:37:38+02:00https://code.thheller.com/from the shadows …random thoughts, probably related to web development
Problem Solved: Source Paths2018-02-08T12:30:12+01:002018-02-08T12:30:12+01:00https://code.thheller.com/blog/shadow-cljs/2018/02/08/problem-solved-source-paths<p>The question what exactly <a href="https://github.com/thheller/shadow-cljs">shadow-cljs</a> does differently compared to other ClojureScript tools comes up every now and again.</p>
<p>At this point the answer is “A lot!” but that is not a very satisfying answer. The long answer however is really long and I thought I make a blog post series out of it going into a few internal details about what problems exactly I was trying to solve with certain features. These sometimes might seem like tiny little details but for me they had a big impact.</p>
<p>I’ll leave it up to the reader to decide whether these are actual problems. They were for me, they might not be for you. Pretty much all the features in <code>shadow-cljs</code> came out of personal experience when building CLJS projects over the last 5 years. <strong>YMMV</strong>.</p>
<h2 id="problem-1-source-paths-in-cljs">Problem #1: Source Paths in CLJS</h2>
<p>ClojureScript by default does not have the concept of source paths, only “inputs”. An input may either be a file or a directory. In case of a directory it is searched recursively for <code>.clj(s|c)</code> files which then become inputs.</p>
<p><em>The problem is that all inputs are included in all builds by default.</em></p>
<p>Suppose you want to build 2 separate things from one code base, maybe a browser client app with a complementary node.js server. For sake of simplicity I’ll trim down the code examples to an absolute minimum.</p>
<p>Imagine this simple file structure</p>
<pre><code class="language-text">.
├── deps.edn
├── build.clj
└── src
└── demo
├── client.cljs
└── server.cljs
</code></pre>
<p>The client</p>
<pre><code class="language-clojure">(ns demo.client)
(js/console.log "client")
</code></pre>
<p>The server</p>
<pre><code class="language-clojure">(ns demo.server)
(js/console.log "server")
</code></pre>
<p>The build file</p>
<pre><code>(require 'cljs.build.api)
(cljs.build.api/build "src"
{:output-to "out/main.js"
:verbose true
:target :nodejs
:optimizations :advanced})
</code></pre>
<p>Compiling this and running the generated code produces</p>
<pre><code class="language-bash">$ clj build.clj
$ node out/main.js
client
server
</code></pre>
<p>As expected the generated output contains ALL the code since the config does not capture which code should be included. This will not always be this obvious since not everything makes itself known like this. It is very easy to overlook files and accidentally include them in a build when you wouldn’t otherwise need them. In theory <code>:advanced</code> should take care of this but that does not always work.</p>
<p>In addition the compiler “inputs” are not known to Clojure at all. So if you want to use macros you need to include those separately via the <code>:source-paths</code> of <code>lein</code> to ensure they end up on the classpath.</p>
<h3 id="solution-1-main">Solution #1: <code>:main</code></h3>
<p>The compiler option <code>:main</code> solves that as it lets us select an “entry” namespace and only its dependencies will be included in the compilation.</p>
<pre><code>(require 'cljs.build.api)
(cljs.build.api/build "src"
{:output-to "out/main.js"
:verbose true
:target :nodejs
:main 'demo.server
:optimizations :advanced})
</code></pre>
<p>Recompile and we only get the desired <code>server</code> output. If you always remember to set this you will be safe.</p>
<h3 id="solution-2-separate-source-paths">Solution #2: Separate Source Paths</h3>
<p>The more common solution is to split out the code into separate source paths from the beginning. So each “target” gets its own folder and each build will only pick the relevant folders.</p>
<pre><code>.
├── deps.edn
├── build.clj
└── src
└── client
└── demo
└── client.cljs
└── server
└── demo
└── server.cljs
└── shared
└── demo
└── foo.cljs
</code></pre>
<p>You will typically end up with an additional <code>src/shared</code> folder for code shared among both targets. I personally find this incredibly frustrating to work with.</p>
<p>I suspect that this pattern became wide-spread since <code>:main</code> was introduced some time after multiple source paths become a thing. I’m partially to blame for this since I was the one that <a href="https://github.com/emezeske/lein-cljsbuild/pull/173">added support</a> for multiple <code>:source-paths</code> in <code>lein-cljsbuild</code>.</p>
<p>I’m not saying that allowing multiple <code>:source-paths</code> is a bad thing, there are several very valid use-cases for doing this. I only think that this pattern is overused and we already have namespaces to separate our code. I’m all for separating <code>src/main</code> from <code>src/test</code> but <code>src/shared</code> goes way too far IMHO.</p>
<h2 id="shadow-cljs-solution">shadow-cljs Solution</h2>
<p>The solution in <code>shadow-cljs</code> is pretty straightforward.</p>
<ul>
<li><code>shadow-cljs</code> expects “entry” namespaces to be configured for all builds. Browser builds do this via <code>:modules</code>, node builds via <code>:main</code>. This is the default and you cannot build those targets without them</li>
<li>Multiple <code>:source-paths</code> are supported but sources are only taken into account when referenced</li>
<li>Multiple <code>:source-paths</code> are always global. You cannot configure a separate source path per build</li>
<li><code>:source-paths</code> are always on the classpath</li>
</ul>
<p>Although the implementation in <code>shadow-cljs</code> is entirely different it doesn’t provide anything that would not be possible with standard CLJS. I do believe that enforcing the config of “entry” namespaces however saves you from unknowingly including too much code in your builds. <code>shadow-cljs</code> just takes care of setting a better default so you don’t have to worry about it. You’ll see this pattern repeated in many of the <code>shadow-cljs</code> features.</p>The question what exactly shadow-cljs does differently compared to other ClojureScript tools comes up every now and again. At this point the answer is “A lot!” but that is not a very satisfying answer. The long answer however is really long and I thought I make a blog post series out of it going into a few internal details about what problems exactly I was trying to solve with certain features. These sometimes might seem like tiny little details but for me they had a big impact. I’ll leave it up to the reader to decide whether these are actual problems. They were for me, they might not be for you. Pretty much all the features in shadow-cljs came out of personal experience when building CLJS projects over the last 5 years. YMMV. Problem #1: Source Paths in CLJS ClojureScript by default does not have the concept of source paths, only “inputs”. An input may either be a file or a directory. In case of a directory it is searched recursively for .clj(s|c) files which then become inputs. The problem is that all inputs are included in all builds by default. Suppose you want to build 2 separate things from one code base, maybe a browser client app with a complementary node.js server. For sake of simplicity I’ll trim down the code examples to an absolute minimum. Imagine this simple file structure . ├── deps.edn ├── build.clj └── src └── demo ├── client.cljs └── server.cljs The client (ns demo.client) (js/console.log "client") The server (ns demo.server) (js/console.log "server") The build file (require 'cljs.build.api) (cljs.build.api/build "src" {:output-to "out/main.js" :verbose true :target :nodejs :optimizations :advanced}) Compiling this and running the generated code produces $ clj build.clj $ node out/main.js client server As expected the generated output contains ALL the code since the config does not capture which code should be included. This will not always be this obvious since not everything makes itself known like this. It is very easy to overlook files and accidentally include them in a build when you wouldn’t otherwise need them. In theory :advanced should take care of this but that does not always work. In addition the compiler “inputs” are not known to Clojure at all. So if you want to use macros you need to include those separately via the :source-paths of lein to ensure they end up on the classpath. Solution #1: :main The compiler option :main solves that as it lets us select an “entry” namespace and only its dependencies will be included in the compilation. (require 'cljs.build.api) (cljs.build.api/build "src" {:output-to "out/main.js" :verbose true :target :nodejs :main 'demo.server :optimizations :advanced}) Recompile and we only get the desired server output. If you always remember to set this you will be safe. Solution #2: Separate Source Paths The more common solution is to split out the code into separate source paths from the beginning. So each “target” gets its own folder and each build will only pick the relevant folders. . ├── deps.edn ├── build.clj └── src └── client └── demo └── client.cljs └── server └── demo └── server.cljs └── shared └── demo └── foo.cljs You will typically end up with an additional src/shared folder for code shared among both targets. I personally find this incredibly frustrating to work with. I suspect that this pattern became wide-spread since :main was introduced some time after multiple source paths become a thing. I’m partially to blame for this since I was the one that added support for multiple :source-paths in lein-cljsbuild. I’m not saying that allowing multiple :source-paths is a bad thing, there are several very valid use-cases for doing this. I only think that this pattern is overused and we already have namespaces to separate our code. I’m all for separating src/main from src/test but src/shared goes way too far IMHO. shadow-cljs Solution The solution in shadow-cljs is pretty straightforward. shadow-cljs expects “entry” namespaces to be configured for all builds. Browser builds do this via :modules, node builds via :main. This is the default and you cannot build those targets without them Multiple :source-paths are supported but sources are only taken into account when referenced Multiple :source-paths are always global. You cannot configure a separate source path per build :source-paths are always on the classpath Although the implementation in shadow-cljs is entirely different it doesn’t provide anything that would not be possible with standard CLJS. I do believe that enforcing the config of “entry” namespaces however saves you from unknowingly including too much code in your builds. shadow-cljs just takes care of setting a better default so you don’t have to worry about it. You’ll see this pattern repeated in many of the shadow-cljs features.The Many Ways to use shadow-cljs2017-11-18T09:30:12+01:002017-11-18T09:30:12+01:00https://code.thheller.com/blog/shadow-cljs/2017/11/18/the-many-ways-to-use-shadow-cljs<p>This <a href="https://clojureverse.org/t/migrating-from-boot-back-to-leiningen/837">post on ClojureVerse</a> prompted a whole discussion about <code>boot</code> vs <code>lein</code> again and all I can think is: <em>Why does it matter?</em></p>
<p>Are we going to add the <code>tools.deps</code> <code>clojure</code> tool to this discussion next?</p>
<p>Why not just use Clojure? You can get very far with just doing that and as a bonus you can do everything from the REPL as well. I’m going to use <code>shadow-cljs</code> as an example here but I think it applies to a whole lot of other “tools” as well.</p>
<h2 id="build-it-as-a-library-first">Build it as a Library first</h2>
<p>First of foremost <code>shadow-cljs</code> is built as a normal Clojure Library. You can add the <code>thheller/shadow-cljs</code> artifact to <strong>any</strong> tool that is able to construct a Java Classpath for you and you can start using it.</p>
<pre><code>lein run -m shadow.cljs.devtools.cli compile app
boot run -m shadow.cljs.devtools.cli compile app # does this exist yet?
clojure -m shadow.cljs.devtools.cli compile app
mvn exec:java ... # not exactly sure how this works but it works
java -cp ... clojure.main -m shadow.cljs.devtools.cli compile app
shadow-cljs compile app
</code></pre>
<p>The <code>.cli</code> namespace is just a small wrapper to process the strings we get from the command line and turn them into proper Clojure data. In the REPL you can just call the <code>.api</code> namespace directly (properly <code>require</code>‘d of course)</p>
<pre><code class="language-clojure">(shadow.cljs.devtools.api/compile :app)
</code></pre>
<p>Want to <code>rsync</code> your <code>release</code> version to a remote server?</p>
<pre><code class="language-clojure">(ns app.build
(:require
[shadow.cljs.devtools.api :as shadow]
[some.lib :refer (rsync)]))
(defn release []
(shadow/release :app)
(rsync "some-dir/*" "user@some-server:/some-dir"))
</code></pre>
<p><code>lein run -m app.build/release</code> or <code>shadow-cljs clj-run app.build/release</code> or <code>(app.build/release)</code>. You get the idea.</p>
<h2 id="why-have-a-command-line-tool-then">Why have a command line tool then?</h2>
<h3 id="convenience">Convenience</h3>
<p>You have to type less. <code>shadow-cljs compile app</code> vs <code>lein run -m shadow.cljs.devtools.cli compile app</code>.</p>
<p>It can also check a lot of things without an actual JVM and can provide faster feedback in those cases.</p>
<h3 id="optimization">Optimization</h3>
<p>Starting a JVM+Clojure+Deps takes a while. This can be improved if the Clojure code is AOT compiled but it still won’t be very fast. Fortunately we don’t need to start a new JVM for everything, we can just re-use one we already started.</p>
<p>This is exactly what the <code>shadow-cljs</code> tool does. It will AOT compile the relevant code on the first startup so subsequent startups are faster. The <code>shadow-cljs server</code> starts the JVM in <code>server</code> mode. Every other command will then use that JVM instead of starting a new one.</p>
<p>This concept is not new. <a href="https://leiningen.org/grench.html"><code>grench</code></a> and <a href="https://github.com/ninjudd/drip"><code>drip</code></a> come to mind or any Clojure REPL.</p>
<p>Here are some numbers to compare the effect this optimization has.</p>
<p>The command used is:</p>
<pre><code>touch src/starter/browser.cljs &amp;&amp; time shadow-cljs compile app
</code></pre>
<table>
<thead>
<tr>
<th>AOT</th>
<th>Server</th>
<th>Cache</th>
<th>Time</th>
</tr>
</thead>
<tbody>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td>0m25.730s</td>
</tr>
<tr>
<td>✓</td>
<td> </td>
<td> </td>
<td>0m14.575s</td>
</tr>
<tr>
<td>✓</td>
<td> </td>
<td>✓</td>
<td>0m7.815s</td>
</tr>
<tr>
<td>-</td>
<td>✓</td>
<td> </td>
<td>0m7.706s</td>
</tr>
<tr>
<td>-</td>
<td>✓</td>
<td>✓</td>
<td><strong>0m0.673s</strong></td>
</tr>
</tbody>
</table>
<ul>
<li><code>touch</code> to force a recompile when using incremental compilation</li>
<li>Server means <code>shadow-cljs server</code> is running, no new JVM is started</li>
</ul>
<p>As you can see the difference is quite dramatic. Given that I get easily distracted when waiting for things this has a huge impact on my focus during the day.</p>
<p>The non-Server code could be optimized a bit since it always loads all development related code. <code>compile</code> for example doesn’t need all the REPL/live-reload related code but given the presence of <code>server</code> this never seemed necessary.</p>
<p>You’ll most likey use <code>watch</code> during actual development and all tools have an optimized experience for this but it still matters for other commands.</p>
<h2 id="but-what-about-boot">But what about boot?</h2>
<p><code>boot</code> tries to do a lot more than just providing a classpath. The problem with this is that it only works if you want to use the exact abstractions <code>boot</code> provides. As soon as you want to do something slightly different it just starts getting in the way. I don’t recommend using <code>shadow-cljs</code> with <code>boot</code> since it breaks all the caching <code>shadow-cljs</code> tries to do. <code>boot-cljs</code> has the same problem as far as I can tell. Restarting the <code>boot</code> process wipes all cache. You could make an argument here that this is a good thing since it prevents stale cache but that just treats the symptom instead of fixing the root cause (which I did in <code>shadow-cljs</code>).</p>
<p>If you separated the classpath/pod management from <code>boot</code> it would make for a very good library I think. I do like some ideas in <code>boot</code> but its too complected for me.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Write everything as a Clojure Library so it works in any tool and the REPL.</p>
<p>I simplified a great deal here. Things are a lot more complicated in the real world but I am convinced that we can get way better results if less code was written specific to one build tool and instead used Clojure as the common ground.</p>
<p>IMHO, YMMV.</p>This post on ClojureVerse prompted a whole discussion about boot vs lein again and all I can think is: Why does it matter?JS Dependencies: In Practice2017-11-10T09:30:12+01:002017-11-10T09:30:12+01:00https://code.thheller.com/blog/shadow-cljs/2017/11/10/js-dependencies-in-practice<p>In my previous posts about JS Dependencies (<a href="/blog/shadow-cljs/2017/09/15/js-dependencies-the-problem.html">The Problem</a>, <a href="/blog/shadow-cljs/2017/09/15/js-dependencies-going-forward.html">Going forward</a>) I explained why and how <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> handles JS Dependencies very differently than ClojureScript does by default. To recap:</p>
<ul>
<li>CLJSJS/<code>:foreign-libs</code> do not scale</li>
<li>Custom bundles are tedious to work with</li>
<li>Closure Compiler can’t yet reliably process a large portion of <code>npm</code> packages</li>
<li><code>shadow-cljs</code> implements a custom JS bundler but removed <code>:foreign-libs</code> support in the process</li>
</ul>
<h2 id="installing-js-dependencies">Installing JS Dependencies</h2>
<p>Almost every package on <code>npm</code> will explain how to install it. Those instructions now apply to <code>shadow-cljs</code> as well. So if a library tells you to run:</p>
<pre><code class="language-text">npm install the-thing
</code></pre>
<p>You do exactly that. Nothing more required. You may use <code>yarn</code> if preferred of course. Dependencies will be added to the <code>package.json</code> file and this will be used to manage them. If you don’t have a <code>package.json</code> yet run <code>npm init</code>.</p>
<p>You can use this <a href="https://github.com/shadow-cljs/quickstart-browser">Quick-Start template</a> to try everything described here.</p>
<h2 id="using-js-dependencies">Using JS Dependencies</h2>
<p>Most <code>npm</code> packages will also include some instructions on how to use the actual code. The “old” CommonJS style just has <code>require</code> calls which translates directly.</p>
<pre><code class="language-js">var react = require("react");
</code></pre>
<pre><code class="language-clojure">(ns my.app
(:require ["react" :as react]))
</code></pre>
<p>Whatever <code>"string"</code> parameter is used when calling <code>require</code> we transfer to the <code>ns</code> <code>:require</code> as-is. The <code>:as</code> alias is up to you. Once we have that we can use the code like any other CLJS namespace.</p>
<pre><code class="language-clojure">(react/createElement "div" nil "hello world")
</code></pre>
<p>This is different than what <code>:foreign-libs</code>/<code>CLJSJS</code> did before where you included the <code>thing</code> in the <code>ns</code> but then used <code>js/Thing</code> (or whatever global it exported) to use the code. Always use the <code>ns</code> form and whatever <code>:as</code> alias you provided. You may also use <code>:refer</code> and <code>:rename</code> if you wish.</p>
<p>Some packages just export a single function which you can call directly by using <code>(:require ["thing" :as thing])</code> and then <code>(thing)</code>.</p>
<p>More recently some packages started using ES6 <code>import</code> statements in their examples. Those also translate pretty much 1:1 with one slight difference related to <code>default</code> exports. Translating <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import">this</a> list of examples</p>
<pre><code class="language-js">import defaultExport from "module-name";
import * as name from "module-name";
import { export } from "module-name";
import { export as alias } from "module-name";
import { export1 , export2 } from "module-name";
import { export1 , export2 as alias2 , [...] } from "module-name";
import defaultExport, { export [ , [...] ] } from "module-name";
import defaultExport, * as name from "module-name";
import "module-name";
</code></pre>
<p>becomes (all inside <code>ns</code> of course)</p>
<pre><code class="language-clojure">(:require ["module-name" :default defaultExport])
(:require ["module-name" :as name])
(:require ["module-name" :refer (export)])
(:require ["module-name" :rename {export alias}])
(:require ["module-name" :refer (export1 export2)])
(:require ["module-name" :refer (export1) :rename {export2 alias2}])
(:require ["module-name" :refer (export) :default defaultExport])
(:require ["module-name" :as name :default defaultExport])
(:require ["module-name"])
</code></pre>
<p>The <code>:default</code> option is currently only available in <code>shadow-cljs</code>, you can vote <a href="https://dev.clojure.org/jira/browse/CLJS-2376">here</a> to hopefully make it standard. You can always use <code>:as alias</code> and then call <code>alias/default</code> if you prefer to stay compatible with standard CLJS in the meantime. IMHO that just gets a bit tedious for some packages.</p>
<h2 id="new-possibilities">New Possibilities</h2>
<p>Previously we were using bundled code, which may include code we don’t actually need. Some packages also describe ways that you can include only parts of the package leading to much less code included in your final build.</p>
<p><a href="https://github.com/bvaughn/react-virtualized"><code>react-virtualized</code></a> has one those examples:</p>
<pre><code class="language-javascript">// You can import any component you want as a named export from 'react-virtualized', eg
import { Column, Table } from 'react-virtualized'
// But if you only use a few react-virtualized components,
// And you're concerned about increasing your application's bundle size,
// You can directly import only the components you need, like so:
import AutoSizer from 'react-virtualized/dist/commonjs/AutoSizer'
import List from 'react-virtualized/dist/commonjs/List'
</code></pre>
<p>This we can also translate easily</p>
<pre><code class="language-clojure">;; all
(:require ["react-virtualized" :refer (Column Table)])
;; one by one
(:require ["react-virtualized/dist/commonjs/AutoSizer" :default virtual-auto-sizer])
(:require ["react-virtualized/dist/commonjs/List" :default virtual-list])
</code></pre>
<h2 id="resolving-js-dependencies">Resolving JS Dependencies</h2>
<p>By default <code>shadow-cljs</code> will resolve all <code>(:require ["thing" :as x])</code> requires following the <code>npm</code> convention. This means it will look at <code>&lt;project&gt;/node_modules/thing/...</code> and follow the code along there. To customize how this works <code>shadow-cljs</code> exposes a <code>:resolve</code> config option that lets you override how things are resolved.</p>
<h3 id="using-a-cdn">Using a CDN</h3>
<p>Say you already have <code>React</code> included in your page via a <code>CDN</code>. You could just start using <code>js/React</code> again but we stopped doing that for a good reason. Instead you continue to use <code>(:require ["react" :as react])</code> but configure how <code>"react"</code> resolves like this in your <code>shadow-cljs.edn</code> config for your build</p>
<pre><code class="language-clojure">{:builds
{:app
{:target :browser
...
:js-options
{:resolve {"react" {:target :global
:global "React"}}}}
:server
{:target :node-script
...}}}
</code></pre>
<p>The <code>:app</code> build will now use the global <code>React</code> instance while the <code>:server</code> build continues using the <code>"react"</code> npm package. No need to fiddle with the code to make this work.</p>
<h3 id="redirecting-require">Redirecting “require”</h3>
<p>Some packages provide multiple “dist” files and sometimes the default one described doesn’t quite work in <code>shadow-cljs</code>. One good example for this is <code>"d3"</code>. Their default <code>"main"</code> points to <code>"build/d3.node.js"</code> but that is not what we want when working with the browser. Their ES6 code runs into a bug in the <a href="https://github.com/google/closure-compiler/issues/2005">Closure Compiler</a>, so we can’t use that. Instead we just redirect the require to some other require.</p>
<pre><code class="language-clojure">{:resolve {"d3" {:target :npm
:require "d3/build/d3.js"}}}
</code></pre>
<p>You could just <code>(:require ["d3/build/d3.js" :as d3])</code> directly as well if you only care about the Browser.</p>
<h3 id="using-local-files">Using local Files</h3>
<p>You may also use <code>:resolve</code> to directly map to files in your project.</p>
<pre><code class="language-clojure">{:resolve {"my-thing" {:target :file
:file "path/to/file.js"}}}
</code></pre>
<p>The <code>:file</code> is always relative to the project directory. The included file may use <code>require</code> or <code>import/export</code> and those will be followed and included properly as well.</p>
<h1 id="migrating-cljsjs">Migrating cljsjs.*</h1>
<p>Many CLJS libraries are still using <code>CLJSJS</code> packages and they would break with <code>shadow-cljs</code> since that no longer supports <code>:foreign-libs</code>. I have a clear migration path for this and it just requires one shim file that maps the <code>cljsjs.thing</code> backs to its original <code>npm</code> package and exposes the expected global variable.</p>
<p>For <code>react</code> this requires a file like <a href="https://github.com/thheller/shadow-cljsjs/blob/master/src/main/cljsjs/react.cljs"><code>src/cljsjs/react.cljs</code></a>:</p>
<pre><code class="language-clojure">(ns cljsjs.react
(:require ["react" :as react]
["create-react-class" :as crc]))
(js/goog.object.set react "createClass" crc)
(js/goog.exportSymbol "React" react)
</code></pre>
<p>Since this would be tedious for everyone to do manually I created the <a href="https://github.com/thheller/shadow-cljsjs"><code>shadow-cljsjs</code></a> library which provides just that. It does not include every package but I’ll keep adding them and contributions are very welcome as well.</p>
<p>It only provides the shim files though, you’ll still need to <code>npm install</code> the actual packages yourself.</p>
<h1 id="what-to-do-when-things-dont-work">What to do when things don’t work?</h1>
<p>Since the JS world is still evolving rapidly and not everyone is using the same way to write and distribute code there are some things <code>shadow-cljs</code> cannot work around automatically and either requires custom <code>:resolve</code> configs. There may also be bugs, this is all very new after all.</p>
<p>Please <a href="https://github.com/thheller/shadow-cljs/issues">report</a> any packages that don’t work as expected. <a href="https://clojurians.slack.com/messages/C6N245JGG/">#shadow-cljs</a> is also a good place to find me.</p>
<p>Discuss on <a href="https://clojureverse.org/t/js-dependencies-in-practice/691">:clojureverse</a>.</p>In my previous posts about JS Dependencies (The Problem, Going forward) I explained why and how shadow-cljs handles JS Dependencies very differently than ClojureScript does by default. To recap:Improved Externs Inference2017-11-06T19:30:12+01:002017-11-06T19:30:12+01:00https://code.thheller.com/blog/shadow-cljs/2017/11/06/improved-externs-inference<p>In my <a href="/blog/shadow-cljs/2017/10/15/externs-the-bane-of-every-release-build.html">previous post</a> I talked about Externs and the options <code>shadow-cljs</code> provides to help deal with them. I sort of skipped over the whole <a href="https://clojurescript.org/guides/externs">Externs Inference</a> subject since I didn’t feel it was “ready” yet. It worked reasonably well but also generated a whole bunch of warnings that weren’t actually issues (eg. any <code>deftype</code>, <code>defrecord</code>, …). It also was way more ambitious in trying to actually generate Typed Externs.</p>
<h2 id="typed-or-untyped">Typed or Untyped?</h2>
<p>I took the work Maria Geller and David Nolen had done as a starting point but decided against using it since I think we can make this process a whole lot easier without sacrificing anything.</p>
<p>The Closure Compiler acts as a type checker and the Closure Library is fully typed, so naturally every carefully crafted <a href="https://github.com/google/closure-compiler/tree/master/externs">externs file</a> you find in the wild is also fully typed. This is great if all your code is typed and the externs code just flows through. However ClojureScript is <strong>not</strong> typed, therefore you gain nothing by using typed externs.</p>
<p>Whenever the Closure Compiler cannot determine the type of something it will do the “safe” thing and neither remove or rename a property if it is defined <strong>anywhere</strong> in the externs, regardless of type. For CLJS code this will be what happens 99% of the time.</p>
<h2 id="untyped-ftw">Untyped FTW!</h2>
<p>A lot of <a href="https://github.com/cljsjs/packages">CLJSJS</a> packages already cheated and used generated externs that are sort of half-in/half-out. The trouble with that is that it may list too many properties and may actually keep more code alive than required. In the very least it will stop the Closure Compiler from renaming a few things that it might otherwise rename. We only need externs for the code we actually call from CLJS, not everything the JS code uses.</p>
<p>Luckily for us the impact of too many externs is rather minuscule and something we could tweak later if we wanted. It is a much better experience to not run into the dreaded <code>ln.S is not a function</code> errors and the effect on code size is surprisingly small. 3KB gain with <a href="/blog/shadow-cljs/2017/10/15/externs-the-bane-of-every-release-build.html">generated externs</a> over manually written externs for a 600KB <code>.js</code> file. I think that is very acceptable already but we can get that do almost zero difference by using <code>:infer-externs</code> to actually only generate the things we need.</p>
<h1 id="how-does-it-work">How does it work?</h1>
<p>The official <a href="https://clojurescript.org/guides/externs">Externs Inference Guide</a> has a good overview and pretty much all of it still applies. However as explained above we do not need to worry about the type.</p>
<p>As the example explains this code</p>
<pre><code class="language-clojure">(set! *warn-on-infer* true)
(defn wrap-baz [x]
(.baz x))
</code></pre>
<p>would produce this warning</p>
<pre><code class="language-text">------ WARNING #1 --------------------------------------------------------------
File: ~/project/src/demo/thing.cljs:23:3
--------------------------------------------------------------------------------
19 |
20 | (set! *warn-on-infer* true)
21 |
22 | (defn wrap-baz [x]
23 | (.baz x))
---------^----------------------------------------------------------------------
Cannot infer target type in expression (. x baz)
--------------------------------------------------------------------------------
</code></pre>
<p>The guide tells you to add this tag metadata</p>
<pre><code class="language-clojure">(defn wrap-baz [^js/Foo.Bar x]
(.baz x))
</code></pre>
<p>which we can simplify down to just <code>^js</code>.</p>
<pre><code class="language-clojure">(defn wrap-baz [^js x]
(.baz x))
</code></pre>
<p>By using the <code>^js</code> tag metadata we are just telling the compiler that this is a native JS value and we are going to need externs for it. In <code>shadow-cljs</code> the <code>/Foo.Bar</code> type info is optional. You can still use it if you like but <code>shadow-cljs</code> won’t bother you if you don’t.</p>
<p>I also did a whole sweep through the compiler to get rid of some warnings we don’t actually need externs for. It will now also account for all properties that are actually defined in externs and won’t warn about those.</p>
<p>There are still a few cases left where the compiler might produce a warning when in fact no externs are needed. This will happen if you do native interop on either CLJS or Closure JS objects. We do not need externs here and you can get rid of the warning by using either the <code>^clj</code> tag for CLJS or the <code>^goog</code> tag. If you know the actual type of the thing you can also use that instead.</p>
<h2 id="how-to-use-it">How to use it</h2>
<p>By default you still need to <code>(set! *warn-on-infer* true)</code> to actually get warnings about Externs Inference for each individual file.</p>
<p>Since that is a bit tedious to do for every file in your project I introduced a new <code>:compiler-options {:infer-externs :auto}</code> setting (in addition to just <code>true|false</code>). This will turn on the Extern Inference warnings for <strong>your</strong> files, ie. not files in <code>.jar</code>s.</p>
<p>You can try <code>:all</code> but that is crazy talk. There are still about 200 warnings in <code>cljs.core</code> but we don’t need externs for any of them, so yeah do not use <code>:all</code>. Only if you really want to see a whole lot of warnings, I got to over 1200 before all the tweaks. ;)</p>
<p>Also make sure you are using at least <code>shadow-cljs@2.0.61</code>.</p>
<h2 id="disclaimer">Disclaimer</h2>
<p>This is not official work. It is an experiment. It is based on my experience with the Closure Compiler and mostly based on some more or less educated guesses. The whole subject is not documented very well and I mostly went through a whole bunch of Java Sources to figure it out. It works well for me but YMMV. It definitely could use a few more real world examples to test this on.</p>
<p>If everything works out the way I hope we can make this official and part of the CLJS itself. Until then it is <code>shadow-cljs</code> only.</p>
<p>I hang out in the <a href="https://clojurians.slack.com/messages/C6N245JGG/">Clojurians</a> Slack if you have questions. Feel free to open a <a href="https://github.com/thheller/shadow-cljs/issues">Github Issue</a> if you run into trouble as well.</p>In my previous post I talked about Externs and the options shadow-cljs provides to help deal with them. I sort of skipped over the whole Externs Inference subject since I didn’t feel it was “ready” yet. It worked reasonably well but also generated a whole bunch of warnings that weren’t actually issues (eg. any deftype, defrecord, …). It also was way more ambitious in trying to actually generate Typed Externs.Externs: The Bane of every Release Build2017-10-15T14:30:12+02:002017-10-15T14:30:12+02:00https://code.thheller.com/blog/shadow-cljs/2017/10/15/externs-the-bane-of-every-release-build<p>Ever run into errors like these?</p>
<blockquote>
<p>Cannot read property ‘c’ of null</p>
</blockquote>
<blockquote>
<p>ln.S is not a function</p>
</blockquote>
<p>Your app runs fine in development and but when trying <code>:advanced</code> it blows up?</p>
<p>Externs or rather the lack thereof are often to blame here and possibly the most frequent issue coming up in the <a href="https://clojurians.slack.com/messages/C03S1L9DN/">#clojurescript</a> Slack (and elsewhere).</p>
<h2 id="externs">Externs?</h2>
<p>Externs if you don’t know what they are can be quite challenging and frustrating to learn. ClojureScript is optimized by the Closure Compiler which attempts to optimize your <strong>whole</strong> program. Sometimes it doesn’t have access to the whole program, eg. when using a foreign JS library, so we need to tell the Compiler about those. This is what externs do. They inform the Compiler about the structure of some external JavaScript it does not see. The externs protect the given property names so Closure does not attempt to shorten (or remove) them.</p>
<p>On top of that the Closure Compiler acts as a type checker and the Closure Library is fully typed. The type information enables some more <code>:advanced</code> optimizations that are not possible without. In theory this means that your <code>:externs</code> also need to be typed which is not so great if you are otherwise writing untyped code. ClojureScript is untyped and so is most code available on <code>npm</code>.</p>
<p>Fortunately whenever the Closure Compiler finds a property but cannot determine the type of the object it is on it will be conservative and check if the property it defined anywhere in the externs. If that is the case the compiler will not rename the property and all is well.</p>
<h2 id="writing-externs-manually">Writing Externs manually</h2>
<p>Unfortunately we still need to tell the compiler about those properties which isn’t always easy. <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> has some built-in utilities to ease this process.</p>
<p>We can create a release build with the <code>--pseudo-names</code> option which will still rename everything but in a way that is still human-readable.</p>
<pre><code class="language-txt">shadow-cljs release app --pseudo-names
# turns
Uncaught TypeError: ln.S is not a function
# into
Uncaught TypeError: something.$foo$ is not a function
</code></pre>
<p>If you have done this before this tells you that <code>foo</code> needs to be defined in externs. This process can be quite tedious and you need to recompile your app with every added externs and not all things happen directly on startup. Some errors only happen after clicking a thing after clicking three other things.</p>
<p>I added <code>shadow-cljs check app</code> to remove some of this burden as it runs all the code through the Closure Compiler type checker so you’d get warnings about externs during compilation. This could look like</p>
<pre><code class="language-txt">------ WARNING #1 --------------------------------------------------------------
File: ~/.m2/repository/org/clojure/clojurescript/1.9.946/clojurescript-1.9.946.jar!/clojure/string.cljs:33:35
--------------------------------------------------------------------------------
29 | (let [r (js/RegExp. (.-source re)
30 | (cond-&gt; "g"
31 | (.-ignoreCase re) (str "i")
32 | (.-multiline re) (str "m")
33 | (.-unicode re) (str "u")))]
-----------------------------------------^--------------------------------------
Property unicode never defined on re
--------------------------------------------------------------------------------
34 | (.replace s r replacement)))
35 |
36 | (defn- replace-with
37 | [f]
38 | (fn [&amp; args]
--------------------------------------------------------------------------------
</code></pre>
<p>Unfortunately this is not totally accurate either and often complained about non-issues given that it works off a mostly untyped codebase.</p>
<p>Manually writing externs is not fun. Period.</p>
<h2 id="cljsjs">CLJSJS</h2>
<p><a href="http://cljsjs.github.io/">CLJSJS</a> is a community effort to pre-bundle existing JS libraries and making them usable by the ClojureScript compiler. As a bonus it is possible to bundle externs as well, which means someone only needs to write externs once and everyone using this library benefits from them.</p>
<p>This is way better but what if the JS code you want to use is not available via CLJSJS?</p>
<h2 id="generating-externs">Generating Externs</h2>
<p>What if we generated externs?</p>
<p>David Nolen implemented the <a href="https://clojurescript.org/guides/externs">Externs Inference</a> for the ClojureScript compiler. This works pretty well but sometimes relies on the user annotating the code in a couple of places which is not something you’d do (well, me at least) naturally. So you typically still run into the <code>ln.S is not a function</code> issue, then enable “pseudo names” and then annotate your code. In a way thats still writing externs manually.</p>
<h2 id="going-for-100-automation">Going for 100% automation</h2>
<p>As described in my <a href="/blog/shadow-cljs/2017/09/15/js-dependencies-going-forward.html">previous post</a> I implemented a custom JS bundler for <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a>. This means that it processes all the JS code from <code>npm</code> and bundles it in a way we can use from ClojureScript. To do this it also runs the code through <code>:simple</code> optimizations which is similar to what <code>UglifyJS</code> in the JS world would do.</p>
<p>Since it is compiling the JS anyways we can collect all the names used in the external JS and use those as externs when running the rest of the build through <code>:advanced</code> compilation.</p>
<p>Combining this with some of the information gathered by the externs inference code we get pretty reliable results and typically do not need to write manual externs.</p>
<h3 id="well-almost-100">Well, almost 100%</h3>
<p>The caveat is that this only works when <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> is processing ALL of the JS. It cannot do this when emitting code for <code>node.js</code> since that just maps directly to <code>require</code> and lets <code>node</code> deal with resolving that. There are also perfectly reasonable scenarios in <code>:browser</code> builds where code won’t be processed by <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> (eg. including some JS via a <code>&lt;script&gt;</code> directly from a CDN).</p>
<h3 id="covering-the-last-few-percent">Covering the last few percent</h3>
<p>Sometimes you are just going to need to write some manual externs. To at least ease this process <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> now supports writing simplified externs files which look something like</p>
<pre><code class="language-txt"># comments start with #
# properties are just listed one per line
propertyA
propertyB
# globals are prefixed by global:
global:someGlobal
</code></pre>
<p>You only need to list all the properties you want to preserve and any globals you might use. No need to do any type annotations.</p>
<p>You put these into a file at <code>&lt;project-root&gt;/externs/your-build-id.txt</code> and <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> will pick them up automatically. Of course the usual <code>:compiler-options {:externs ["path/to/externs.js"]}</code> still works and can be combined with all of the above.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope that the new externs generation features of <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> can reduce some of the pain caused by externs. I have done a few tests where I was able to remove all of the externs I had to add manually before. Getting to truly 100% is not possible due to the dynamic nature some of the JS on <code>npm</code>.</p>
<p>I do think that all the tools <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> provides do make this whole process a whole lot less painful. I’m curious to gather more data on this and want to hear about your experiences when you try it. Everything mentioned here is available since <code>shadow-cljs@2.0.18</code>, it is enabled by default so no configuration required.</p>Ever run into errors like these?Bootstrapped ClojureScript Support2017-10-14T16:30:12+02:002017-10-14T16:30:12+02:00https://code.thheller.com/blog/shadow-cljs/2017/10/14/bootstrap-support<p><code>shadow-cljs</code> (since about <code>2.0.15</code>) supports compiling builds that make use of the self-hosted ClojureScript Compiler. Previously this was not supported since the build needs to be modified in a few places to ensure that all the required files are available. <code>shadow-cljs</code> itself continues to use the JVM compiler.</p>
<p>This feature is split into two parts. First you need a “host” build which will be your “app” (currently limited to <code>:browser</code> builds). This build will host the compiler and then use the support files generated by the second <code>:bootstrap</code> build. These support files include the CLJS analyzer cache, macro JS files, <code>.cljs</code> files, etc.</p>
<p>The <code>:bootstrap</code> build itself will generate an “index” with useful information for the compiler. When the “host” build starts up it calls the provided <code>shadow.cljs.bootstrap.browser/init</code> function to load the index and fetch all resources necessary to start using the compiler (which usually involves the analyzer data for <code>cljs.core</code> and the <code>cljs.core$macros</code> JS file).</p>
<p>Once <code>init</code> completes the compiler can be used. The provided <code>shadow.cljs.bootstrap.browser/load</code> function takes care of properly loading dependencies. The CLJS analyzer will call this function whenever a dependency is required and it will load the analyzer data, macro and JS files. Only namespaces pre-compiled by the <code>:bootstrap</code> build will be available.</p>
<p>The “host” can look something like this:</p>
<pre><code class="language-clojure">(ns demo.selfhost.simple
(:require [cljs.js :as cljs]
[cljs.env :as env]
[shadow.cljs.bootstrap.browser :as boot]))
(defn print-result [{:keys [error value] :as result}]
(js/console.log "result" result)
(set! (.-innerHTML (js/document.getElementById "dump")) value))
(def code
"
(ns simpleexample.core
(:require [clojure.string :as str]
[reagent.core :as r]))
(defonce timer (r/atom (js/Date.)))
(defonce time-color (r/atom \"#f34\"))
(defonce time-updater (js/setInterval
#(reset! timer (js/Date.)) 1000))
(defn greeting [message]
[:h1 message])
(defn clock []
(let [time-str (-&gt; @timer .toTimeString (str/split \" \") first)]
[:div.example-clock
{:style {:color @time-color}}
time-str]))
(defn color-input []
[:div.color-input
\"Time color: \"
[:input {:type \"text\"
:value @time-color
:on-change #(reset! time-color (-&gt; % .-target .-value))}]])
(defn simple-example []
[:div
[greeting \"Hello world, it is now\"]
[clock]
[color-input]])
(r/render [simple-example] (js/document.getElementById \"app\"))" )
(defonce compile-state-ref (env/default-compiler-env))
(defn compile-it []
(cljs/eval-str
compile-state-ref
code
"[test]"
{:eval cljs/js-eval
:load (partial boot/load compile-state-ref)}
print-result))
(defn start []
(boot/init compile-state-ref
{:path "/bootstrap"}
compile-it))
(defn stop [])
</code></pre>
<p>The <code>shadow-cljs.edn</code> config</p>
<pre><code class="language-clojure">{:dependencies
[[reagent "0.8.0-alpha1" :exclusions [cljsjs/create-react-class]]
:source-paths
["src"]
:builds
{:bootstrap-host
{:target :browser
:output-dir "out/demo-selfhost/public/simple/js"
:asset-path "/simple/js"
:compiler-options
{:optimizations :simple}
:modules
{:base
{:entries [demo.selfhost.simple]}}
:devtools
{:http-root "out/demo-selfhost/public"
:http-port 8700
:before-load demo.selfhost.simple/stop
:after-load demo.selfhost.simple/start}}
:bootstrap-support
{:target :bootstrap
:output-dir "out/demo-selfhost/public/bootstrap"
:exclude #{cljs.js}
:entries [cljs.js demo.macro reagent.core]
:macros []}}}
</code></pre>
<p>The config option for the <code>:bootstrap</code> build are</p>
<ul>
<li><code>:entries</code> a sequence of namespaces you want to have available for the self-hosted compiler</li>
<li><code>:exclude</code> for <strong>macro</strong> namespaces that are not self-host compatible as they would otherwise break the build. This would include things like <code>cljs.core.async.macros</code>, <code>cljs.js</code>, etc.</li>
<li><code>:macros</code> will usually be optional since all macros used by the <code>:entries</code> will already be included.</li>
</ul>
<p>Everything is written to <code>:output-dir</code>. The path where those files are available must be passed to the <code>boot/init</code> function.</p>
<p><a href="https://twitter.com/mhuebert">@mhuebert</a> created a <a href="https://github.com/mhuebert/shadow-bootstrap-example">standalone example</a> and the <code>shadow-cljs</code> repo itself contains the example above.</p>
<p>You can try it by running</p>
<pre><code class="language-bash">npm install -g shadow-cljs
git clone https://github.com/thheller/shadow-cljs.git
cd shadow-cljs
shadow-cljs watch bootstrap-host bootstrap-support
open http://localhost:8700
</code></pre>
<p>Usually <code>shadow-cljs</code> does not require <code>lein</code> but it is required in this case since I’m using <code>lein</code> to build the <code>shadow-cljs</code> project. The standalone example is probably better suited for testing.</p>
<p>I didn’t cover a whole lot of technical details. Come by <a href="https://clojurians.slack.com/messages/C6N245JGG/"><code>#shadow-cljs</code></a> if you have questions. Please report issues <a href="https://github.com/thheller/shadow-cljs/issues">here</a>.</p>shadow-cljs (since about 2.0.15) supports compiling builds that make use of the self-hosted ClojureScript Compiler. Previously this was not supported since the build needs to be modified in a few places to ensure that all the required files are available. shadow-cljs itself continues to use the JVM compiler.JS dependencies: Going forward2017-09-15T19:21:12+02:002017-09-15T19:21:12+02:00https://code.thheller.com/blog/shadow-cljs/2017/09/15/js-dependencies-going-forward<p>This post is split into 2 parts aiming to answer the question:</p>
<blockquote>
<p>How do I use library “xyz”?</p>
</blockquote>
<p>The <a href="/blog/shadow-cljs/2017/09/15/js-dependencies-the-problem.html">first post</a> defined the Problem and described the solutions people use today, this post will go over how <code>shadow-cljs</code> will handle the issue going forward.</p>
<h2 id="the-goal">The Goal</h2>
<p>I want to automate using JS dependencies as much as possible while eliminating the issues I described in the first post. It should require as little configuration as possible but still allow the most common scenarios that people commonly use in JS projects (eg. using <code>jQuery</code> or <code>react</code> from a CDN). Interop should be seamless.</p>
<h2 id="using-js-dependencies">Using JS dependencies</h2>
<p>CLJSJS popularized the use of <code>:foreign-libs</code> which introduced pseudo-namespaces that had no functionality beyond telling the compiler to include that certain lib in the build. You would <code>(:require [cljsjs.react])</code> in your code but then access it using <code>js/React</code>. This is problematic for various reason and would require changing the code when importing the code via Closure.</p>
<p>Going forward the recommended solution is:</p>
<ul>
<li>CLJS/Closure you <code>:require</code> by using a <code>symbol</code> (eg. <code>(:require [clojure.string :as str])</code>)</li>
<li>JS dependencies are required using a <code>"string"</code> which exactly matches the string you’d use in JS via <code>require</code> or <code>import</code>.</li>
</ul>
<p>The JS version</p>
<pre><code class="language-js">import { createElement } as react from "react";
import { render } as rdom from "react-dom";
// or
const react = require("react");
...
</code></pre>
<p>then becomes</p>
<pre><code class="language-clojure">(ns your.app
(:require [clojure.string :as str]
["react" :as react :refer (createElement)]
["react-dom" :as rdom :refer (render)]))
</code></pre>
<p>You could always use <code>(def react (js/require "react"))</code> in CLJS (which most <code>node.js</code> users do) but the problem here is that the compiler does not understand this information. It does not collect <code>js/require</code> calls and therefore is not aware of any JS dependencies it might need to deal with.</p>
<p>By moving it into the <code>ns</code> form it becomes static metadata (just like ES6 <code>import</code>) so tools can easily work with this data.</p>
<h4 id="a-small-rant-dont-use-symbols-for-js">A small rant: Don’t use symbols for JS</h4>
<p>Other CLJS contributors do not share my opinion that you should use strings for JS requires. They allow using <code>(:require [react])</code> as a short-hand which is a magical symbol that only becomes available when the <code>./node_modules/react</code> package exists (or a <code>:foreign-lib</code> provides it).</p>
<p>IMHO it is much cleaner to make it immediately obvious whether you are including a CLJS/Closure namespace by using a <code>symbol</code> or a JS dependency by using a <code>"string"</code>. Not only does this make it easier for tools to do their work it also deals with all ambiguities you might encounter when using npm packages and trying to convert their name to a valid symbol (eg. <code>-</code> vs <code>_</code>, <code>"@scoped/packages"</code>, <code>"react-dom/server"</code>).</p>
<p><strong>Just use a <code>"string"</code> to include JS.</strong></p>
<p><code>shadow-cljs</code> will support symbols for compatibility but really doesn’t like it.</p>
<h2 id="choosing-a-provider">Choosing a Provider</h2>
<p><code>shadow-cljs</code> introduces a new <code>:js-provider</code> build config option that controls which mechanism will be used to provide the JS support. There are currently 3 options to choose from:</p>
<ul>
<li><code>:require</code> will map directly to JS <code>require</code> calls, which works natively on <code>node</code> or when using other build tools to fill them in for us (ie. <code>webpack</code>, <code>react-native</code> packager, etc)</li>
<li><code>:shadow</code> is the default for <code>:browser</code> builds and will package the <code>npm</code> code into a format we can consume in the browser.</li>
<li><code>:closure</code> will attempt to import all JS dependencies using the Closure Compiler and should only be considered for <code>:browser</code> builds as well.</li>
</ul>
<h2 id="shadow-vs-closure">:shadow vs :closure</h2>
<p>Both options attempt to import code from <code>npm</code> and rewrite it in a browser friendly way.</p>
<p><code>:closure</code> is much more aggressive and will rewrite basically the entire file. It is the better option when it works since it will work reliably with <code>:advanced</code> optimizations. In theory no <code>:externs</code> would be required and the produced code will fully benefit from everything the Closure Compiler has to offer (dead-code-removal, code motion, etc). The downside is that it does not yet understand every form of <code>npm</code> package out there. A lot of work is being done to improve this situation but it might not yet work for some <code>npm</code> packages you want to use.</p>
<p><code>:shadow</code> is the default until that changes. It basically only rewrites <code>require</code> calls since the browser does not natively understand those. It initially used <code>browserify</code> to bundle JS code for us but that had significant issues and did not work reliably enough. Going forward <code>shadow-cljs</code> uses its own JS bundler implementation with the goal of supporting every package on <code>npm</code>.</p>
<p><code>:shadow</code> already has pretty good support for most <code>npm</code> packages I tested. Since there are so many different ways <code>npm</code> packages are built there might be some it does not yet support. Please let me know if you run into issues.</p>
<p>Using <code>:closure</code> is the goal and ideal path but it might take some time to get working properly. Definitely try and see if it works for your project.</p>
<p>Either way there are some issues we need to deal with going forward.</p>
<h3 id="externs">Externs</h3>
<p>The downside of <code>:shadow</code> is that <code>:externs</code> are required since the JS code will not be run through <code>:advanced</code> compilation. It will minify the code for <code>release</code> builds which should still produce acceptable bundle sizes similar to what <code>webpack</code> and others would achieve but far from <code>:closure</code>.</p>
<p>Writing <code>:externs</code> is annoying but <code>shadow-cljs check</code> might be able to help.</p>
<p>We should be able to create a repository of <code>:externs</code> so the compiler can automatically apply the matching externs depending on the packages you use. Similar to the way <code>CLJSJS</code> automated most <code>:externs</code>.</p>
<p>Some <code>:externs</code> will still be required even for <code>:closure</code> since code on <code>npm</code> does not follow the strict coding style Closure expects. Some more “dynamic” styles are not properly detected when compiling and <code>:externs</code> sort of work around those cases (eg. <code>react</code> still needs externs with <code>:closure</code>).</p>
<h3 id="breaking-change-removing-foreign-libs-support">BREAKING CHANGE: Removing :foreign-libs support</h3>
<p>Popular libraries like <code>reagent</code> just include <code>cljsjs.react</code> but then use <code>js/React</code> to call the actual code. That is a problem since the compiler doesn’t (and shouldn’t) know that some foreign JS provided that global. <code>reagent</code> has a newer alpha version that removed the <code>cljsjs.react</code> require and now uses <code>react</code> (as a symbol, not a string. <code>:(</code>) but pretty much all libraries need to be updated to do this.</p>
<p>Pre-bundled <code>:foreign-libs</code> (ie. CLJSJS) themselves are a problem due to the “scaling” problem mentioned in the first post. The systems simply do not mix well and frequently conflict with each other. One library might use <code>cljsjs.react</code> while another uses the newer <code>"react</code>”. This would lead to two versions of <code>React</code> being included in your page.</p>
<p><strong><code>shadow-cljs</code> will not support :foreign-libs going forward</strong> (which includes CLJSJS).</p>
<p>This might make it impossible to build your project with <code>shadow-cljs</code> without changing some code.</p>
<p>Since most CLJSJS packages should have <code>npm</code> packages as well you can work around a few issues by manually creating the <code>cljsjs</code> namespaces. Take <code>react</code> for example, usually you’d use <code>cljsjs.react</code> and then the global <code>js/React</code>. To make that available you can create that namespace and manually add the global.</p>
<pre><code class="language-clojure">(ns cljsjs.react
(:require ["react" :as react]))
(js/goog.exportSymbol "React" react)
</code></pre>
<p>This should only be done if you are using a library that uses a <code>cljsjs</code> namespace that has not been updated. Your own code should be updated to use string requires.</p>
<p>I’m happy to provide an interim solution to make the transition easier if someone has a good idea how to do so.</p>
<h3 id="npm-deps">:npm-deps</h3>
<p>CLJS core supports the <code>:npm-deps</code> compiler option to control how things are compiled. In addition it also attempts to manage JS dependencies by trying to install packages for you.</p>
<p>I do not think that <code>:npm-deps</code> is adequate to cover this topic. <code>shadow-cljs</code> does not support this now and won’t manage JS dependencies for you. You must manually install packages using <code>npm</code> or <code>yarn</code> and manage them using the usual <code>package.json</code>.</p>
<h2 id="work-to-be-done">Work to be done</h2>
<p>All work was focused on <code>npm</code> packages and using local <code>.js</code> files is not as convenient as I’d like it to be. It will be properly supported at some point I just didn’t get around to it yet.</p>
<h1 id="conclusion">Conclusion</h1>
<p>I decided to do an entirely different implementation than what is currently available in CLJS core. I do think that it is a tooling concern anyways and necessary to be able to deal with all the complex build scenarios users might have.</p>
<p>Due to the problems described in the first post I never used many JS dependencies in my projects. Going forward I feel much more confident that I can safely use them and not worry about all the problems (well, except <code>:externs</code> I guess).</p>
<p>I do think that using JS dependencies is easier and more reliable this way. There are some adjustments you might need to make to your code migrating off <code>:foreign-libs</code> but I do think it is worth it in the end. You gain access to the whole <code>npm</code> ecosystem and do not rely on third-party re-packaged CLJSJS packages.</p>
<h1 id="feedback-wanted">Feedback wanted!</h1>
<p>I need testers to make it as reliable as possible. Just <code>yarn add shadow-cljs</code> and get started.</p>
<p>A simple example can be found <a href="https://github.com/thheller/shadow-cljs-examples/tree/master/browser">here</a>.</p>
<p>Please report a <a href="https://github.com/thheller/shadow-cljs/issues">Github issue</a> in case something doesn’t work as expected. You can find me on <a href="https://www.reddit.com/r/Clojure/comments/72wf86/shadowcljs_js_dependencies_going_forward/">reddit</a>, <a href="https://twitter.com/thheller/status/913194903290761216">Twitter</a> or <a href="https://clojurians.slack.com/messages/C6N245JGG/">@thheller</a> in the <a href="https://clojurians.slack.com/">Clojurians Slack</a> if you have questions.</p>This post is split into 2 parts aiming to answer the question:JS dependencies: The Problem2017-09-15T15:21:12+02:002017-09-15T15:21:12+02:00https://code.thheller.com/blog/shadow-cljs/2017/09/15/js-dependencies-the-problem<p>This post is split into 2 parts aiming to answer the question:</p>
<blockquote>
<p>How do I use library “xyz”?</p>
</blockquote>
<p>Using JS dependencies is by far the most common problem for CLJS users. Many answers exist and the topic got a whole lot more complicated <a href="https://clojurescript.org/news/2017-07-12-clojurescript-is-not-an-island-integrating-node-modules">recently</a>. No solution is perfect and pretty much all of them require some sort of manual tuning. In the second post I’ll go over how I have chosen to tackle this problem within <code>shadow-cljs</code>.</p>
<p>This post will focus on defining the problem and describing the current solutions that people use today.</p>
<h2 id="the-problem">The Problem</h2>
<p>To run code in the browser we want to optimize the code as much as possible. This means that we want to reduce the number of requests the browser has to do to a minimum and cut all unnecessary code out of our build. For CLJS this is done by the Closure Compiler which yields very good results but requires that code is written in a certain way which pretty much no one likes writing by hand.</p>
<p>Luckily the CLJS compiler can do that for us and we can build on top of the extensive Closure Library. Beyond that we had to resort to all kind of trickery described below to use “other” JS dependencies that are not part of this world and are written for the more generic <code>npm</code> world.</p>
<h2 id="cljsjs">CLJSJS</h2>
<p><a href="https://github.com/cljsjs/packages">CLJSJS</a> is a community effort to pre-bundle most <code>npm</code> packages so they can be consumed by the CLJS compiler. This is by far the easiest option but has certain “scaling” issues. Not all <code>npm</code> packages are available and sometimes packages contain duplicate code since they were bundle separately leading to bloated file sizes or even very hard to track bugs since you may end up with multiple versions of the same library loaded on the page.</p>
<p>If you just want <code>"react"</code> this solution works perfectly fine but it gets hairy quick with every additional dependency. One example would be the <a href="https://github.com/cljsjs/packages/tree/master/material-ui">material-ui</a> package which ships with its own version of <code>"react"</code>, meaning that you have to manually exclude all other versions other libraries might be using.</p>
<p>Since CLJSJS packages typically bundle <code>:externs</code> the user often doesn’t have to worry about providing these. BUT the <code>:externs</code> are often generated and overly generic which might actually hurt your file size in bigger projects.</p>
<h2 id="keep-it-separate">“Keep it separate”</h2>
<p>The compromise is to manually collect all JS dependecies and bundle them together using tools like <code>webpack</code> and then just including the file separately (or via the <code>:foreign-libs</code> compiler option). You do need <code>:externs</code> for this and code-splitting gets extra complicated but this solution is very reliable when using many different <code>npm</code> packages that you just bundle up into one big blob.</p>
<p>The problem here is you’ll access the bundled JS code via certain global variables and the compiler can’t do anything to help. You also need to setup the other build setup which usually is not very easy.</p>
<h2 id="closure-compiler">Closure Compiler</h2>
<p>Until recently the Closure Compiler did not understand CommonJS/ES6 very well. A lof of work is being done to change that and the goal is to transpile ES6/CommonJS code into a format that Closure can understand and optimize.</p>
<p>Given the vast differences in coding/packaging conventions out there this is a rather big task. Some packages work, some packages work with a few tweaks and some packages just don’t work at all.</p>
<p>Using this option is by far the best <strong>if it works</strong>. That is a big “if” though and getting it to work can be quite challenging. Ideally <code>:externs</code> would not be required anymore but in practice you still have to use them to work around a few kinks (for now).</p>
<p>I hope this will work reliably soon but we are not quite there yet.</p>
<h2 id="part-2">Part #2</h2>
<p>In the <a href="/blog/shadow-cljs/2017/09/15/js-dependencies-going-forward.html">second post</a> I will describe how <code>shadow-cljs</code> will handle this going forward.</p>This post is split into 2 parts aiming to answer the question:shadow-cljs - Introduction2017-09-15T12:21:12+02:002017-09-15T12:21:12+02:00https://code.thheller.com/blog/shadow-cljs/2017/09/15/shadow-cljs-introduction<p><a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> is my attempt at improving ClojureScript tooling. I started it a couple years ago (then called <code>shadow-build</code>) since I wanted to do things other tools were not able to do (yet, maybe still can’t).</p>
<p>It does use the default ClojureScript compiler and analyzer but not much beyond that. Everything related to “bundling” is rewritten completely. <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> works quite differently compared to other CLJS tools when it comes to deciding which code gets compiled, optimized and then bundled together. The focus has always been on ensuring that the final “release/production” version that gets shipped to your user is as optimized as possible. Code-splitting (aka. <code>:modules</code>) was the initial motivation to start this tool 3+ years ago but it has grown far beyond that.</p>
<p>Development is important as well so the usual live-reload workflow we all love works out of the box. The CLJS REPL is usually injected automatically so there is no need to configure it most of the time.</p>
<h1 id="configuration">Configuration</h1>
<p>A couple months ago I started working on a complete rework of the configuration code since I was unhappy with my own build configs. They had a lot of repetition and often were exact copies of each other with one parameter changed (eg. <code>:none</code> -&gt; <code>:advanced</code>).</p>
<p>Frequently I wasn’t able to express exactly what I wanted in my configuration which resulted in trying to hack something together in code to emulate the final result. Everyone that ever tried to work with <code>module.exports</code> for <code>node</code> code might be able to relate.</p>
<p>So from that the actual <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> project was born which threw away pretty much all of the configuration code and started from scratch. If you have used other CLJS tools before you might need to forget some stuff you have learned since they might not apply anymore.</p>
<h2 id="target-and-mode">:target and :mode</h2>
<p>The basic configuration concept starts with <code>:target</code> which defines how your code is bundle together based on the platform you plan on running the code in (eg. the Browser or <code>node</code>). CLJS has a <code>:target</code> option as well but was not specific enough for my goals.</p>
<p><code>:mode</code> is either <code>:dev</code> during development or <code>:release</code> for production builds. This means that you have ONE configuration that is used for both modes, if the configuration has development-only parts they will not apply when building a release and vice versa. There is no need to copy the configuration, the tool is smart enough to do the correct thing most of the time. Most of the time you’d probably just change the <code>:optimization</code> setting anyways, at least I did.</p>
<p>A whole lot of configuration is based on some reasonable defaults so the configuration you actually need to do is reduced to a minimum. You can still change the defaults if you need to though.</p>
<h3 id="browser---building-for-the-the-browser">:browser - building for the the Browser</h3>
<p>The <code>:browser</code> target is the most complex and deserves its own book probably. Shipping code for the browser has its own set of challenges since you usually want to optimize for size and speed. Depending on the amount of code you want to ship it might be important to split code via <code>:modules</code> to reduce (or delay) the amount of code the user has to download. Caching can also improve performance so the <code>:browser</code> target has some hooks to help with that. Using JS dependencies is also especially complicated because you can’t just load them via <code>require</code> at runtime. Everything must be carefully bundled together and the “best” option is usually very specific to each individual deployment scenario.</p>
<h3 id="npm-module---building-for-js-tools-or-nodejs">:npm-module - building for JS tools or node.js</h3>
<p>The goal here is to make it easy to consume ClojureScript from JS. Directly in <code>node.js</code> or by other tools such as <code>webpack</code> and higher level “frameworks” like <code>create-react-app</code>, <code>create-react-native-app</code>, etc.</p>
<p>Since ClojureScript uses Google Closure to do most of the <code>:advanced</code> stuff the code generated by the compiler targets the Closure coding conventions. These are very strict and quite different from what <code>node.js</code> or the other tools expect and understand.</p>
<p><code>:npm-module</code> enhances the code so non-closure tools can understand and consume it directly without sacrificing what Closure gives us. The result is that you can use <code>require</code> directly to consume the compiled (and even optimized) CLJS code.</p>
<p>A CLJS namespace like this</p>
<pre><code class="language-clojure">(ns demo.foo)
(defn hello [who]
(str "Hello, " who "!"))
</code></pre>
<p>Can be directly consumed in <code>node</code> via:</p>
<pre><code class="language-javascript">&gt; var x = require("shadow-cljs/demo.foo");
undefined
&gt; x.hello("JS")
'Hello, JS!'
</code></pre>
<p>You can use <code>:npm-module</code> to build your own <code>npm</code> packages as well.</p>
<h3 id="node-script-and-node-library">:node-script and :node-library</h3>
<p>These are slightly more focused versions of <code>:npm-module</code> and basically just optimize the last few bits. <code>:node-script</code> creates a <code>.js</code> file that you can run directly via <code>node</code>. <code>:node-library</code> allows you to compact multiple namespaces down to one map of exports.</p>
<h2 id="further-reading">Further reading</h2>
<p>This post only introduced a few concepts of <a href="https://github.com/thheller/shadow-cljs"><code>shadow-cljs</code></a> with a bit of background of why it exists and why it is different. Please refer to the <a href="https://github.com/thheller/shadow-cljs/blob/master/README.md">README</a> on how to use it.</p>
<p>I will go more in-depth about certain aspects of the tool in future posts, for now I just wanted a post I can refer to which answers the “Why/What”.</p>shadow-cljs is my attempt at improving ClojureScript tooling. I started it a couple years ago (then called shadow-build) since I wanted to do things other tools were not able to do (yet, maybe still can’t).Better HTML-&gt;Javascript Integration2014-12-20T14:40:14+01:002014-12-20T14:40:14+01:00https://code.thheller.com/blog/web/2014/12/20/better-javascript-html-integration<p>Article is a work in progress! I’ve been meaning to write it for 2 years but never quite know what to say or how to introduce the concept. I doubt the concept is new, I just don’t see it used very often.</p>
<p>This article is meant to present a solution to calling Javascript code from HTML, where the Javascript takes one or many arguments which may or may not be generated on the server-side. I believe the “common” approaches to this are not optimal and either conflict with best-practices regarding Performance or are simply too hard to use.</p>
<h3 id="what-do-we-want">What do we want?</h3>
<ul>
<li><code>&lt;script src="..."&gt;</code> at the bottom, just before the <code>&lt;/body&gt;</code></li>
<li>Javascript calls must be parameterizable (eg. with data from the server)</li>
<li>Javascript concatenated &amp; minified to keep requests to a minimum (obviously)</li>
<li>Execute Javascript ASAP!</li>
</ul>
<p>If you don’t yet minify your Javascript in Production you can stop reading here since you don’t seem to care about performance. I consider these points mandatory and they WILL have an impact on your page load times, which might have a serious impact on your User.</p>
<h3 id="how-do-we-get-there">How do we get there?</h3>
<p>It should look something like:</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span></span><span class="p">&lt;</span><span class="nt">head</span><span class="p">&gt;</span>
<span class="p">&lt;/</span><span class="nt">head</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">body</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">h1</span><span class="p">&gt;</span>My Super Page<span class="p">&lt;/</span><span class="nt">h1</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">&quot;target&quot;</span><span class="p">&gt;&lt;/</span><span class="nt">div</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">&quot;/js/jquery+app.js&quot;</span><span class="p">&gt;&lt;/</span><span class="nt">script</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">script</span><span class="p">&gt;</span>
<span class="nx">myFeature</span><span class="p">(</span><span class="nx">$</span><span class="p">(</span><span class="s2">&quot;#target&quot;</span><span class="p">),</span> <span class="p">{</span><span class="nx">config</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">data</span><span class="o">:</span> <span class="nx">big_json_blob_from_server</span><span class="p">});</span>
<span class="p">&lt;/</span><span class="nt">script</span><span class="p">&gt;</span>
<span class="p">&lt;/</span><span class="nt">body</span><span class="p">&gt;</span></code></pre></figure>
<p>While this seems to do everything we want it is usually not what I see on the Web since it rarely is that simple. HTML is constructed in Template Engines that employ various techniques to enable “reuse” of HTML and more often than not you’ll have some kind of “Layout” to fit it all together. Now you get the dilemma of having Javascript coupled to specific HTML but since we can’t execute Javascript just yet (remember it is available just before the <code>&lt;/body&gt;</code>) we have to resort to some trickery. Either your Template Engine allows to target specific areas of the output (eg. HTML goes here, JS goes here) or more commonly you resort to using <a href="http://api.jquery.com/ready/">jQuery.ready</a>.</p>
<p>So in practice things look more like:</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span></span><span class="p">&lt;</span><span class="nt">head</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">&quot;/js/jquery.js&quot;</span><span class="p">&gt;&lt;/</span><span class="nt">script</span><span class="p">&gt;</span>
<span class="p">&lt;/</span><span class="nt">head</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">body</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">h1</span><span class="p">&gt;</span>My Super Page<span class="p">&lt;/</span><span class="nt">h1</span><span class="p">&gt;</span>
<span class="c">&lt;!-- START some/include.html --&gt;</span>
<span class="p">&lt;</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">&quot;target&quot;</span><span class="p">&gt;&lt;/</span><span class="nt">div</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">script</span><span class="p">&gt;</span>
<span class="nx">$</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">myFeature</span><span class="p">(</span><span class="nx">$</span><span class="p">(</span><span class="s2">&quot;#target&quot;</span><span class="p">),</span> <span class="p">{</span><span class="nx">config</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">data</span><span class="o">:</span> <span class="nx">big_json_blob_from_server</span><span class="p">});</span>
<span class="p">});</span>
<span class="p">&lt;/</span><span class="nt">script</span><span class="p">&gt;</span>
<span class="c">&lt;!-- END some/include.html --&gt;</span>
<span class="p">&lt;</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">&quot;/js/app.js&quot;</span><span class="p">&gt;&lt;/</span><span class="nt">script</span><span class="p">&gt;</span>
<span class="p">&lt;/</span><span class="nt">body</span><span class="p">&gt;</span></code></pre></figure>
<p>Since we rely on <code>$(function() {});</code> (which is jQuery for: execute function when ready) we need to have jQuery available early, so we move that to the <code>&lt;head&gt;</code>. One blocking Javascript in the <code>&lt;head&gt;</code> is probably fine, might even be via a CDN to further reduce delay. Also I don’t quite like leaving “marker” elements (eg. <code>#target</code>) in the DOM just to refer to a specific place.</p>
<p>To my knowledge using “ready” is “recommended” in jQuery, other frameworks might suggest other ways. But it is very common to either use the document.onload or DOM ready events.</p>
<h3 id="what-can-we-do">What can we do?</h3>
<p>Eliminate all inline Javascript from HTML.</p>
<p>Yes, it is that simple. Well, we still want to be able to parameterize our function calls and refer to specific places in our HTML. So we put some extra data into the DOM, but please do not use <code>data-attr="huge-json-blob-from-server"</code> ever, thanks.</p>
<p>Instead I’d recommend:</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span></span><span class="p">&lt;</span><span class="nt">head</span><span class="p">&gt;</span>
<span class="p">&lt;/</span><span class="nt">head</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">body</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">h1</span><span class="p">&gt;</span>My Super Page<span class="p">&lt;/</span><span class="nt">h1</span><span class="p">&gt;</span>
<span class="c">&lt;!-- START some/include.html --&gt;</span>
<span class="p">&lt;</span><span class="nt">script</span> <span class="na">type</span><span class="o">=</span><span class="s">&quot;shadow/run&quot;</span> <span class="na">data-fn</span><span class="o">=</span><span class="s">&quot;my_app.feature1&quot;</span> <span class="na">data-ref</span><span class="o">=</span><span class="s">&quot;self&quot;</span><span class="p">&gt;</span>
<span class="p">{</span><span class="nx">config</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="nx">data</span><span class="o">:</span> <span class="nx">big_json_blob_from_server</span><span class="p">}</span>
<span class="p">&lt;/</span><span class="nt">script</span><span class="p">&gt;</span>
<span class="c">&lt;!-- END some/include.html --&gt;</span>
<span class="p">&lt;</span><span class="nt">script</span> <span class="na">async</span> <span class="na">src</span><span class="o">=</span><span class="s">&quot;/js/app.js&quot;</span><span class="p">&gt;&lt;/</span><span class="nt">script</span><span class="p">&gt;</span>
<span class="p">&lt;/</span><span class="nt">body</span><span class="p">&gt;</span></code></pre></figure>
<p>Couple that with <code>/js/app.js</code></p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span></span><span class="c1">// contents of jquery.js here</span>
<span class="c1">// contents of api-helper.js</span>
<span class="nx">my_app</span><span class="p">.</span><span class="nx">feature1</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">target</span><span class="p">,</span> <span class="nx">args</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// target is the &lt;script&gt; dom node aka our position in the DOM</span>
<span class="c1">// insert a new node before/after it, replace it, ...</span>
<span class="c1">// args is whatever the body of &lt;script&gt; is after JSON.parse</span>
<span class="p">};</span>
<span class="nx">api</span><span class="p">.</span><span class="nx">ready</span><span class="p">(</span><span class="s2">&quot;my_app.feature1&quot;</span><span class="p">);</span>
<span class="c1">// more of my super app</span></code></pre></figure>
<p><code>api.ready</code> is an imaginary function that just does</p>
<p><code>document.querySelectorAll('script[data-fn="my_app.feature1"]')</code></p>
<p>takes the contents of the <code>&lt;script&gt;</code> Tag, runs it through <code>JSON.parse</code> and passes that and itself as arguments to the function defined at <code>my_app.feature1</code>.</p>
<h3 id="where-is-the-code">Where is the code?</h3>
<p>Well, I have been using this technique for over 2 years now and it is working quite well. But I don’t use jQuery and all my code is ClojureScript. I tried to present the general concept since I feel it is applicable for every library/language. ClojureScript provides a lot of neat things that make this concept even easier to use (namespaces, dependency graphs, …). Plain JS might require some work.</p>
<p>Anyways, on the client this looks like:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">my.app</span>
<span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">shadow.api</span> <span class="ss">:refer</span> <span class="p">(</span><span class="nf">ns-ready</span><span class="p">)]))</span>
<span class="p">(</span><span class="kd">defn </span><span class="o">^</span><span class="ss">:export</span> <span class="nv">feature1</span> <span class="p">[</span><span class="nb">node </span><span class="p">{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">config</span> <span class="nv">data</span><span class="p">}</span> <span class="ss">:as</span> <span class="nv">args</span><span class="p">}]</span>
<span class="p">(</span><span class="nf">js/console.log</span> <span class="s">&quot;feature1&quot;</span> <span class="nb">node </span><span class="nv">args</span><span class="p">))</span>
<span class="p">(</span><span class="nf">ns-ready</span><span class="p">)</span></code></pre></figure>
<p>On the server:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span></span><span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nv">my-data</span> <span class="p">(</span><span class="nf">get-data-from-db</span><span class="p">)]</span>
<span class="p">(</span><span class="nf">hiccup/html</span>
<span class="p">[</span><span class="ss">:body</span>
<span class="p">[</span><span class="ss">:h1</span> <span class="s">&quot;My Super Page&quot;</span><span class="p">]</span>
<span class="p">(</span><span class="nf">cljs</span> <span class="ss">&#39;my.app/feature1</span> <span class="ss">:self</span> <span class="p">{</span><span class="ss">:config</span> <span class="nv">true</span> <span class="ss">:data</span> <span class="nv">my-data</span><span class="p">})]))</span></code></pre></figure>
<p>The client code is available <a href="https://github.com/thheller/shadow/blob/master/src/cljs/shadow/api.cljs">here</a>, you are welcome to use it. Server code is really simple so I won’t include it here. Note that I use EDN as transport format. You might also do this with <a href="https://github.com/cognitect/transit-format">transit</a> or plain JSON, whichever you prefer. Another neat side effect is that the DOM now is pure data again and all code is where it should be.</p>Article is a work in progress! I’ve been meaning to write it for 2 years but never quite know what to say or how to introduce the concept. I doubt the concept is new, I just don’t see it used very often.