Clojure JIRAhttp://dev.clojure.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=labels+%3D+Compiler
An XML representation of a search requesten-us4.464925-07-2011[CLJS-1059] Simple interface wanted to convert cljs forms to jshttp://dev.clojure.org/jira/browse/CLJS-1059
ClojureScript<p>In our project (a clojurescript debugger) we want to convert cljs forms or a sequence of forms into javascript so that they can be executed in the javascript console.</p>
<p>We would like something similar to closure/compile-form-seq (<a href="https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/closure.clj#L308">https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/closure.clj#L308</a>)</p>
<p>However, we need to supply, the namespace requires and locals in an env like this</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">{:ns {:name <span class="code-quote">"test.core"</span> :requires {(quote gstring) (quote goog.string)}} :locals {}}</pre>
</div></div>
<p>This code seems to do what we want.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(defn compile-form-seq
\<span class="code-quote">"Compile a sequence of forms to a JavaScript source string.\"</span>
[forms env]
(env/ensure
(compiler/with-core-cljs nil
(fn []
(with-out-str
(doseq [form forms]
(compiler/emit (analyzer/analyze env form))))))))</pre>
</div></div>
<p>I am not sure why I need env/ensure.</p>
<p>Would you be able to patch compile-form-seq to provide the needed interface, or suggest what we should be doing.</p>
<p>Thanks<br/>
Stu</p>CLJS-1059Simple interface wanted to convert cljs forms to jsEnhancementMajorOpenUnresolvedUnassignedStuart MitchellanalyzercompilerSun, 22 Feb 2015 21:38:52 -0600Mon, 23 Feb 2015 03:19:03 -060000
<p>Just to be clear: <br/>
1. when our debugger is at a breakpoint, <br/>
2. the user can type in an expression at the repl<br/>
3. in response, our debugger has to compile the user-typed-in expression to javascript (and then execute it, showing a result)<br/>
4. taking into account any local bindings. &lt;---- this is the key bit. </p>
<p>To satisfy point 4, our tool extracts all the 'locals' from the current call-frame, and then supplies all these local bindings in env/locals, so the compiler doesn't stick a namespace on the front of them.</p>
<p>For example, if there was a local binding for 'x' in the callstack, and the user's repl-entered-expression involves 'x', then we want the compiler to leave the symbol 'x' alone and to not put some namespace on the front of it. In the final javascript, it must still be 'x', not 'some.namespace.x'</p>
<p>Our method to achieve this is to put 'x' into env/locals when compiling &#8211; and it all works. Except, with the recent changes this has become more of a challenge. Hence this ticket asking for a way to pass in env. </p><p>You could wrap the user expression in an fn, that would allow you to skip messing with the locals. The REPL basically does the same trick for *1,*2,...</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(fn [x]
~user-expression-here)</pre>
</div></div>Global RankPatchCode[CLJS-1036] cljs.closure/build does not find upstream dependencies when called from worker threadhttp://dev.clojure.org/jira/browse/CLJS-1036
ClojureScript<p>Example stacktrace: <a href="https://www.refheap.com/97198">https://www.refheap.com/97198</a> (context is using figwheel in a clojure REPL from CIDER)</p>
<p>Because cljs.closure/build calls the 0-arity form of get-upstream-deps, it is implicitly using the current thread's classloader to find the deps.cljs resources. It is then assoc'ing the result into opts, so the caller has no way of gathering the dependencies themselves and passing them in.</p>CLJS-1036cljs.closure/build does not find upstream dependencies when called from worker threadDefectMajorClosedCompletedUnassignedMichael GriffithscljsclosurecompilerFri, 13 Feb 2015 13:38:46 -0600Fri, 20 Feb 2015 16:36:40 -0600Fri, 20 Feb 2015 16:36:38 -060011<p>Patch <a href="http://dev.clojure.org/jira/browse/CLJS-1036" title="cljs.closure/build does not find upstream dependencies when called from worker thread"><del>CLJS-1036</del></a>--001.patch, attached, uses merge instead of assoc to allow passing of the upstream dependency data via opts.</p><p>We could also allow passing of a :resources-classloader opt so callers don't have to gather the dependencies themselves, if you'd prefer a patch that does that.</p><p>This is a problem with calling cljs.closure/build inside nREPL in general when upstream deps are involved.</p><p>I would like Chas Emerick to chime in on this thread before considering it.</p><p>Could someone point me to a simple (sample?) project that uses upstream dependencies? I haven't used them yet, and don't have such a thing handy.</p>
<p>The proposed patch is confusing to me; if the problem is in resolving dependencies due to classloader state/configuration, how does it help to give <tt>opts</tt> the opportunity to clobber what <tt>get-upstream-deps</tt> returns? That is, doesn't this just move the problem downstream to who/whatever is calling <tt>build</tt>?</p><p>Chas: yes, the proposed patch simply moves the responsibility to the caller. I was under the assumption that changing how get-upstream-deps/build resolves the classloader by default would be too-breaking of a change, and had not even considered the fact that this might be a broader issue with nREPL itself.</p>
<p>Example usage of get-upstream-deps in the wild (that even mentions classloader issues): <a href="https://github.com/immutant/immutant-release/blob/ea538feb548bde86ebce20ec679da7a19b639259/integration-tests/apps/ring/basic-ring/src/basic_ring/core.clj#L51">https://github.com/immutant/immutant-release/blob/ea538feb548bde86ebce20ec679da7a19b639259/integration-tests/apps/ring/basic-ring/src/basic_ring/core.clj#L51</a></p>
<p>Note that Weasel (at least) is currently gathering the upstream deps itself too: <a href="https://github.com/tomjakubowski/weasel/commit/5cd7009f584100f0d89fc7f00079458bb1f2c016">https://github.com/tomjakubowski/weasel/commit/5cd7009f584100f0d89fc7f00079458bb1f2c016</a></p>
<p>I'll set up an example project tonight if you like, but I think all you need is to add <tt><span class="error">&#91;cljsjs/react &quot;0.12.2-5&quot;&#93;</span></tt> as a dependency to an existing cljs project and then require <tt><span class="error">&#91;cljsjs.react&#93;</span></tt> in one of its namespaces.</p><p>Chas, David here is a very quick example:</p>
<p>If you do a </p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">lein <span class="code-keyword">new</span> om-mies hello-world
cd hello-world
lein repl
user&gt; (require '[cljs.closure :as cc])
user&gt; (cc/build <span class="code-quote">"src"</span> {})</pre>
</div></div>
<p>Assuming <tt>lein repl</tt> started an nREPL repl. This will produce the error mentioned above. The problem is that om is utilizing the new deps.js facility to autmatically import <tt>cljsjs.react</tt>. get-upstream-deps depends on being run on the main thread.</p>
<p><a href="https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/closure.clj#L1116">https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/closure.clj#L1116</a></p>
<p>A solution that fixes this in <tt>get-upstream-deps*</tt> would be preferable. </p>
<p>Is this simple solution to use the system classloader when not on the main thread:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(<span class="code-keyword">if</span> (not= (.getName (<span class="code-object">Thread</span>/currentThread)) <span class="code-quote">"main"</span>)
(java.lang.<span class="code-object">ClassLoader</span>/getSystemClassLoader)
(. (<span class="code-object">Thread</span>/currentThread) (getContextClassLoader)))</pre>
</div></div>
<p>To naive? My experience with Java threads and classloaders is minimal.</p>
<p>Or should we always use the (java.lang.ClassLoader/getSystemClassLoader) in this case?</p>
<p>Using the system classloader isn't necessary. <tt>get-upstream-deps</tt> is currently using <tt>.findResources</tt>, not <tt>.getResources</tt>; the difference is that the former looks only for matching resources in the target classloader, while the latter first delegates to the parent classloader, and only looks locally if nothing is found there. So, just using <tt>.getResources</tt> instead will make all current usage work as expected, I think. e.g.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">;; `lein repl` nREPL here
user=&gt; (enumeration-seq (.getResources (. (<span class="code-object">Thread</span>/currentThread) (getContextClassLoader)) <span class="code-quote">"deps.cljs"</span>))
(#&lt;URL jar:file:/home/chas/.m2/repository/cljsjs/react/0.12.2-5/react-0.12.2-5.jar!/deps.cljs&gt;)
user=&gt; (enumeration-seq (.findResources (. (<span class="code-object">Thread</span>/currentThread) (getContextClassLoader)) <span class="code-quote">"deps.cljs"</span>))
nil</pre>
</div></div>
<p>There are some classloader edge cases that might motivate making <tt>get-upstream-deps</tt> perform a more comprehensive search, explicitly pinging all classloaders for matching resources. This is sometimes necessary for these sorts of scanning scenarios if e.g. a deps.cljs file is available via a parent classloader, which will therefore "shadow" other deps.cljs files in child classloaders. In that case, you need something like <a href="https://github.com/cemerick/pomegranate/blob/master/src/main/clojure/cemerick/pomegranate.clj#L107"><tt>cemerick.pomegranate/resources</tt></a>.</p>
<p>Hope that helps. <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p><p>Chas's suggested fix has been committed to master. It works on the simple tests I've run but I'd like to hear more confirmation. Thanks everyone.</p>
<p><a href="https://github.com/clojure/clojurescript/commit/79208f5bf15825a2ba2d0ce95aae6d2b71966494">https://github.com/clojure/clojurescript/commit/79208f5bf15825a2ba2d0ce95aae6d2b71966494</a></p><p>Closing this one. Chas's solution is the correct one.</p>Global Rank[CLJS-905] Dependency tree failhttp://dev.clojure.org/jira/browse/CLJS-905
ClojureScript<p>In the following repo</p>
<p><a href="https://github.com/skrat/deporder-bug-cljs">https://github.com/skrat/deporder-bug-cljs</a></p>
<p>Looking at the compiled source, I would expect the `bug.b` module to be initialized before `bug.a`. It is not, and it leads to a runtime error about `bug.b/registry` being undefined. Now, when I switch things around (`around` branch in the repo), rename `bug.a` to `bug.b` and vice versa, I will get a compiler warning about `bug.a/registry` being undeclared, but at runtime everything works as expected.</p>
<p>This example was extracted from a large project that uses similar macro that expands to `swap!`, where with `none` optimizations everything works, but with `simple` and `advanced` it throws during runtime. </p>CLJS-905Dependency tree failDefectMajorClosedDeclinedUnassignedDusan MaliarikbugcompilerThu, 11 Dec 2014 05:30:13 -0600Sun, 14 Dec 2014 05:32:21 -0600Fri, 12 Dec 2014 06:22:56 -060001<p>Technically not a "bug" but a weakness due to the nature ClojureScript handles macros.</p>
<p>'bug.macros requires 'bug.b cause it emits code calling it. 'bug.a requires 'bug.macro and therefore technically requires 'bug.b but since it cannot figure this out you technically have to tell it by also doing a (:require <span class="error">&#91;bug.b&#93;</span>) in 'bug.a.</p>
<p>I had this problem a while back <span class="error">&#91;1&#93;</span> and implemented a workarround by letting the macro declare what namespaces it requires, which in your case would look like</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(def macro
^{:js-requires '[bug.b]}
zwap [v]
`(swap! bug.b/registry conj ~v))</pre>
</div></div>
<p>But I was turned down so the feature never made it.</p>
<p>The recommended way is to have macros in namespaces that have a matching CLJS namespace and using :include-macros/:refer-macros when requiring them. Has its own issues cause a) everyone needs to know that bug.b uses macros and b) everyone needs to know what actually is a macro.</p>
<p>Anyways, in your case move bug/macros.clj to bug/b.clj and rewrite bug/a.cljs to</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns bug.a
(:require [bug.b :refer-macros (zwap)]))</pre>
</div></div>
<p><span class="error">&#91;1&#93;</span> <a href="https://groups.google.com/forum/?fromgroups#!searchin/clojurescript/macro$20require/clojurescript/sLRGCIa8E1c/N9sFcTP_i9wJ">https://groups.google.com/forum/?fromgroups#!searchin/clojurescript/macro$20require/clojurescript/sLRGCIa8E1c/N9sFcTP_i9wJ</a></p><p>Shameless plug: shadow-build <span class="error">&#91;1&#93;</span> supports the macro meta feature. Still sort of hack-ish since it only works when macros are referred directly but my use of macros is very limited, basically only whats in <span class="error">&#91;2&#93;</span>.</p>
<p><span class="error">&#91;1&#93;</span> <a href="https://github.com/thheller/shadow-build">https://github.com/thheller/shadow-build</a><br/>
<span class="error">&#91;2&#93;</span> <a href="https://github.com/thheller/shadow/blob/master/src/clj/shadow/macros.clj">https://github.com/thheller/shadow/blob/master/src/clj/shadow/macros.clj</a></p><p>I wonder how this works in Clojure land, but I would still say it's a bug in ClojureScript compiler, because, afaik, it first expands macros, which results in a symbol from another namespace ('bug.b), and at that point, 'bug.b should be added as a dependency, resulting in goog.require('bug.b') in the compiled 'bug.a, isn't that right?</p><p>In java Clojure you still need to ensure that the symbols you expand in your macros are resolvable by the time the expanded form is executed. For example, if you create a macro that includes a clojure.data/diff call, you need to require clojure.data somewhere in your program, ideally in the namespace that defines the macro. Symbols which refer to other namespaces do not automatically require those namespaces. Most likely you will get a {{ClassNotFoundException <span class="error">&#91;namespace-part&#93;</span>}} when you try to execute the form.</p>
<p>However Clojurescript introduces the extra problem that the macro expansion and execution environments are different. (In Clojure they are the same.) So there are additional constraints not present in Clojure:</p>
<p>1. A separate, completely different namespace dependency tree exists at compile time vs runtime.<br/>
2. Because of the Google Closure compiler, the entire runtime dependency tree must be known statically at compile time. (Otherwise some code you need at runtime might not be compiled into the final js.)<br/>
3. Because of javascript's dynamism, some symbols may not be known <b>even to the Google Closure compiler</b>. For example, if you have a macro which expands to calls to jQuery code, neither clojure, clojurescript, nor google closure, nor even your <b>browser</b> have a mechanism to "require" jQuery. The code must simply trust that you have done whatever is necessary to ensure jQuery/whatever is resolvable when it finally executes.</p>
<p>Additionally, automatically requiring namespaces from macro expansion is impossible (not a bug in the ClojureScript compiler):<br/>
1. The namespaces might not exist (i.e. the symbol is being used as a symbol, rather than as a reference or var).<br/>
2. The namespace/symbol might not be resolveble <b>right now</b>. This is clearer in the clojurescript case: is that symbol a reference to a <b>clojure</b> namespace that I should require right now, or is it a reference to a clojurescript namespace which I need to expand code to require, but I can't actually expect it to exist until runtime?<br/>
3. The macro might create symbols dynamically, in which case there is no way to know what namespaces the macro requires without executing the macro.</p><p>Yes that's clear to me. Let's factor the non-ClojureScript deps like jQuery out, that's of course impossible to get right for everyone. What I propose is:</p>
<ol>
<li>run the macros</li>
<li>walk through the expanded code</li>
<li>when a symbol from another namespace is found, if this namespace isn't already required in that package, do either
<ul>
<li>add the require for that namespace</li>
<li>print out a warning</li>
</ul>
</li>
</ol>
<p>Currently it doesn't do anything, it just breaks during the runtime.</p><p>I get a Warning?</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">WARNING: Use of undeclared Var bug.b/registry at line 8 src/bug/a.cljs</pre>
</div></div>
<p>Comes down to a tooling problem again (hint: use shadow-build) that the warning is only printed once with lein-cljsbuild.</p><p>Still, why does the different naming produce different results?</p><p>This is not a bug.</p><p>The behavior we talked about earlier might not be a bug indeed, but the fact that merely changing namespace names breaks things, sure is a bug. David, I advise to checkout the repo, try for yourself.</p><p>I looked at the repo there is no compiler bug that I can discern. You just have two different invalid ClojureScript programs. All the salient points have been covered by other commenters.</p><p>I won't insist anymore, nuff was said. Still the point about naming choice affecting the compiler output, wasn't addressed (compare 'master' and 'around' branches). "invalid ClojureScript programs" made me laugh though <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/> thanks</p><p>Naming has very little to do with it. If you declare no dependency when using the macro (either bug.b or bug.a depending on branch) the analyzer cannot establish its "correct" position in the dependency graph. Therefore it is basically "luck" whether it will end up in the correct position or not. Declare the correct position and it will always be in the correct position.</p>
<p>Pretty sure the around branch will behave exaclty like master when you switch the order of the :require, but again: DO NOT RELY on such undefined behavior. Declare the dependencies!</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns bug.core
(:require [bug.b :refer [woot]]
[bug.a :refer [registry]]
))</pre>
</div></div>Global Rank[CLJS-893] Nodejs target shouldn't insert shebang by defaulthttp://dev.clojure.org/jira/browse/CLJS-893
ClojureScript<p>If people want, they will prepend the shebang themselves, or run the file using node, where-ever it is on PATH. This is not good default behavior, and breaks things when the intention is to compile a node.js module. </p>CLJS-893Nodejs target shouldn't insert shebang by defaultDefectMajorClosedDuplicateUnassignedDusan MaliarikbugcompilerTue, 2 Dec 2014 12:20:08 -0600Tue, 2 Dec 2014 13:29:30 -0600Tue, 2 Dec 2014 13:29:30 -060001<p>Whether it was a good or bad default is water under the bridge at this point. Do you have any suggested solutions? <tt>:node-module</tt> comes to mind for a new target that doesn't append the shebang. Another option would be something to explicitly suppress it.</p><p>Sorry, my fault<br/>
<a href="https://github.com/clojure/clojurescript/commit/00a9b7bde0c8823175560b453a00e7be09ddd250">https://github.com/clojure/clojurescript/commit/00a9b7bde0c8823175560b453a00e7be09ddd250</a></p>
<p>I just have to :hashbang false. Unfortunately I was unable to find a complete list of supported options to the compiler. Is it documented somewhere?</p><p>It is not but the wiki is publicly editable - nothing is stopping anyone from creating a page that documents all the knobs and their relationships.</p>Global Rank[CLJS-885] read-string result raising a warninghttp://dev.clojure.org/jira/browse/CLJS-885
ClojureScript<p>The result of read-string is not correctly recognized as a number in the example discussed here:</p>
<p><a href="https://groups.google.com/forum/#!topic/clojurescript/kwJNH2MBbao">https://groups.google.com/forum/#!topic/clojurescript/kwJNH2MBbao</a></p>
<p>Wrapping it in a call to the identity function makes the symptom disappear.</p>Ubuntu 12.04 (64 bit)
<br/>
Oracle JDK 1.7.0_72
<br/>
leiningen 2.5.0
<br/>
ClojureScript 0.0-2371
<br/>
lein-cljsbuild 1.0.4-SNAPSHOTCLJS-885read-string result raising a warningDefectMajorClosedCompletedDavid NolenStefano PugnettiCompilerbugTue, 11 Nov 2014 11:12:31 -0600Fri, 5 Dec 2014 13:37:03 -0600Fri, 5 Dec 2014 13:37:00 -060001<p>fixed <a href="https://github.com/clojure/clojurescript/commit/ecc5765d599cadb0d22e2d66635df9eb68c60774">https://github.com/clojure/clojurescript/commit/ecc5765d599cadb0d22e2d66635df9eb68c60774</a></p>Global Rank[CLJS-875] Compiler error on :"http://dev.clojure.org/jira/browse/CLJS-875
ClojureScript<p>(str "this will break" :"the parser")<br/>
This simple form (note the misplaced ':' causes the compiler to barf with the following error<br/>
Caused by: clojure.lang.ExceptionInfo: EOF while reading string</p>
<p>no mention of the ':' and in larger blocks of code the unmatched brackets several lines away is flagged</p>
<p>in clojure the error is<br/>
RuntimeException Invalid token: : clojure.lang.Util.runtimeException (Util.java:221)<br/>
RuntimeException Unmatched delimiter: ) clojure.lang.Util.runtimeException (Util.java:221)</p>
<p>Note it mentions the errant ':'</p>Ubuntu 14.04
<br/>
org.clojure/clojurescript &quot;0.0-2277&quot;CLJS-875Compiler error on :"DefectMajorClosedCompletedDavid NolenStuart MitchellcompilererrormsgsWed, 22 Oct 2014 23:00:57 -0500Wed, 5 Nov 2014 06:33:58 -0600Wed, 5 Nov 2014 06:33:56 -060002<p>I would like the error reporting from the compiler improved in this case, so that it mentions that the problem is caused by the ':'.</p><p>This appears to be an issue caused by tools.reader, I'm working on a fix for it</p><p>I just pushed a 0.8.10 release of tools.reader that fixes this issue</p><p>fixed <a href="https://github.com/clojure/clojurescript/commit/ecb13f06e766182e1daa7d227def3bad98bd8c49">https://github.com/clojure/clojurescript/commit/ecb13f06e766182e1daa7d227def3bad98bd8c49</a></p>Global Rank[CLJS-869] When preamble is not found in source directory, compiler does not report ithttp://dev.clojure.org/jira/browse/CLJS-869
ClojureScript<p>file target is not validated<br/>
<a href="https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/closure.clj#L525-L531">https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/closure.clj#L525-L531</a><br/>
externs might need checking also.</p>
<p>Steps:<br/>
in project.clj add :preamble <span class="error">&#91;&quot;nosuchfile&quot;&#93;</span> to the :compiler map</p>
<p>Expected:<br/>
"Could not find preamble file X in source path"</p>
<p>Actual:<br/>
Stacktrace gives no indication of the problem (without reading the compiler code)::</p>
<p>Compiling "resources/public/js/device_manager.js" failed.<br/>
java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil<br/>
at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:544)<br/>
at clojure.java.io$fn_<em>8628$G</em><em>8610</em>_8635.invoke(io.clj:69)<br/>
at clojure.java.io$reader.doInvoke(io.clj:102)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:410)<br/>
at clojure.lang.AFn.applyToHelper(AFn.java:154)<br/>
at clojure.lang.RestFn.applyTo(RestFn.java:132)<br/>
at clojure.core$apply.invoke(core.clj:626)<br/>
at clojure.core$slurp.doInvoke(core.clj:6390)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:410)<br/>
at cljs.closure$preamble_from_paths$fn__3018.invoke(closure.clj:524)<br/>
at clojure.core$map$fn__4245.invoke(core.clj:2557)<br/>
at clojure.lang.LazySeq.sval(LazySeq.java:40)<br/>
at clojure.lang.LazySeq.seq(LazySeq.java:49)<br/>
at clojure.lang.RT.seq(RT.java:484)<br/>
at clojure.core$seq.invoke(core.clj:133)<br/>
at clojure.core$apply.invoke(core.clj:624)<br/>
at cljs.closure$preamble_from_paths.invoke(closure.clj:524)<br/>
at cljs.closure$make_preamble.invoke(closure.clj:529)<br/>
at cljs.closure$optimize.doInvoke(closure.clj:592)<br/>
at clojure.lang.RestFn.applyTo(RestFn.java:139)<br/>
at clojure.core$apply.invoke(core.clj:626)<br/>
at cljs.closure$build.invoke(closure.clj:980)<br/>
at cljs.closure$build.invoke(closure.clj:923)<br/>
at cljsbuild.compiler$compile_cljs$fn__3200.invoke(compiler.clj:58)<br/>
at cljsbuild.compiler$compile_cljs.invoke(compiler.clj:57)<br/>
at cljsbuild.compiler$run_compiler.invoke(compiler.clj:159)<br/>
at user$eval3326$iter_<em>3344</em><em>3348$fn</em><em>3349$fn</em>_3361.invoke(form-init6680721970243583147.clj:1)<br/>
at user$eval3326$iter_<em>3344</em><em>3348$fn</em>_3349.invoke(form-init6680721970243583147.clj:1)<br/>
at clojure.lang.LazySeq.sval(LazySeq.java:40)<br/>
at clojure.lang.LazySeq.seq(LazySeq.java:49)<br/>
at clojure.lang.RT.seq(RT.java:484)<br/>
at clojure.core$seq.invoke(core.clj:133)<br/>
at clojure.core$dorun.invoke(core.clj:2855)<br/>
at clojure.core$doall.invoke(core.clj:2871)<br/>
at user$eval3326.invoke(form-init6680721970243583147.clj:1)<br/>
at clojure.lang.Compiler.eval(Compiler.java:6703)<br/>
at clojure.lang.Compiler.eval(Compiler.java:6693)<br/>
at clojure.lang.Compiler.load(Compiler.java:7130)<br/>
at clojure.lang.Compiler.loadFile(Compiler.java:7086)<br/>
at clojure.main$load_script.invoke(main.clj:274)<br/>
at clojure.main$init_opt.invoke(main.clj:279)<br/>
at clojure.main$initialize.invoke(main.clj:307)<br/>
at clojure.main$null_opt.invoke(main.clj:342)<br/>
at clojure.main$main.doInvoke(main.clj:420)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:421)<br/>
at clojure.lang.Var.invoke(Var.java:383)<br/>
at clojure.lang.AFn.applyToHelper(AFn.java:156)<br/>
at clojure.lang.Var.applyTo(Var.java:700)<br/>
at clojure.main.main(main.java:37)</p>CLJS-869When preamble is not found in source directory, compiler does not report itDefectMinorClosedCompletedTimothy PratleyTimothy PratleycljsbuildcompilerpatchFri, 3 Oct 2014 11:04:18 -0500Tue, 2 Dec 2014 10:43:18 -0600Wed, 8 Oct 2014 16:59:45 -050001<p>Reports a warning</p>
<p>Preamble resource file not found: &lt;file1&gt; &lt;file2&gt; &lt;file3&gt;</p><p>fixed <a href="https://github.com/clojure/clojurescript/commit/9fd6bf5bd55421c3d5becacc5230ed661d6fb3c3">https://github.com/clojure/clojurescript/commit/9fd6bf5bd55421c3d5becacc5230ed661d6fb3c3</a></p>Global Rank[CLJS-812] Recurring from a case statement emits invalid JavaScripthttp://dev.clojure.org/jira/browse/CLJS-812
ClojureScript<p>In 0.2227, compiling the following form produces syntactically invalid JavaScript:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(defn reproduce
[value]
(case value
:a (recur :b)
:b 0))
</pre>
</div></div>
<p>Yields:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>bug_repro_test.reproduce = (function reproduce(value) {
while (true) {
var G__5832 = (((value instanceof cljs.core.Keyword)) ? value.fqn : null);
var caseval__5833;
switch (G__5832) {
case "b":
caseval__5833 = 0
break;
case "a":
caseval__5833 = {
var G__5834 = cljs.core.constant$keyword$67;
value = G__5834;
continue;
}
break;
default:
caseval__5833 = (function () {
throw (new Error(("No matching clause: " + cljs.core.str.cljs$core$IFn$_invoke$arity$1(value))))
})()
}
return caseval__5833;
break;
}
});
</pre>
</div></div>
<p>When evaluated in any JavaScript environment (including the Google Closure compiler) environment, this yields a syntax error in this code:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>caseval__5833 = {
var G__5834 = cljs.core.constant$keyword$67;
value = G__5834;
continue;
}
</pre>
</div></div>CLJS-812Recurring from a case statement emits invalid JavaScriptDefectCriticalClosedCompletedUnassignedLuke VanderHartbugcompilerMon, 9 Jun 2014 22:37:32 -0500Fri, 13 Jun 2014 15:59:38 -0500Fri, 13 Jun 2014 15:59:35 -050001<p>Good catch. I suspect that throw may be similarly problematic.</p><p>fixed in master <a href="https://github.com/clojure/clojurescript/commit/cc11c7996ba8522d6767fb45df2f76e20e4c1773">https://github.com/clojure/clojurescript/commit/cc11c7996ba8522d6767fb45df2f76e20e4c1773</a></p>Global Rank[CLJS-748] Dump analyzer state during a compilation.http://dev.clojure.org/jira/browse/CLJS-748
ClojureScript<p>CLJS doesn't have the luxury of a fully-reified runtime environment, therefore effective tooling needs a view of the internal compiler state. This issue addresses this need by enabling a dump of the compiler state based on additional cljs compiler options.</p>
<p>The compiler state is a map contained within an atom, either passed in as the 'compiler-env' arg to cljs.closure/build or contained globally within cljs.env/<b>compiler</b>.</p>
<p>The prototype is implemented as such:</p>
<p>:dump-analysis-to <br/>
either a string or :print, when :print, output will be written to <b>out</b>, when a string, the argument will be passed to clojure.java.io/writer.</p>
<p>:dump-analysis-full<br/>
checked for truthiness, default is false<br/>
When this is switched on, the full structure of the compiler state will be printed, which is impractically large for most use cases. In normal operation, :env keys and extended :method-params will be removed from the output, making the analysis represent simply the globally reachable environment. Additionally, only the contents under :cljs.analyzer/namespaces and :cljs.analyzer/constants will be printed.</p>CLJS-748Dump analyzer state during a compilation.EnhancementMinorClosedCompletedUnassignedGary TrakhmanCompilerMon, 13 Jan 2014 18:43:29 -0600Sat, 3 Jan 2015 18:13:56 -0600Sat, 3 Jan 2015 18:13:53 -060001<p>Added implementing patch</p><p>I question the utility of :dump-analysis-full for now. Lets remove it.</p><p>I was thinking it might be good along with :print followed by a pipe, but... I don't have an explicit use-case. I'll create a new patch.</p><p>This ticket is subsumed by <tt>:cache-analysis</tt> support</p>Global RankPatchCode[CLJS-740] undeclared-ns warnings are brokenhttp://dev.clojure.org/jira/browse/CLJS-740
ClojureScript<p>In the recent versions of cljs, the undeclared-ns warnings have stopped working. <a href="https://github.com/clojure/clojurescript/commit/abd44502079425f76f31c5432e7366ecd33d866a">This patch</a> seems to be the culprit.</p>CLJS-740undeclared-ns warnings are brokenDefectMinorClosedCompletedUnassignedErik OuchterlonyCompilerbugpatch,Thu, 2 Jan 2014 06:58:30 -0600Fri, 17 Jan 2014 09:26:05 -0600Fri, 17 Jan 2014 09:26:02 -060001<p>Great thanks!</p><p><a href="http://dev.clojure.org/jira/browse/CLJS-615" title="Warning when required lib does not exist on disk"><del>CLJS-615</del></a></p><p>fixed, <a href="https://github.com/clojure/clojurescript/commit/b8cf302b1500e88e0602e72fa6aec6f7328a1a00">https://github.com/clojure/clojurescript/commit/b8cf302b1500e88e0602e72fa6aec6f7328a1a00</a></p>Global RankPatchCode[CLJS-697] top-level symbol reference doesn't get an automatically inserted ns-namehttp://dev.clojure.org/jira/browse/CLJS-697
ClojureScript<p>I'm trying to use a Node.js module (with nested namespaces) in ClojureScript - the code goes like this:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns myapp)
(def evernote (js/require <span class="code-quote">"evernote"</span>))
(def token <span class="code-quote">"TOKEN"</span>)
(defn <span class="code-keyword">do</span>-sth []
(let [client (evernote.Evernote.Client. (js-obj <span class="code-quote">"token"</span> token))]
(.log js/console client)))
(<span class="code-keyword">do</span>-sth)</pre>
</div></div>
<p>which gets compiled (with :simple optimization) to:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java"><span class="code-keyword">var</span> myapp = {}
myapp.evernote = require(<span class="code-quote">"evernote"</span>)
myapp.token = <span class="code-quote">"TOKEN"</span>
myapp.do_sth = function() {
<span class="code-keyword">var</span> a = <span class="code-keyword">new</span> evernote.Evernote.Client({token:myapp.token})
<span class="code-keyword">return</span> console.log(a)
}
myapp.do_sth()</pre>
</div></div>
<p>which will obviously fail with error "Uncaught ReferenceError: evernote is not defined".</p>org.clojure/clojurescript &quot;0.0-2030&quot;CLJS-697top-level symbol reference doesn't get an automatically inserted ns-nameDefectMajorClosedCompletedUnassignedLimbo PengCompilerbugnamespaceSat, 23 Nov 2013 20:41:17 -0600Sat, 23 Nov 2013 23:55:41 -0600Sat, 23 Nov 2013 23:55:39 -060001<p>fixed, <a href="https://github.com/clojure/clojurescript/commit/d4bf88269e1d96468a19fd481f32628d4eafec9d">https://github.com/clojure/clojurescript/commit/d4bf88269e1d96468a19fd481f32628d4eafec9d</a></p>Global Rank[CLJS-685] Cannot call method 'fromArray' of undefined -- Clojurescript 0.0-2030http://dev.clojure.org/jira/browse/CLJS-685
ClojureScript<p>Clojurescript 0.0-2030</p>
<p>This line from compile cljs.core is causing problems:</p>
<p>cljs.core.PersistentQueue.EMPTY = (new cljs.core.PersistentQueue(null, 0, null, cljs.core.with_meta(cljs.core.PersistentVector.EMPTY, cljs.core.PersistentArrayMap.fromArray(<span class="error">&#91;new cljs.core.Keyword(null, &quot;end-line&quot;, &quot;end-line&quot;, 2693041432), 3820, new cljs.core.Keyword(null, &quot;end-column&quot;, &quot;end-column&quot;, 3799845882), 69&#93;</span>, true)), 0));</p>
<p>error message: Uncaught TypeError: Cannot call method 'fromArray' of undefined.</p>
<p>That's the first mention of fromArray in that file. I don't know if it's an ordering problem.</p>Linux 3.2.0-52-generic x86_64 GNU/Linux, java 1.7, clojure 1.5.1CLJS-685Cannot call method 'fromArray' of undefined -- Clojurescript 0.0-2030DefectMinorClosedCompletedUnassignedJohn ChijiokeCompilerbugerrormsgsSun, 17 Nov 2013 22:36:48 -0600Tue, 26 Nov 2013 06:32:44 -0600Fri, 22 Nov 2013 08:02:28 -060003<p>I solved it by replacing [] with cljs.core.PersistentVector.EMPTY. I think this must be a reader problem. </p><p>This ticket needs more details, how can this error be reproduced?</p><p>Hi, I'm seeing the same problem with tools.reader 0.8.0.</p>
<p>Any Clojurescript file (even an empty file) will produce the error.</p>
<p>Clojure: 1.6.0-alpha2<br/>
Clojurescript: 0.0-2030<br/>
Cljsbuild: 1.0.0<br/>
tools.reader: 0.8.0</p>
<p>Tried `lein cljsbuild clean`.</p>
<p>Problem is resolved by dropping back to tools.reader 0.7.10.</p>
<p>Update: have created an issue on the tools.reader GitHub page: <a href="https://github.com/clojure/tools.reader/issues/7">https://github.com/clojure/tools.reader/issues/7</a></p>
<p>Update 2: this isn't something specific to Cljs 0.0-2030 btw, tools.reader 0.8.0 seems to produce the same error against at least Cljs 0.0-2060, 0.0-2027, 0.0-2024.</p><p>tools.reader 0.8.0 introduces end-column/end-line metadata, this needs to be elided as per line/column to avoid this bootstrapping issue.</p><p>fixed, <a href="http://github.com/clojure/clojurescript/commit/36d401797f85c99794eef8a71239641930c36871">http://github.com/clojure/clojurescript/commit/36d401797f85c99794eef8a71239641930c36871</a></p><p>Thanks a lot David, Nicola - much appreciated! Cheers <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p><p>Thanks David. Cheers!</p>Global Rank[CLJS-575] cljsc.bat emit FileNotFoundException when compile samples in windowshttp://dev.clojure.org/jira/browse/CLJS-575
ClojureScript<p>cljsc.bat emit FileNotFoundException when it compile samples of the ClojureScript project in windows like below.</p>
<p>------------------------------------------------<br/>
Exception in thread "main" java.io.FileNotFoundException: Could not locate cljs/closure__init.class<br/>
or cljs/closure.clj on classpath:<br/>
------------------------------------------------</p>
<p>It is caused by lack of a backslash in the end of path of the system environment variable, %CLOJURESCRIPT_HOME% set by a user.<br/>
In the case CLASSPATH is set to "C:\\clojure\clojurescriptsrc\clj;C:\\clojure\clojurescriptsrc\cljs" and this make it impossible for javac to find cljs/clojure.clj file.</p>
<p>So it can be solved by adding a backslash to the path of %CLOJURESCRIPT_HOME%.</p>
<p>I attached the patched file, "cljsc-path.bat"</p>in windows 7CLJS-575cljsc.bat emit FileNotFoundException when compile samples in windowsDefectMajorOpenUnresolvedUnassignedPark Sang KyuCompilerbugpatchSun, 25 Aug 2013 01:13:55 -0500Thu, 19 Jun 2014 10:27:48 -050001<p>Can we please get a proper git diff thanks (and please send in your CA)! Also would be nice to get Windows users to check this out.</p><p>git diff</p><p>Thank you! Have you sent in your CA? <a href="http://clojure.org/contributing">http://clojure.org/contributing</a></p><p>Yes i have sent my CA.</p><p>Excellent, the patch is not correctly formatted. Can we get a new patch that conforms to <a href="http://github.com/clojure/clojurescript/wiki/Patches">http://github.com/clojure/clojurescript/wiki/Patches</a></p>Global RankPatchCode[CLJS-573] checkboxes 'indeterminate' attribute gets munged in advanced modehttp://dev.clojure.org/jira/browse/CLJS-573
ClojureScript<p>HTML5 defines an <tt>indeterminate</tt> attribute for checkboxes<br/>
<a href="http://www.w3.org/TR/html5/forms.html#checkbox-state-(type=checkbox">http://www.w3.org/TR/html5/forms.html#checkbox-state-(type=checkbox</a>)<br/>
Trying to set this attribute with ClojureScript<br/>
<tt>(set! (.-indeterminate checkbox-element) true))</tt><br/>
doesn't work when compiling in advanced mode.</p>
<p>Looking at the generated JavaScript code, I see that the <tt>indeterminate</tt> property gets munged. I don't know which <tt>externs</tt> files the ClojureScript compiler uses by default, but the <tt>indeterminate</tt> property is not referenced in Closure Compiler's <tt>externs</tt> for HTML5<br/>
<a href="https://code.google.com/p/closure-compiler/source/browse/externs/html5.js">https://code.google.com/p/closure-compiler/source/browse/externs/html5.js</a></p>
<p>In my project I've worked around this by adding this to my own <tt>externs</tt> file</p>
<p>/** @type {boolean} */<br/>
HTMLInputElement.prototype.indeterminate;</p>
<p>but I guess this should be "built-in" into the ClojureScript compiler (or Google's Closure Compiler)?</p>ClojureScript 0.0-1853, Clojure 1.5.1, Java 1.6.0_51, OS X 10.7.5CLJS-573checkboxes 'indeterminate' attribute gets munged in advanced modeDefectMajorClosedDeclinedUnassignedXavi CaballéCompilerbugMon, 19 Aug 2013 05:17:49 -0500Wed, 21 Aug 2013 15:18:38 -0500Wed, 21 Aug 2013 15:18:34 -050000<p>This issue should be addressed in the <a href="https://code.google.com/p/closure-compiler/issues/list">Closure Compiler bugtracker</a></p>
<p>Until they fix it, you can use your own externs file (<a href="http://lukevanderhart.com/2011/09/30/using-javascript-and-clojurescript.html">tutorial</a>), which is a perfectly valid approach.</p><p>ok, thanks.<br/>
I just submitted a patch to fix this in the Closure Compiler<br/>
<a href="https://code.google.com/p/closure-compiler/issues/detail?id=1068">https://code.google.com/p/closure-compiler/issues/detail?id=1068</a></p>Global Rank[CLJS-476] Reading a value from a module does not work if the module is def'edhttp://dev.clojure.org/jira/browse/CLJS-476
ClojureScript<p>Referring to a value in a module can have a scoping issue when using the "static accessor" operator of module/VALUE_NAME. The static accessor works if the module is loaded into a local value but not if def'ed. This example uses the mmap module for Node.js, and successfully reads the PROT_READ value:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns stat.core)
(defn -main []
(let [m (js/require <span class="code-quote">"mmap"</span>)]
(println <span class="code-quote">"value: "</span> m/PROT_READ)))
(set! *main-cli-fn* -main)</pre>
</div></div>
<p>This correctly prints "value: 1"</p>
<p>However, if the value for m is def'ed instead, then the compiler assumes that the reference to m is local to the function and therefore not defined:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns stat.core)
(def m (js/require <span class="code-quote">"mmap"</span>))
(defn -main []
(println <span class="code-quote">"value: "</span> m/PROT_READ))
(set! *main-cli-fn* -main)
/Users/pag/src/test/clj/stat/target/stat.js:12836
<span class="code-keyword">return</span> cljs.core.println.call(<span class="code-keyword">null</span>, <span class="code-quote">"value: "</span>, m.PROT_READ)
^
ReferenceError: m is not defined
at Function.stat.core._main (/Users/pag/src/test/clj/stat/target/stat.js:12836:50)
at cljs.core.apply.b (/Users/pag/src/test/clj/stat/target/stat.js:5621:14)
at cljs.core.apply.a (/Users/pag/src/test/clj/stat/target/stat.js:5656:18)
at <span class="code-object">Object</span>.&lt;anonymous&gt; (/Users/pag/src/test/clj/stat/target/stat.js:12844:17)
at Module._compile (module.js:449:26)
at <span class="code-object">Object</span>.Module._extensions..js (module.js:467:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.runMain (module.js:492:10)
at process.startup.processNextTick.process._tickCallback (node.js:244:9)</pre>
</div></div>
<p>In this case, the value of m.PROT_READ should have been stat.core.m.PROT_READ.</p>
<p>On the other hand, using . syntax fixes the scoping issue:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns stat.core)
(def m (js/require <span class="code-quote">"mmap"</span>))
(defn -main []
(println <span class="code-quote">"value: "</span> (.-PROT_READ m)))
(set! *main-cli-fn* -main)</pre>
</div></div>
Clojure 1.5.0-RC16
<br/>
Clojurescript 0.0-1586
<br/>
java version &quot;1.7.0_04&quot;
<br/>
Java(TM) SE Runtime Environment (build 1.7.0_04-b21)
<br/>
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)
<br/>
OSX Mountain Lion 10.8.2CLJS-476Reading a value from a module does not work if the module is def'edDefectMinorClosedDeclinedUnassignedPaul GearonCompilerbugscopeFri, 22 Feb 2013 13:13:40 -0600Tue, 19 Nov 2013 21:26:40 -0600Tue, 19 Nov 2013 21:26:38 -060002<p>the slash syntax is reserved for real CLJS namespaces for everything else the dot syntax must be used.</p>Global Rank[CLJS-475] Node.js target fails with optimizations set to :none or :whitespacehttp://dev.clojure.org/jira/browse/CLJS-475
ClojureScript<p>Compiling a hello world program for Node.js works fine if using optimizations of :advanced or :simple, but if using :none or :whitespace then an error will be reported for either "goog undefined" or "goog.string" undefined respectively.</p>
<p>The program is shown here:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns pr.core)
(defn -main []
(println <span class="code-quote">"Hello World!"</span>))
(set! *main-cli-fn* -main)</pre>
</div></div>
<p>This program is in src/cljs/pr/core.cljs. The repl line used to compile is:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(cljs.closure/build <span class="code-quote">"src/cljs"</span> {:output-to <span class="code-quote">"src/js/pr.js"</span> :target :nodejs :pretty-print <span class="code-keyword">true</span> :optimizations :none})</pre>
</div></div>
<p>When compiled with optimizations of :none, the output is:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">$ node src/js/pr.js
/Users/pag/src/test/clj/pr/src/js/pr.js:1
(function (exports, require, module, __filename, __dirname) { goog.addDependen
^
ReferenceError: goog is not defined
at <span class="code-object">Object</span>.&lt;anonymous&gt; (/Users/pag/src/test/clj/pr/src/js/pr.js:1:63)
at Module._compile (module.js:449:26)
at <span class="code-object">Object</span>.Module._extensions..js (module.js:467:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.runMain (module.js:492:10)
at process.startup.processNextTick.process._tickCallback (node.js:244:9)</pre>
</div></div>
<p>When running with optimizations of :whitespace the output is:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">$ node src/js/pr.js
/Users/pag/src/test/clj/pr/src/js/pr.js:493
goog.string.Unicode = {NBSP:<span class="code-quote">"\u00a0"</span>};
^
TypeError: Cannot set property 'Unicode' of undefined
at <span class="code-object">Object</span>.&lt;anonymous&gt; (/Users/pag/src/test/clj/pr/src/js/pr.js:493:21)
at Module._compile (module.js:449:26)
at <span class="code-object">Object</span>.Module._extensions..js (module.js:467:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.runMain (module.js:492:10)</pre>
</div></div>
<p>When running with optimizations of either :simple or :advanced, the output is:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">$ node src/js/pr.js
Hello World!</pre>
</div></div>
<p>I have included the two javascript output files that match the above errors.</p>Clojure 1.5.0-RC16
<br/>
Clojurescript 0.0-1586
<br/>
java version &quot;1.7.0_04&quot;
<br/>
Java(TM) SE Runtime Environment (build 1.7.0_04-b21)
<br/>
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)
<br/>
OSX Mountain Lion 10.8.2CLJS-475Node.js target fails with optimizations set to :none or :whitespaceDefectMinorClosedCompletedUnassignedPaul GearonCompilerbugThu, 21 Feb 2013 16:34:15 -0600Tue, 2 Dec 2014 06:21:00 -0600Tue, 2 Dec 2014 06:20:50 -060002<p>Remaining generated files</p><p>This is a known bug. We need goog.require/provide to actually mean something to Node.js. I'm not sure how this can be made to work. I've been hoping for a patch for this since ClojureScript was first announced, but I haven't seen anything yet.</p><p>long ago fixed in master.</p>Global Rank[CLJS-100] binding breaks in macroshttp://dev.clojure.org/jira/browse/CLJS-100
ClojureScript<p>When creating a macro that results in a binding the name of the global bound variables resolves into namespace bound names - this breaks the assert in clojurescripts let code.</p>
<p>example (simplified):<br/>
(ns clgl.macros)<br/>
(defmacro with-gl [<span class="error">&#91;gl prog&#93;</span> &amp; body]<br/>
`(binding <span class="error">&#91;*gl* ~gl *prog* ~brog&#93;</span> ~@body))</p>
<p>(ns clgl.core<br/>
(:require-macros <span class="error">&#91;clgl.macros :as m&#93;</span>))<br/>
...<br/>
(with-gl <span class="error">&#91;gl prog&#93;</span><br/>
(do-something))</p>unimportantCLJS-100binding breaks in macrosDefectMajorClosedDuplicateUnassignedHeinz N. GiesCompilerFri, 11 Nov 2011 19:21:09 -0600Sat, 27 Jul 2013 14:37:43 -0500Tue, 10 Jan 2012 23:20:59 -060001<p>I'm not sure I follow. What does <b>gl</b> and <b>prog</b> get namespaced to?</p><p>I think he has a problem with <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">*gl*</pre>
</div></div> resolving into <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">clgl.macros/*gl*</pre>
</div></div> when expanding macro, and this produces error in clojurescript (no such ns exists in cljs).</p>
<p>Solution is to use <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">~'*gl*</pre>
</div></div> instead of <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">*gl*</pre>
</div></div> inside macro.</p>
<p>So IMO this issue is not a CLJS defect and should be closed.</p><p><a href="http://dev.clojure.org/jira/browse/CLJS-114">http://dev.clojure.org/jira/browse/CLJS-114</a></p>Global Rank[CLJ-1659] compile leaks fileshttp://dev.clojure.org/jira/browse/CLJ-1659
Clojure<p>clojure's compile function leaks file descriptors, i.e. it relies on garbage collection to close the files. I'm trying to use boot <span class="error">&#91;1&#93;</span> on windows and ran into the problem, that files could not be deleted intermittently <span class="error">&#91;2&#93;</span>. The problem is that clojure's compile function, or rather clojure.lang.RT.lastModified relies on garbage collection to close files. lastModified looks like:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java"><span class="code-keyword">static</span> <span class="code-keyword">public</span> <span class="code-object">long</span> lastModified(URL url, <span class="code-object">String</span> libfile) <span class="code-keyword">throws</span> IOException{
<span class="code-keyword">if</span>(url.getProtocol().equals(<span class="code-quote">"jar"</span>)) {
<span class="code-keyword">return</span> ((JarURLConnection) url.openConnection()).getJarFile().getEntry(libfile).getTime();
}
<span class="code-keyword">else</span> {
<span class="code-keyword">return</span> url.openConnection().getLastModified();
}
}</pre>
</div></div>
<p>Here's the stacktrace from file leak detector <span class="error">&#91;3&#93;</span>:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">#205 C:\Users\ralf\.boot\tmp\Users\ralf\home\steinmetz\2mg\-x24pa9\steinmetz\fx\config.clj by thread:clojure-agent-send-off-pool-0 on Sat Feb 14 19:58:46 UTC 2015
at java.io.FileInputStream.(FileInputStream.java:139)
at java.io.FileInputStream.(FileInputStream.java:93)
at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
at sun.net.www.protocol.file.FileURLConnection.initializeHeaders(FileURLConnection.java:110)
at sun.net.www.protocol.file.FileURLConnection.getLastModified(FileURLConnection.java:178)
at clojure.lang.RT.lastModified(RT.java:390)
at clojure.lang.RT.load(RT.java:421)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5066.invoke(core.clj:5641)
at clojure.core$load.doInvoke(core.clj:5640)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5446)
at clojure.core$compile$fn__5071.invoke(core.clj:5652)
at clojure.core$compile.invoke(core.clj:5651)
at pod$eval52.invoke(NO_SOURCE_FILE:0)
at clojure.lang.<span class="code-object">Compiler</span>.eval(<span class="code-object">Compiler</span>.java:6703)
at clojure.lang.<span class="code-object">Compiler</span>.eval(<span class="code-object">Compiler</span>.java:6693)
at clojure.lang.<span class="code-object">Compiler</span>.eval(<span class="code-object">Compiler</span>.java:6666)
at clojure.core$eval.invoke(core.clj:2927)
at boot.pod$eval_in_STAR_.invoke(pod.clj:203)
at clojure.lang.Var.invoke(Var.java:379)
at org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke(ClojureRuntimeShimImpl.java:88)
at org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke(ClojureRuntimeShimImpl.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at boot.pod$eval_in_STAR_.invoke(pod.clj:206)
at boot.task.built_in$fn__1417$fn__1418$fn__1421$fn__1422.invoke(built_in.clj:433)
at boot.task.built_in$fn__1443$fn__1444$fn__1447$fn__1448.invoke(built_in.clj:446)
at boot.task.built_in$fn__1190$fn__1191$fn__1194$fn__1195.invoke(built_in.clj:232)
at boot.core$run_tasks.invoke(core.clj:663)
at clojure.lang.Var.invoke(Var.java:379)
at boot.user$eval297$fn__298.invoke(boot.user4212477544188689077.clj:33)
at clojure.core$binding_conveyor_fn$fn__4145.invoke(core.clj:1910)
at clojure.lang.AFn.call(AFn.java:18)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.<span class="code-object">Thread</span>.run(<span class="code-object">Thread</span>.java:745)</pre>
</div></div>
<p>So, it looks like getLastModified opens an InputStream internally. On Stackoverflow <span class="error">&#91;4&#93;</span> there's a discussion on how to close the URLConnection correctly.</p>
<p>On non-Windows operating systems this shouldn't be much of a problem. But on windows this hurts very much, since you can't delete<br/>
files that are opened by some process.</p>
<p>I've tested with clojure 1.6.0, but I assume other version are also affected.</p>
<p><span class="error">&#91;1&#93;</span> <a href="http://boot-clj.com/">http://boot-clj.com/</a><br/>
<span class="error">&#91;2&#93;</span> <a href="https://github.com/boot-clj/boot/issues/117">https://github.com/boot-clj/boot/issues/117</a><br/>
<span class="error">&#91;3&#93;</span> <a href="http://file-leak-detector.kohsuke.org/">http://file-leak-detector.kohsuke.org/</a><br/>
<span class="error">&#91;4&#93;</span> <a href="http://stackoverflow.com/questions/9150200/closing-urlconnection-and-inputstream-correctly">http://stackoverflow.com/questions/9150200/closing-urlconnection-and-inputstream-correctly</a></p>CLJ-1659compile leaks filesDefectMajorOpenUnresolvedUnassignedRalf SchmittcompilerMon, 16 Feb 2015 03:08:36 -0600Mon, 16 Feb 2015 07:56:52 -0600Release 1.601ApprovalTriagedGlobal Rank[CLJ-1648] Use equals() instead of == when resolving Symbolhttp://dev.clojure.org/jira/browse/CLJ-1648
Clojure<p>In Compiler.java, resolveSymbol() uses == to compare a Symbol's ns and the found namespace's name. This can result in a false comparison result, though the name's may be equal. In the following example:</p>
<p>ond.core=&gt; (require '<span class="error">&#91;clojure.string&#93;</span>) <br/>
nil <br/>
ond.core=&gt; `(clojure.string/join "," <span class="error">&#91;1 2&#93;</span>) <br/>
false : true<br/>
nil </p>
<p>The symbol resolution of clojure.string/join within the syntax quote is false by ==, but true by equals(). (The "false : true" above is reported from System.out.println code I had put into Compiler.java while testing.) The result is that a new Symbol is allocated, when the previous one should be returned.</p>
<p>As noted by Alex in this clojure-dev thread<span class="error">&#91;1&#93;</span>:</p>
<p>"Prior to Clojure 1.7, Symbol name and ns were interned so == would actually have worked, but that is no longer the case."</p>
<p>Attached is a patch resolve-symbol-equals.diff that modifies the comparison to use equals(). </p>
<p><span class="error">&#91;1&#93;</span> - <a href="https://groups.google.com/forum/#!topic/clojure-dev/58fYUSIEfxg">https://groups.google.com/forum/#!topic/clojure-dev/58fYUSIEfxg</a></p>CLJ-1648Use equals() instead of == when resolving SymbolDefectMinorOpenUnresolvedUnassignedSteven YiCompilerThu, 22 Jan 2015 15:43:07 -0600Fri, 23 Jan 2015 07:57:23 -0600Release 1.701ApprovalTriagedGlobal RankPatchCode[CLJ-1645] 'javap -v' on protocol class reveals no source filehttp://dev.clojure.org/jira/browse/CLJ-1645
Clojure<p>Through "javap -v" I can find source filename information in Clojure-generated datatype class files but not in protocol ones.</p>Mac OS X Yosemite.
<br/>
<br/>
java version &quot;1.8.0_25&quot;
<br/>
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
<br/>
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)CLJ-1645'javap -v' on protocol class reveals no source fileDefectMinorOpenUnresolvedUnassignedFabio TudonecompilerprotocolssourceFri, 16 Jan 2015 10:40:32 -0600Sun, 8 Feb 2015 10:00:01 -0600Release 1.601ApprovalTriagedGlobal RankPatchCode and Test[CLJ-1632] Mark Clojure-generated classes in order for instrumenters to identify them easilyhttp://dev.clojure.org/jira/browse/CLJ-1632
Clojure<p>Instrumenting logic specific to Clojure-generated classes should be able to identify them easily.</p>CLJ-1632Mark Clojure-generated classes in order for instrumenters to identify them easilyEnhancementMinorClosedDeclinedUnassignedFabio TudonecompilerMon, 5 Jan 2015 02:21:16 -0600Mon, 5 Jan 2015 14:03:42 -0600Mon, 5 Jan 2015 08:17:51 -0600Release 1.601<p>One way could be annotations, another could be interface-marking. Would that be feasible? Any drawbacks?</p><p>deftypes have a marker interface clojure.lang.IType.<br/>
defrecords have a marker interface clojure.lang.IRecord.<br/>
proxy classes have marker interface clojure.lang.IProxy.</p>
<p>I think generic markers for protocols or gen-interface would be undesirable as they may be used to create APIs for external use.</p><p>Not sure I understand your point about the non-marked (as of now and AFAIK) Clojure features such as protocols and gen-interface; could you elaborate? Does it apply to marking with annotations as well?</p><p>My point was that many people wish to generate interfaces that do not extend from interfaces in Clojure and adding those marker interfaces would be seen as a downside for them. Annotations are slightly better but have the same problem (dependencies on parts of Clojure core). You are of course free to add those interfaces or annotations yourself in your own code if that's useful to you!</p><p>It's clear now, thanks!</p>
<p>Actually my use case is about general tooling that will inspect and instrument all (and only) Clojure-generated code in any application making use of it, so I don't control the code I'm going to examine. This is done in order to add specific runtime features in a general fashion.</p>
<p>I can't find a way to do that reliably on everything that has been generated by Clojure; I could use some imperfect heuristics but I'd rather use a reliable way if one exists. Can you see of any other way of doing this I might have overlooked? Or is some other enhancement possible that would allow me to do this and would not compromise external integrations?</p><p>I don't see a general-purpose way to do this now. I do not believe supporting this is something we would spend time on.</p><p><a href="http://docs.oracle.com/javase/7/docs/api/java/lang/Class.html#isSynthetic()">Class#isSynthetic()</a> might be relevant here. If I am not mistaken asm generated classes will be flagged as synthetic.</p>Global Rank[CLJ-1620] Constants are leaked in case of a reentrant evalhttp://dev.clojure.org/jira/browse/CLJ-1620
Clojure<p>Compiling a function that references a non loaded (or unitialized) class triggers its init static. When the init static loads clojure code, some constants (source code I think) are leaked into the constants pool of the function under compilation.</p>
<p>It prevented CCW from working in some environments (Rational) because the static init of the resulting function was over 64K.</p>
<h4><a name="Stepstoreproduce%3A"></a>Steps to reproduce:</h4>
<p>Load the leak.main ns and run the code in comments: the first function has 15 extra fiels despite being identical to the second one.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns leak.main)
(defn first-to-load []
leak.Klass/foo)
(defn second-to-load []
leak.Klass/foo)
(comment
=&gt; (map (comp count #(.getFields %) class) [first-to-load second-to-load])
(16 1)
)</pre>
</div></div>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java"><span class="code-keyword">package</span> leak;
<span class="code-keyword">import</span> clojure.lang.IFn;
<span class="code-keyword">import</span> clojure.lang.RT;
<span class="code-keyword">import</span> clojure.lang.Symbol;
<span class="code-keyword">public</span> class Klass {
<span class="code-keyword">static</span> {
RT.<span class="code-keyword">var</span>(<span class="code-quote">"clojure.core"</span>, <span class="code-quote">"require"</span>).invoke(Symbol.intern(<span class="code-quote">"leak.leaky"</span>));
}
<span class="code-keyword">public</span> <span class="code-keyword">static</span> IFn foo = RT.<span class="code-keyword">var</span>(<span class="code-quote">"leak.leaky"</span>, <span class="code-quote">"foo"</span>);
}</pre>
</div></div>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns leak.leaky)
(defn foo
<span class="code-quote">"Some doc"</span>
[]
<span class="code-quote">"hello"</span>)
(def unrelated 42)</pre>
</div></div>
<p><a href="https://gist.github.com/cgrand/5dcb6fe5b269aecc6a5b#file-main-clj-L10">https://gist.github.com/cgrand/5dcb6fe5b269aecc6a5b#file-main-clj-L10</a></p>
<p><b>Patch</b>: 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1620" title="Constants are leaked in case of a reentrant eval">CLJ-1620</a>-avoid-constants-leak-in-static-initalizer-v4.patch</p>CLJ-1620Constants are leaked in case of a reentrant evalDefectCriticalOpenUnresolvedUnassignedChristophe GrandaotcompilerThu, 18 Dec 2014 15:50:47 -0600Wed, 7 Jan 2015 17:05:46 -0600Release 1.732<p>Patch from Nicola Mometto</p><p>Attached the same patch with a more informative better commit message</p><p>I'd like to thank Christophe and Alex for their invaluable help in understanding what was happening, formulating the right hypothesis and then finding a fix.</p>
<p>I would also mention that even if non IBM rational environments where not affected by the bug to the point were CCW would not work, they were still affected. For instance the class for a one-liner function wrapping an interop call weighs 700bytes once the patch is applied, when it weighed 90kbytes with current 1.6 or 1.7.</p><p>In CCW for the initial problematic function, the -v2 patch produces exactly the same bytecode as if the referenced class does not load any namespace in its static initializers.<br/>
That is, the patch is valid. I will test it live in the IBM Rational environment ASAP.</p><p>I confirm the patch fixes the issue detected initially in the IBM Rational environment</p><p>I have absolutely no idea why, but if I apply this patch, and the patch for <a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a> to master, and then try to build a war from this test project <a href="https://github.com/pdenhaan/extend-test">https://github.com/pdenhaan/extend-test</a> I get a scary-looking traceback:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">$ lein <span class="code-keyword">do</span> clean, war!
Exception in thread <span class="code-quote">"main"</span> java.lang.NoSuchFieldError: __thunk__0__, compiling:(route.clj:1:1)
at clojure.lang.<span class="code-object">Compiler</span>$InvokeExpr.eval(<span class="code-object">Compiler</span>.java:3606)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7299)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7289)
at clojure.lang.<span class="code-object">Compiler</span>.compile(<span class="code-object">Compiler</span>.java:7365)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
at clojure.core$load_lib.doInvoke(core.clj:5667)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5706)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$require.doInvoke(core.clj:5789)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at extend_test.core.handler$loading__5301__auto____66.invoke(handler.clj:1)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.<span class="code-object">Compiler</span>$InvokeExpr.eval(<span class="code-object">Compiler</span>.java:3601)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7299)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7289)
at clojure.lang.<span class="code-object">Compiler</span>.compile(<span class="code-object">Compiler</span>.java:7365)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
at clojure.core$load_lib.doInvoke(core.clj:5667)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5706)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$require.doInvoke(core.clj:5789)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at extend_test.core.servlet$loading__5301__auto____7.invoke(servlet.clj:1)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.<span class="code-object">Compiler</span>$InvokeExpr.eval(<span class="code-object">Compiler</span>.java:3601)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7299)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7289)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7289)
at clojure.lang.<span class="code-object">Compiler</span>.compile(<span class="code-object">Compiler</span>.java:7365)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$compile$fn__5420.invoke(core.clj:5834)
at clojure.core$compile.invoke(core.clj:5833)
at user$eval5.invoke(form-init180441230737245034.clj:1)
at clojure.lang.<span class="code-object">Compiler</span>.eval(<span class="code-object">Compiler</span>.java:6776)
at clojure.lang.<span class="code-object">Compiler</span>.eval(<span class="code-object">Compiler</span>.java:6765)
at clojure.lang.<span class="code-object">Compiler</span>.eval(<span class="code-object">Compiler</span>.java:6766)
at clojure.lang.<span class="code-object">Compiler</span>.load(<span class="code-object">Compiler</span>.java:7203)
at clojure.lang.<span class="code-object">Compiler</span>.loadFile(<span class="code-object">Compiler</span>.java:7159)
at clojure.main$load_script.invoke(main.clj:274)
at clojure.main$init_opt.invoke(main.clj:279)
at clojure.main$initialize.invoke(main.clj:307)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.doInvoke(main.clj:420)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.NoSuchFieldError: __thunk__0__
at instaparse.core__init.load(Unknown Source)
at instaparse.core__init.&lt;clinit&gt;(Unknown Source)
at java.lang.<span class="code-object">Class</span>.forName0(Native Method)
at java.lang.<span class="code-object">Class</span>.forName(<span class="code-object">Class</span>.java:344)
at clojure.lang.RT.loadClassForName(RT.java:2141)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
at clojure.core$load_lib.doInvoke(core.clj:5667)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5706)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$require.doInvoke(core.clj:5789)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at clout.core$loading__5301__auto____273.invoke(core.clj:1)
at clout.core__init.load(Unknown Source)
at clout.core__init.&lt;clinit&gt;(Unknown Source)
at java.lang.<span class="code-object">Class</span>.forName0(Native Method)
at java.lang.<span class="code-object">Class</span>.forName(<span class="code-object">Class</span>.java:344)
at clojure.lang.RT.loadClassForName(RT.java:2141)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
at clojure.core$load_lib.doInvoke(core.clj:5667)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5706)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$require.doInvoke(core.clj:5789)
at clojure.lang.RestFn.invoke(RestFn.java:482)
at compojure.core$loading__5301__auto____68.invoke(core.clj:1)
at compojure.core__init.load(Unknown Source)
at compojure.core__init.&lt;clinit&gt;(Unknown Source)
at java.lang.<span class="code-object">Class</span>.forName0(Native Method)
at java.lang.<span class="code-object">Class</span>.forName(<span class="code-object">Class</span>.java:344)
at clojure.lang.RT.loadClassForName(RT.java:2141)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
at clojure.core$load_lib.doInvoke(core.clj:5667)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5706)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$require.doInvoke(core.clj:5789)
at clojure.lang.RestFn.invoke(RestFn.java:457)
at compojure.route$loading__5301__auto____1508.invoke(route.clj:1)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.<span class="code-object">Compiler</span>$InvokeExpr.eval(<span class="code-object">Compiler</span>.java:3601)
... 75 more
Subprocess failed</pre>
</div></div><p><a href="https://github.com/MichaelBlume/clojure/tree/no-field">https://github.com/MichaelBlume/clojure/tree/no-field</a><br/>
<a href="https://github.com/MichaelBlume/extend-test/tree/no-field">https://github.com/MichaelBlume/extend-test/tree/no-field</a></p>
<p>mvn clean install in the one, lein ring uberwar in the other.</p><p>Michael, thanks for the report, I've tried investigating this a bit but the big amount of moving parts involved make it really hard to figure out why the combination of the two patches causes this issue.</p>
<p>A helpful minimal case would require no lein and no external dependencies, I'd appreciate some help in debugging this issue if anybody has time.</p><p>Ok, looks like the minimal case is</p>
<p>(ns foo (:require <span class="error">&#91;instaparse.core&#93;</span>))</p>
<p>(ns bar (:require <span class="error">&#91;foo&#93;</span>))</p>
<p>and then attempt to AOT-compile both foo and bar.</p>
<p>I don't yet know what's special about instaparse.core.</p><p>Well, not a minimal case, of course, but one without lein, at least.</p><p>ok, problem is instaparse's defclone macro, I've extracted it to a test repo</p>
<p><a href="https://github.com/MichaelBlume/thunk-fail">https://github.com/MichaelBlume/thunk-fail</a></p>
<p>lein do clean, compile will get you a failure, but the repo has no dependencies so I'm sure there's a way to do that without lein.</p><p>Sorry for the barrage of questions, but these classloader bugs are subtle (and close to being solved I hope). Your report is immensely valuable, and yet it will help to be even more specific. There are a cluster of these bugs &#8211; and keeping them laser-focused is key.</p>
<p>The minimal case to which you refer is the NoSuchFieldError?<br/>
How are is this being invoked this without lein?<br/>
What are you calling to AOT? (compile 'bar) ? <br/>
What is the classpath? When you invoke originally, is ./target/classes empty?<br/>
Does the problem go away with <a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>-7 applied?</p><p>I have tried and failed to replicate without leiningen. When I just run </p>
<p>java -Dclojure.compile.path=target -cp src:../clojure/target/clojure-1.7.0-aot-SNAPSHOT.jar clojure.lang.Compile thunk-fail.first thunk-fail.second</p>
<p>everything works fine.</p><p>The NoSuchFieldError is related to the keyword lookup sites.</p>
<p>Replacing defclone's body with<br/>
`(do (:foo {})) is enough to trigger it, with the same ns structure.</p><p>I have updated the patch for <a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a>, now the combination of the new patch + the patch from this ticket should not cause any exception.</p>
<p>That said, a bug in this patch still exists since while the patch for <a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a> had a bug, it was causing a perfectly valid (albeit hardly reproducible) compilation scenario so we should keep debugging this patch with the help of the bugged patch for <a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a>.</p>
<p>I guess the first thing to do is figure out what lein compile is doing differently than clojure.Compile</p><p>Also Ghadi is right, infact replacing the whole body of thunk-fail.core with (:foo {}) is enough.</p>
<p>It would seem like the issue is with AOT (re)compiling top-level keyword lookup sites, my guess is that for some reason this patch is preventing correct generation of the __init static initializer.</p><p>I still have absolutely no idea what lein compile is doing but I figured out the issue.<br/>
The updated patch binds (in eval) the appropriate vars only when already bounded.</p><p>Would it be worth using transients on the bindings map now?</p><p>Makes sense, updated the patch to use a transient map</p><p>Is there a test we can add that'll fail in the presence of the v2 patch? preferably independent of the <a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a> patch? I can try to write one myself, but I don't have a lot of familiarity with the Clojure compiler internals.</p><p>I'll have to think about a way to reproduce that bug, it's not a simple scenario to reproduce.<br/>
It involves compiling a namespace from an evaluated context.</p>ApprovalTriagedGlobal RankPatchCode[CLJ-1614] Clojure does not start: ClassCastExceptionhttp://dev.clojure.org/jira/browse/CLJ-1614
Clojure<p>The clojure.lang.Compiler class static code throws the ClassCastException when reading compiler options from System properties (Compiler.java, line 260 in the git master release). When running Clojure from Eclipse RCP application the System properties may have non-string values.</p>
<p>Checking if the value is String and ignoring non-strings fixes this problem.</p>Eclipse RCPCLJ-1614Clojure does not start: ClassCastExceptionDefectMinorOpenUnresolvedUnassignedVladimir TsichevskicompilerFri, 12 Dec 2014 11:03:19 -0600Fri, 12 Dec 2014 12:40:03 -0600Release 1.701Global Rank[CLJ-1604] AOT'ed code that defs a var with clojure.core symbol name causes IllegalStateExceptionhttp://dev.clojure.org/jira/browse/CLJ-1604
Clojure<p>AOT'ed code that defs a var that is also a symbol in clojure.core results in an exception at runtime. This problem can be avoided with (:refer-clojure :exclude ...) but this requires a library author to update and release a new version. AOT'ed applications must then wait for all transitive dependencies to update before they can update to a new Clojure version. For some users, this problem prevents them from trying or adopting new releases.</p>
<p>For example, the contrib library <tt>data.int-map</tt> defines an <tt>update</tt> function. clojure.core will also have a new <tt>update</tt> function as of 1.7.0. If this library is AOT'ed, then users of the clojure.data.int-map/update function will see the exception below. This situation can commonly occur when an application uses <tt>lein uberjar</tt> to compile all of the project+libs. In this case, applications or libraries that use data.int-map (either directly or indirectly) are affected.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">java.lang.IllegalStateException: Attempting to call unbound fn: #'clojure.data.<span class="code-object">int</span>-map/update
at clojure.lang.Var$Unbound.throwArity (Var.java:43)
clojure.lang.AFn.invoke (AFn.java:40)
compiler_update_not_referenced_bug.core$foo.invoke (core.clj:5)</pre>
</div></div>
<p>Reproduce with this sample project: <a href="https://github.com/yeller/compiler_update_not_referenced_bug">https://github.com/yeller/compiler_update_not_referenced_bug</a></p>
<p><b>Cause:</b> When AOT compiling a namespace, the def forms are hoisted into the ns_&#95;init class (in the example here, clojure.data.int_map__init). The static initializer in this class creates each var in the ns via a call to RT.var(ns, name). For data.int-map the static initializer will properly create the var for clojure.data.int-map/update. But when the ns is loaded (via the clojure.data.int_map.load() method), (refer-clojure) will be called, which will remap clojure.data.int-map/update to point to clojure.core/update.</p>
<p>This problem does not affect non-AOT loading (which doesn't use the ns_&#95;init class) and does not affect collisions from any other namespace. Only collisions from clojure.core create this possibility.</p>
<p><b>Proposed:</b> The proposed patch explicitly refers the Var during ns_&#95;init.load() (after Clojure symbols are referred) rather than implicitly during ns__init static {}. </p>
<p>This change in behavior only happens during AOT in the specific case where a core symbol is being shadowed. In that case, clojure.core has already been loaded and v (the looked up var) will have ns=clojure.core. The currentNS will be (for example) data.int-map. If that's the case, and the sym has no ns, then the new logic will be emitted. </p>
<p>In the case of clojure.core itself, NO new bytecode is emitted. From tests on several projects, only shadowed vars during AOT get this additional bytecode.</p>
<p><b>Patch:</b> 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu-v2.patch</p>
<p><b>Screened by:</b> Alex Miller</p>CLJ-1604AOT'ed code that defs a var with clojure.core symbol name causes IllegalStateExceptionDefectCriticalClosedCompletedUnassignedNicola MomettoaotcompilerTue, 25 Nov 2014 15:42:04 -0600Sat, 21 Feb 2015 12:28:32 -0600Fri, 20 Feb 2015 11:42:50 -0600Release 1.7Release 1.734<p>When I try latest Clojure master plus patch <a href="http://dev.clojure.org/jira/browse/CLJ-1604" title="AOT&#39;ed code that defs a var with clojure.core symbol name causes IllegalStateException"><del>CLJ-1604</del></a>-only-core.patch with the small test project created by Tom Crayford to demonstrate this issue: <a href="https://github.com/yeller/compiler_update_not_referenced_bug">https://github.com/yeller/compiler_update_not_referenced_bug</a></p>
<p>In that project, I get the same exception thrown when attempting 'lein do clean, uberjar, test' using this patch, as without it. It is because int-map/update in namespace compiler-update-not-referenced-bug.core is an unbound var.</p><p>Andy, you're right. For some reason I attached the wrong patch to the ticket, this is the correct one</p><p>I wasn't able to write a test for this, so here's a repl session using the clojure jar demonstrating this issue:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">[˷/test]&gt; ls
classes clojure.jar test.clj
[˷/test]&gt; cat test.clj
(in-ns 'test)
(clojure.core/refer 'clojure.core)
(def foo <span class="code-quote">"bar"</span>)
(def update <span class="code-quote">"foo"</span>)
[˷/test]&gt; java -cp classes:clojure.jar:. clojure.main
Clojure 1.7.0-master-SNAPSHOT
user=&gt; (binding [*compile-files* <span class="code-keyword">true</span>] (load <span class="code-quote">"test"</span>))
WARNING: update already refers to: #'clojure.core/update in namespace: test, being replaced by: #'test/update
nil
user=&gt; test/foo
<span class="code-quote">"bar"</span>
user=&gt; test/update
<span class="code-quote">"foo"</span>
user=&gt;
[˷/test]&gt; java -cp classes:clojure.jar:. clojure.main
Clojure 1.7.0-master-SNAPSHOT
user=&gt; (load <span class="code-quote">"test"</span>)
nil
user=&gt; test/foo
<span class="code-quote">"bar"</span>
user=&gt; test/update
CompilerException java.lang.RuntimeException: No such <span class="code-keyword">var</span>: test/update, compiling: (NO_SOURCE_PATH:0:0)
user=&gt;</pre>
</div></div><p>Thanks. I have not tried to assess the details of the change, other than to say that patch 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu.patch dated 26 Nov 2014, when applied to latest Clojure master as of today, enables both 'lein do clean, test' and 'lein do clean, uberjar, test' to work as expected with Tom Crayford's test project, linked above, whereas 'lein do clean, uberjar, test' fails without this patch, due to a var being unbound that should have a value.</p><p>Copying a comment here from <a href="http://dev.clojure.org/jira/browse/CLJ-1591" title="Symbol not being bound in namespace when name clashes with clojure.core">CLJ-1591</a>, since it is more appropriate here. It is responding to Tom Crayford's posting of his example project to demonstrate the issue: <a href="https://github.com/yeller/compiler_update_not_referenced_bug">https://github.com/yeller/compiler_update_not_referenced_bug</a></p>
<p>Tom, looked at your project. Thanks for that. It appears not to have anything like (def inc inc) in it. It throws exception during test step of 'lein do clean, uberjar, test' consistently for me, too, but compiles with only warnings and passes tests with 'lein do clean, test'. I have more test results showing in which Clojure versions these results change. To summarize, the changes to Clojure that appear to make the biggest difference in the results are below (these should be added to the new ticket you create – you are welcome to do so):</p>
<p>Clojure 1.6.0, 1.7.0-alpha1, and later changes up through the commit with description "<a href="http://dev.clojure.org/jira/browse/CLJ-1378" title="Hints don&#39;t work with #() form of function"><del>CLJ-1378</del></a>: Allows FnExpr to override its reported class with a type hint": No errors or warnings for either lein command above.</p>
<p>Next commit with description "Add clojure.core/update, like update-in but takes a single key" that adds clojure.core/update: 'lein do clean, test' is fine, but 'lein do clean, uberjar' throws exception during compilation, probably due to <a href="http://dev.clojure.org/jira/browse/CLJ-1241" title="NPE when AOTing overrided clojure.core functions"><del>CLJ-1241</del></a>.</p>
<p>Next commit with description "fix <a href="http://dev.clojure.org/jira/browse/CLJ-1241" title="NPE when AOTing overrided clojure.core functions"><del>CLJ-1241</del></a>": 'lein do clean, test' and 'lein do clean, uberjar' give warnings about clojure.core/update, but no errors or exceptions. 'lein do clean, uberjar, test' throws exception during test step that is same as the one I see with Clojure 1.7.0-alpha4. Debug prints of values of clojure.core/update and int-map/update (in data.int-map and in Tom's namespace compiler-update-not-referenced-bug.core) show things look fine when printed inside data.int-map, and in Tom's namespace when not doing the uberjar, but when doing the uberjar, test, int-map/update is unbound in Tom's namespace.</p>
<p>In case it makes a difference, my testing was done with Mac OS X 10.9.5, Leiningen 2.5.0 on Java 1.7.0_45 Java HotSpot(TM) 64-Bit Server VM</p><p>The updated patch only emits the interning bytecode when necessary, avoiding the emission when a clojure.core var with the same name exists but is not mapped to the current namespace</p><p>Attached 1604-context.diff for purely informational purposes - same diff just more context in it for easier reading.</p><p>Thought I'd add a minor note in here to say I tried testing this patch out on my app (which is where I discovered this AOT bug), and the bug doesn't turn up with this patch applied to clojure (tested by applying 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu-v2.patch to 1.7-alpha5)</p><p>I ran into this issue with Korma 0.4.0. I'm still running into it, but there is a twist.</p>
<p>My project depends on an artifact that was built with Clojure 1.7.0-alpha1. If I remove this dependency, everything is fine. However, with this dependency, I run into this issue, even if I declare a dependency on 1.7.0-master-SNAPSHOT in my project and exclude any dependency on clojure-1.7.0-alpha1.</p>
<p>I'm not sure if this is a Maven issue or a Clojure issue. Running Maven with debug on seems to show that it's using the correct version of Clojure.</p>
<p>I have created a dummy project that reproduces this issue if you are interested.</p>
<p><a href="https://github.com/deaddowney/UpdateProblem">https://github.com/deaddowney/UpdateProblem</a></p>
<p>Check it out, run "mvn install", and you will get <br/>
java.lang.RuntimeException: No such var: korma.core/update<br/>
.</p>
ApprovalOkGlobal RankPatchCode[CLJ-1598] Make if forms compile directly to the appropriate branch expression if the test is a literalhttp://dev.clojure.org/jira/browse/CLJ-1598
Clojure<p>This allows expressions like `(cond (some-expr) 1 :else 2)` to be eligible for unboxed use, which is not currently possible since the cond macro always ends up with a nil else branch that the compiler currently takes into account.</p>
<p>With the attached patch, the trailing (if :else 2 nil) in the macroexpansion will be treated as 2 by the Compiler, thus allowing the unboxed usage of the cond expression.</p>CLJ-1598Make if forms compile directly to the appropriate branch expression if the test is a literalEnhancementMinorOpenUnresolvedUnassignedNicola MomettocompilerperformanceprimitivesMon, 24 Nov 2014 13:44:20 -0600Mon, 24 Nov 2014 13:50:23 -060011ApprovalTriagedGlobal Rank[CLJ-1596] Using keywords in place of symbols for defrecord fields causes a compiler exception with incorrect line numberhttp://dev.clojure.org/jira/browse/CLJ-1596
Clojure<p>Possibly related to <a href="http://dev.clojure.org/jira/browse/CLJ-1261:">http://dev.clojure.org/jira/browse/CLJ-1261:</a> a defrecord like</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(defn foo [x])
(defrecord Bar [:b])</pre>
</div></div>
<p>Throws an exception, like you'd expect:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">java.lang.ClassCastException: clojure.lang.Keyword cannot be <span class="code-keyword">cast</span> to clojure.lang.IObj, compiling:(tesser/quantiles_test.clj:45:15)</pre>
</div></div>
<p>However, this exception's line and character indicates the error is in the <b>previous</b> form: the defn, not the defrecord. This can be really tricky to figure out when the expressions are more complicated. <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/wink.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p>CLJ-1596Using keywords in place of symbols for defrecord fields causes a compiler exception with incorrect line numberDefectMinorOpenUnresolvedUnassignedKyle KingsburycompilerdefrecordThu, 20 Nov 2014 16:12:40 -0600Thu, 20 Nov 2014 16:23:16 -0600Release 1.600<p>Related: <a href="http://dev.clojure.org/jira/browse/CLJ-1261" title="Invalid defrecord results in exception attributed to namespace that imports namespace with defrecord"><del>CLJ-1261</del></a> </p><p>Possibly fixed by <a href="http://dev.clojure.org/jira/browse/CLJ-1561" title="Incorrect line numbers are emitted">CLJ-1561</a>, not sure.</p>ApprovalTriagedGlobal Rank[CLJ-1593] Use PAM for small maps when assigned to a var rather than always using PHMshttp://dev.clojure.org/jira/browse/CLJ-1593
Clojure<p>I'm reproposing the fix I implemented for <a href="http://dev.clojure.org/jira/browse/CLJ-944">http://dev.clojure.org/jira/browse/CLJ-944</a> a while ago as an enhancement rather than as a defect.</p>
<p>Currently when a map is used as the value of a `def` expression, unless it's an empty map, it will always be a PersistentHashMap even if it's a small map.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def a {:foo :bar})
#'user/a
user=&gt; (class a)
clojure.lang.PersistentHashMap</pre>
</div></div>
<p>The current patch makes makes small maps be compiled to PAMs, consistently with how it's handled in lexical contexts, only using PHMs when the number of elements is above the threshold</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def a {:foo :bar})
#'user/a
user=&gt; (class a)
clojure.lang.PersistentArrayMap
user=&gt; (class (let [a {:foo :bar}] a))
clojure.lang.PersistentArrayMap
user=&gt; (def a {0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9})
#'user/a
user=&gt; (class a)
clojure.lang.PersistentHashMap</pre>
</div></div>
CLJ-1593Use PAM for small maps when assigned to a var rather than always using PHMsEnhancementMinorOpenUnresolvedUnassignedNicola MomettocollectionscompilermapsSat, 15 Nov 2014 12:14:29 -0600Mon, 8 Dec 2014 09:47:00 -060000<p>This might be subsumed under the small collections <a href="http://dev.clojure.org/jira/browse/CLJ-1517" title="Unrolled small vectors">CLJ-1517</a>, not sure. </p><p>This is now out of scope for <a href="http://dev.clojure.org/jira/browse/CLJ-1517" title="Unrolled small vectors">CLJ-1517</a> now that's focused only on vectors.</p><p>We're just splitting the ticket apart, maps will be a separate ticket/patch.</p>Global RankPatchCode[CLJ-1586] Compiler doesn't preserve metadata for LazySeq literalshttp://dev.clojure.org/jira/browse/CLJ-1586
Clojure<p>The analyzer in Compiler.java forces evaluation of lazyseq literals, but loses the compile time original metadata of that form, meaning that a type hint will be lost.</p>
<p>Example demonstrating this issue:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (set! *warn-on-reflection* <span class="code-keyword">true</span>)
<span class="code-keyword">true</span>
user=&gt; (list '.hashCode (with-meta (concat '(identity) '(<span class="code-quote">"foo"</span>)) {:tag '<span class="code-object">String</span>}))
(.hashCode (identity <span class="code-quote">"foo"</span>))
user=&gt; (eval (list '.hashCode (with-meta (concat '(identity) '(<span class="code-quote">"foo"</span>)) {:tag '<span class="code-object">String</span>})))
Reflection warning, NO_SOURCE_PATH:1:1 - reference to field hashCode can't be resolved.
101574</pre>
</div></div>
<p>Forcing the concat call to an ASeq rather than a LazySeq fixes this issue:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (eval (list '.hashCode (with-meta (seq (concat '(identity) '(<span class="code-quote">"foo"</span>))) {:tag '<span class="code-object">String</span>})))
101574</pre>
</div></div>
<p>This ticket blocks <a href="http://dev.clojure.org/jira/browse/CLJ-1444">http://dev.clojure.org/jira/browse/CLJ-1444</a> since clojure.core/sequence might return a lazyseq.</p>
<p>This bug affected both tools.analyzer and tools.reader and forced me to commit a fix in tools.reader to work around this issue, see: <a href="http://dev.clojure.org/jira/browse/TANAL-99">http://dev.clojure.org/jira/browse/TANAL-99</a></p>
<p>The proposed patch trivially preserves the form metadata after realizing the lazyseq</p>CLJ-1586Compiler doesn't preserve metadata for LazySeq literalsDefectMajorOpenUnresolvedUnassignedNicola MomettocompilermetadatatypehintsWed, 12 Nov 2014 10:01:16 -0600Wed, 12 Nov 2014 11:16:20 -060000ApprovalTriagedGlobal RankPatchCode[CLJ-1573] Support (Java) transient fields in deftype, e.g. for hashcodeshttp://dev.clojure.org/jira/browse/CLJ-1573
Clojure<p>Enhance deftypes to allow fields to be marked ACC_TRANSIENT.</p>
<p>strawman syntax:<br/>
(deftype AType <span class="error">&#91;^:transient hash&#93;</span>)</p>
<p>Came across this need while experimenting with a <a href="https://github.com/ghadishayban/clojure/commit/906cd6ed4d7c624dc4e553373dabfd57550eeff2">reified range written in a deftype</a>, not in Java.</p>
<p>Patch doesn't include docstring change, but has a test.</p>CLJ-1573Support (Java) transient fields in deftype, e.g. for hashcodesEnhancementMinorOpenUnresolvedUnassignedGhadi ShaybancompilerdeftypeSun, 26 Oct 2014 21:04:02 -0500Mon, 29 Dec 2014 12:25:53 -060042<p>Perhaps ^:transient-mutable would be a more appropriate modifier name to be consistent with the ^:unsynchronized-mutable and ^:volatile-mutable field modifiers. In any event, this feature would eliminate the need to drop down to Java for types that require transient fields. </p><p>Roberto, there is a "Vote" word you can click on to actually vote for tickets, and ticket wranglers actually look at those votes at times to examine popular ones sooner. +1 comments don't do that.</p>Global RankPatchCode and Test[CLJ-1529] Significantly improve compile time by reducing calls to Class.forNamehttp://dev.clojure.org/jira/browse/CLJ-1529
Clojure<p>Compilation speed has been a real problem for a number of my projects, especially Aleph <span class="error">&#91;1&#93;</span>, which in 1.6 takes 18 seconds to load. Recently I realized that Class.forName is being called repeatedly on symbols which are lexically bound. Hits on Class.forName are cached, but misses are allowed to go through each time, which translates into tens of thousands of calls after calling `(use 'aleph.http)`.</p>
<p><b>Proposed:</b> Avoid calling Class.forName() on non-namespaced symbols that do not contain "." or start with "[", don't map to a Class in the ns, and <b>are</b> names in the current local env. Also, adjust the ordering of checks in tagToClass to check for hints before checking for class. </p>
<p><span class="error">&#91;Note that the latest variant of the patch moves the check from the original patch farther down in the logic to avoid changing the semantics. This still yields virtually all of the performance gains. See comments for details.&#93;</span></p>
<p><b>Patch:</b> clj-1529-no-cache-2.diff</p>
<p><b>Screened by:</b> Stu Halloway. Note that for this change the patch ended up being so small it is easier follow the code than the prose description.</p>
<p><span class="error">&#91;1&#93;</span> <a href="https://github.com/ztellman/aleph">https://github.com/ztellman/aleph</a></p>CLJ-1529Significantly improve compile time by reducing calls to Class.forNameEnhancementCriticalClosedCompletedUnassignedZach TellmancompilerperformanceSun, 21 Sep 2014 14:08:08 -0500Fri, 14 Nov 2014 10:08:16 -0600Fri, 14 Nov 2014 10:08:16 -0600Release 1.7Release 1.7289<p>One of our larger projects (not macro-laden) just went from 36 seconds to 23 seconds to start with this patch.</p><p>I <a href="https://github.com/clojure-unity/clojure-clr/commit/db1dabf57855cdc4524a8b35b07c20a7d481ca09">ported this patch</a> to Clojure-CLR for the Unity integration project and we have seen significant speedups as well. I too agree that this is the behavior I expect as a user.</p><p>I ran this on a variety of open-source projects. I didn't find that it produced any unexpected behavior or test errors. Most projects were about 10% faster to run equivalent of "lein test" with a few as high as 35% faster. </p><p>We're interested in comparing this and the class caching in fastload branch to get something in for 1.7. Next step is to extract a patch of the stuff in fastload so we can compare them better.</p><p>Add maybe class cache patch from fastload branch</p><p>Times below to run "time lein test" on a variety of projects with columns:</p>
<ul class="alternate" type="square">
<li>master = current 1.7.0 master</li>
<li>maybe-cache = maybe-class-cache.patch extracted from Rich's fastload branch</li>
<li>class-for-name = class-for-name.diff from Zach</li>
<li>% maybe-cache = % improvement for maybe-cache over master</li>
<li>% class-for-name = % improvement for class-for-name patch over master (sorted desc)</li>
</ul>
<p>project,master,maybe-cache,class-for-name,% maybe-cache,% class-for-name<br/>
aleph,25.605,16.572,14.460,35.278,43.527<br/>
riemann,40.550,27.656,24.734,31.798,39.004<br/>
lamina,37.247,30.072,29.045,19.263,22.021<br/>
trapperkeeper,11.979,11.158,10.3,6.854,14.016<br/>
plumbing,73.777,68.388,66.922,7.304,9.292<br/>
cheshire,5.583,5.089,5.086,8.848,8.902<br/>
tools.analyzer,5.411,5.289,5.023,2.255,7.171<br/>
core.async,19.161,18.090,17.942,5.589,6.362<br/>
tools.reader,4.686,4.435,4.401,5.356,6.082<br/>
clara-rules,43.964,42.140,41.542,4.149,5.509<br/>
core.typed,158.885,154.954,151.445,2.474,4.683<br/>
instaparse,9.286,8.922,8.859,3.920,4.598<br/>
schema,45.3,43.914,43.498,3.060,3.978<br/>
mandoline,76.295,74.831,74.425,1.919,2.451</p>
<p>The summary is that both patches improve times on all projects. In most cases, the improvement from either is &lt;10% but the first few projects have greater improvements. The class-for-name patch has a bigger improvement in all projects than the maybe-cache patch (but maybe-cache has no change in semantics). </p>
<p>Are the two patches mutually exclusive? </p><p>They are non-over-lapping. I have not considered whether they could both be applied or whether that makes any sense.</p><p>The two patches both essentially cut off the same hot code path, just at different points (class-for-name is earlier), so applying them both effectively should give you about the performance of class-for-name.</p><p>Added a picture of the data for easier consumption.</p><p>One of our bigger projects saw a reduction of startup time of 16% with class-for-name, 14% with maybe-cache, and a whopping 23% with both patches applied. This was actually starting up the program, as opposed to running "lein test", FWIW.</p>
<p>Maybe it's worth re-running the benchmarks with a "both-patches" variant?</p><p>Hey Deepak, I did actually run some of them with both patches and saw times similar to class-for-name. </p>
<p>Were your times consistent across restarts? The times in the data above are the best of 3 trials for every data point (although they were relatively consistent).</p><p>Hi Alex, the tests I ran did 20-iteration loops, and I took the mean (though it was pretty consistent between restarts). I can redo stuff and upload the raw data for you if that will help.</p><p>So repeating the experiment several times does in fact behave as you suspected...apologies for my previous LOLDATA. <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p><p>maybe-class-cache-2.patch removes some debugging stuff</p><p>I've done more testing and made mods to both patches and moved them closer together. </p>
<p>On the maybe-class-cache patch (new version = clj-1529-with-cache.diff):<br/>
1) I found that adding a final else branch that returned null was an improvement - this avoids caching things that will never hit in the future (Cons, PersistentList, Symbols with namespaces, etc). That's both a perf improvement (avoids hashing those things) and a memory improvement.<br/>
2) The tagToClass improvement from Zach's patch is orthogonal and also valid here so I added it.<br/>
3) I added Zach's check, but moved the placement lower so that it doesn't alter semantics. It helps avoid caching locals that aren't classes.</p>
<p>On the class-for-name patch (new version = clj-1529-no-cache.diff):<br/>
1) Same change as #3 above - moved check lower to avoid semantics change.</p>
<p>With these changes, both patches have tagToClass and local checks, neither patch changes semantics, and the only difference is whether to keep or remove the cache.</p>
<p>aleph timings (for "lein test"):</p>
<ul class="alternate" type="square">
<li>1.7 master = 25.415 s</li>
<li>1.7 + clj-1529-with-cache.diff = 14.329 s</li>
<li>1.7 + clj-1529-no-cache.diff = 14.808 s</li>
</ul>
<p>lamina timings (for "lein test"):</p>
<ul class="alternate" type="square">
<li>1.7 master = 37.340 s</li>
<li>1.7 + clj-1529-with-cache.diff = 28.680 s</li>
<li>1.7 + clj-1529-no-cache.diff = 28.759 s</li>
</ul>
<p>The cache helps slightly in both cases, but it does not seem worth adding the new dynamic var and unbounded memory use inherent in the cache.</p><p>Talked to Rich, focusing on no-cache patch. Added new version that fixes tabbing and restores Zach's name to the patch, which seems appropriate.</p>ApprovalOkGlobal RankPatchCode[CLJ-1516] Throw an exception if def name contains a dothttp://dev.clojure.org/jira/browse/CLJ-1516
Clojure<p>In this comment: <a href="http://dev.clojure.org/jira/browse/CLJ-1100?focusedCommentId=35510&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-35510">http://dev.clojure.org/jira/browse/CLJ-1100?focusedCommentId=35510&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-35510</a> Rich said that Vars whose name contains a dot are not supported, but the current implementation allows their definition.<br/>
This patch makes `(def foo.bar)` throw a compile-time exception</p>CLJ-1516Throw an exception if def name contains a dotEnhancementMinorOpenUnresolvedUnassignedNicola MomettocompilerFri, 29 Aug 2014 09:26:54 -0500Fri, 29 Aug 2014 11:03:36 -0500Release 1.601<p>I'm curious whether this breaks existing code in the wild.</p><p>I find this hard to believe given the current behaviour:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def a.b 1)
#'user/a.b
user=&gt; a.b
CompilerException java.lang.ClassNotFoundException: a.b, compiling:(NO_SOURCE_PATH:0:0)</pre>
</div></div>
<p>one would need to go out of his way and refer to the var namespace qualified everywhere to make it work</p><p>After a brief conversation on #clojure, I updated the patch to only throw on non-macro defs so that macros like clojure.core/.. and clojure.core.incubator/.?. will work fine</p>ApprovalTriagedGlobal RankPatchCode[CLJ-1492] PersistentQueue objects are improperly eval'd and compiledhttp://dev.clojure.org/jira/browse/CLJ-1492
Clojure<p>PersistentQueue objects do not follow the correct evaluation path in the Compiler.</p>
<p>The simplest case:</p>
<p>user=&gt; (def q (conj clojure.lang.PersistentQueue/EMPTY 1 2 3))<br/>
#'user/q<br/>
user=&gt; q<br/>
#&lt;PersistentQueue clojure.lang.PersistentQueue@7861&gt;<br/>
user=&gt; (eval q)<br/>
CompilerException java.lang.ClassCastException: clojure.lang.PersistentQueue cannot be cast to java.util.List, compiling:(NO_SOURCE_PATH:4:1)</p>
<p>And you get the same exception when embedding a PersistentQueue:</p>
<p>user=&gt; (eval `(fn [] ~q))<br/>
CompilerException java.lang.ClassCastException: clojure.lang.PersistentQueue cannot be cast to java.util.List, compiling:(NO_SOURCE_PATH:2:1)</p>
<p>Instead of the expected:</p>
<p>CompilerException java.lang.RuntimeException: Can't embed unreadable object in code: #&lt;PersistentQueue clojure.lang.PersistentQueue@7861&gt;, compiling:(NO_SOURCE_PATH:3:1)</p>
<p>Since PersistentQueue implements IPersistentCollection and IPersistentList, and is not called out explicitly in the compiler, it is falling into the same compile path as a list. The exception comes from the call to emitValue inside the emitConstants portion of the FnExpr emit path. PersistentQueue does not implement java.util.List and thus the cast in emitListAsObjectArray (Compiler.java:4479) throws. Implementing List would NOT, however, resolve this issue, but would mask it by causing all eval'd PersistedQueues to be compiled as PersistentLists.</p>
<p>The first case is resolved by adding `&amp;&amp; !(form instanceof PersistentQueue)` to the IPersistentCollection branch of Compiler.eval() (Compiler.java:6695-8), allowing the PersistentQueue to fall through to the ConstantExpr case in analyze (Compiler.java:6459). The embedding case is resolved by adding `&amp;&amp; !(value instanceof PersistentQueue)` to the IPersistentList branch in ObjExpr's emitValue (Compiler.java:4639).</p>
<p>This bug also precludes definition of data-readers for PersistentQueue as the read object throws an exception when it is passed to the Compiler.</p>
<p>The attached patch includes the two changes mentioned above, and tests for each case that illustrates the bug.</p>
<p>Clojure-dev thread: <a href="https://groups.google.com/forum/#!topic/clojure-dev/LDUQfqjFg9w">https://groups.google.com/forum/#!topic/clojure-dev/LDUQfqjFg9w</a></p>OS X 10.9.4
<br/>
java version &quot;1.7.0_60&quot;
<br/>
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
<br/>
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)CLJ-1492PersistentQueue objects are improperly eval'd and compiledDefectMinorOpenUnresolvedUnassignedJon DistadcompilerWed, 6 Aug 2014 16:40:11 -0500Thu, 7 Aug 2014 08:40:25 -0500Release 1.6Release 1.702ApprovalTriagedGlobal RankPatchCode and Test[CLJ-1491] External type hint inconsistency between regular functions and primitive functionshttp://dev.clojure.org/jira/browse/CLJ-1491
Clojure<p>Consider the following example.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(set! *warn-on-reflection* <span class="code-keyword">true</span>)
(defn f [n] (java.util.ArrayList. (<span class="code-object">int</span> n)))
(let [al ^java.util.ArrayList (f 10)]
(.add al 23))</pre>
</div></div>
<p>As expected this does not warn about reflection. The following example shows the same scenario for a primitive function.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(set! *warn-on-reflection* <span class="code-keyword">true</span>)
(defn g [^<span class="code-object">long</span> n] (java.util.ArrayList. n))
(let [al ^java.util.ArrayList (g 10)]
(.add al 23))
; Reflection warning, NO_SOURCE_PATH:2:3 - call to method add on java.lang.<span class="code-object">Object</span> can't be resolved (no such method).</pre>
</div></div>
<p>So the behavior of external type hints is inconsistent for regular functions and primitive functions.<br/>
Most likely, the external type hint information is somehow ignored for primitive functions since the case where they return no primitive value is not treated separately.</p>CLJ-1491External type hint inconsistency between regular functions and primitive functionsDefectMinorClosedDuplicateUnassignedGunnar VölkelcompilertypehintsTue, 5 Aug 2014 04:08:52 -0500Thu, 25 Sep 2014 10:31:02 -0500Thu, 25 Sep 2014 10:31:02 -0500Release 1.600<p>The following patch preserves the original metadata of the invoke form on the transformed .invokePrim expression</p><p>Not challenging the premise at all but workaround:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(let [^java.util.ArrayList al (g 10)]
(.add al 23))</pre>
</div></div> <p>Well, the example above was already changed such that you can also place the type hint on the binding to check whether that works.<br/>
The actual problem arose when using the return value of the function exactly once without an additional binding.</p><p>Responding to Alex's comment, is there a consensus on which variant is (more) idiomatic? IMHO latter variant seems to be more reliable (as this issue shows, and for primitive hits too), and is consistent with 'place hint on a symbol' idiom which is applied when type hinting vars or fn args.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(let [symbol ^typehint expr] body)</pre>
</div></div>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(let [^typehint symbol expr] body)</pre>
</div></div><p>They have different meanings. Generally the latter covers some cases that the former does not so it's probably the better one. I believe one of the cases is that if expr is a macro, the typehint is lost in the former. </p><p>The patch for <a href="http://dev.clojure.org/jira/browse/CLJ-1533?jwupdated=61127&amp;focusedCommentId=35814">http://dev.clojure.org/jira/browse/CLJ-1533?jwupdated=61127&amp;focusedCommentId=35814</a> fixes this issue and more and should be preferred over this</p><p>Dupe of <a href="http://dev.clojure.org/jira/browse/CLJ-1533" title="Oddity in type tag usage for primInvoke">CLJ-1533</a></p>ApprovalTriagedGlobal RankPatchCode[CLJ-1475] :post condition causes compiler error with recurhttp://dev.clojure.org/jira/browse/CLJ-1475
Clojure<p>Michael O'Keefe &lt;michael.p.okeefe@gmail.com&gt; posted on the mailing list an example of code that causes a compiler error only if a :post condition is added. Here's my slightly modified version:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(defn g
[xs acc]
{:pre [(or (nil? xs) (sequential? xs))]
:post [(number? %)]}
(<span class="code-keyword">if</span> (seq xs)
(recur (next xs) (+ (first xs) acc))
acc))</pre>
</div></div>
<p>CompilerException java.lang.UnsupportedOperationException: Can only recur from tail position</p>
<p>The work-around is to wrap the body in a loop that simply rebinds the original args.</p>CLJ-1475:post condition causes compiler error with recurDefectMinorOpenUnresolvedUnassignedSteve MinercompilerFri, 25 Jul 2014 09:45:18 -0500Tue, 29 Jul 2014 01:35:16 -0500Release 1.604<p>A macro expansion shows that body is placed in a let form to capture the result for later testing with the post condition, but the recur no longer has a proper target. The work-around of using a loop form is easy once you understand what's happening but it's a surprising limitation.</p><p>Use a local fn* around the body and call it with the original args so that the recur has a proper target. Update: not good enough for handling destructuring. Patch withdrawn.</p><p>Link to the original topic discussion: <a href="https://groups.google.com/d/topic/clojure/Wb1Nub6wVUw/discussion">https://groups.google.com/d/topic/clojure/Wb1Nub6wVUw/discussion</a></p><p>Patch withdrawn because it breaks on destructured args.</p><p>While working on a patch, I came up against a related issue: Should the :pre conditions apply to every recur "call". Originally, I thought the :pre conditions should be checked just once on the initial function call and never during a recur. People on the mailing list pointed out that the recur is semantically like calling the function again so the :pre checks are part of the contract. But no one seemed to want the :post check on every recursion, so the :post would happen only at the end.</p>
<p>That means automatically wrapping a loop (or nested fn* call) around the body is not going to work for the :pre conditions. A fix would have to bring the :pre conditions inside the loop.</p><p>I'm giving up on this bug. My approach was adding too much complexity to handle an edge case. I recommend the "loop" work-around to anyone who runs into this problem.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(defn g2
[xs acc]
{:pre [(or (nil? xs) (sequential? xs))]
:post [(number? %)]}
(loop [xs xs acc acc]
(<span class="code-keyword">if</span> (seq xs)
(recur (next xs) (+ (first xs) acc))
acc)))</pre>
</div></div><p>Add patch that handles rest arguments and destructuring.</p><p>With regard to Steve's question on interpreting :pre, to me I would expect g to act like the case g3 below which uses explicit recursion (which does work and does appear to check the :pre conditions each time and :post condition once):</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(defn g3
[xs acc]
{:pre [(or (sequential? xs) (nil? xs)) (number? acc)]
:post [(number? %)]}
(<span class="code-keyword">if</span> (seq xs)
(g3 (next xs) (+ (first xs) acc))
acc))</pre>
</div></div><p>Patch clj-1475.diff handles destructuring, preconditions and rest arguments</p><p>The clj-1475.diff patch looks good to me.</p><p>Please don't use "patch" as a label - that is the purpose of the Patch field. There is a list of good and bad labels at <a href="http://dev.clojure.org/display/community/Creating+Tickets">http://dev.clojure.org/display/community/Creating+Tickets</a></p><p>More knowledgeable commenters might take a look at <a href="http://dev.clojure.org/jira/browse/CLJ-701" title="Compiler loses &#39;loop&#39;s return type in some cases">CLJ-701</a> just in case that's applicable to the proposed patch.</p><p>re clj-701</p>
<p>it is tricky to express loop expression semantics in jvm byte code, so the compiler sort of punts, hoisting expression loops in to anonymous functions that are immediately invoked, closing over whatever is in scope that is required by the loop, this has some problems like those seen in <a href="http://dev.clojure.org/jira/browse/CLJ-701" title="Compiler loses &#39;loop&#39;s return type in some cases">CLJ-701</a>, losing type data which the clojure compiler doesn't track across functions, the additional allocation of function objects (the jit may deal with that pretty well, I am not sure) etc.</p>
<p>where the world of clj-701 and this ticket collide is the patch on this ticket lifts the function body out as a loop expression, which without the patch in clj-701 will have the issues I listed above, but we already have those issues anywhere something that is difficult to express in bytecode as an expression (try and loop) is used as an expression, maybe it doesn't matter, or maybe clj-701 will get fixed in some way to alleviate those issues.</p>
<p>general musings</p>
<p>it seems like one feature people like from asserts is the ability to disable them in production (I have never actually seen someone do that with clojure), assert and :pre/:post have some ability to do that (it may only work at macroexpansion time, I don't recall) since the hoisting of the loop could impact performance it might be nice to have some mechanism to disable it (maybe using the same flag assert does?).</p>ApprovalTriagedGlobal RankPatchCode and Test[CLJ-1463] Providing own ClassLoader for eval is brokenhttp://dev.clojure.org/jira/browse/CLJ-1463
Clojure<p>clojure.lang.Compiler has a method with the signature</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java"><span class="code-keyword">public</span> <span class="code-keyword">static</span> <span class="code-object">Object</span> eval(<span class="code-object">Object</span> form, <span class="code-object">boolean</span> freshLoader)</pre>
</div></div>
<p>but the freshLoader argument is ignored since <a href="https://github.com/clojure/clojure/commit/2c2ed386ed0f6f875342721bdaace908e298c7f3">https://github.com/clojure/clojure/commit/2c2ed386ed0f6f875342721bdaace908e298c7f3</a></p>
<p>Is there a good reason this still needs to be "hotfixed" like this?</p>
<p>We would like to provide our own ClassLoader for eval to manage the lifecycle of the generated classes.</p>Clojure 1.6.0CLJ-1463Providing own ClassLoader for eval is brokenDefectMinorOpenUnresolvedUnassignedVolkert Oakley JurgenscompilerThu, 10 Jul 2014 18:12:14 -0500Thu, 10 Jul 2014 18:12:14 -0500Release 1.2Release 1.3Release 1.5Release 1.601Global Rank[CLJ-1456] The compiler ignores too few or too many arguments to throwhttp://dev.clojure.org/jira/browse/CLJ-1456
Clojure<p>The compiler does not fail on "malformed" throw forms:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (defn foo [] (<span class="code-keyword">throw</span>))
#'user/foo
user=&gt; (foo)
NullPointerException user/foo (NO_SOURCE_FILE:1)
user=&gt; (defn bar [] (<span class="code-keyword">throw</span> Exception baz))
#'user/bar
user=&gt; (bar)
ClassCastException java.lang.<span class="code-object">Class</span> cannot be <span class="code-keyword">cast</span> to java.lang.Throwable user/bar (NO_SOURCE_FILE:1)
; This one works, but ignored-symbol, should probably not be ignored
user=&gt; (defn quux [] (<span class="code-keyword">throw</span> (Exception. <span class="code-quote">"Works!"</span>) ignored-symbol))
#'user/quux
user=&gt; (quux)
Exception Works! user/quux (NO_SOURCE_FILE:1)</pre>
</div></div>
<p>The compiler can easily avoid these by counting forms.</p>
<p><b>Patch:</b> v3_0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1456" title="The compiler ignores too few or too many arguments to throw">CLJ-1456</a>-counting-forms-to-catch-malformed-throw-for.patch</p>
<p><b>Screened by:</b> Alex Miller</p>CLJ-1456The compiler ignores too few or too many arguments to throwEnhancementMajorOpenUnresolvedUnassignedAlf Kristian StøylecompilerftMon, 30 Jun 2014 11:52:14 -0500Sat, 10 Jan 2015 09:12:30 -0600Release 1.601<p>Not sure how to create a test for the attached patch. Will happily do so if anyone has a suggestion.</p><p>Re testing, I think the examples you give are good - you should add tests to test/clojure/test_clojure/compilation.clj that eval the form and expect compilation errors. I'm sure you can find similar examples.</p><p>Newest patch also contains a few tests.</p><p>All patches dated Jun 30 2014 and earlier no longer applied cleanly to latest master after some commits were made to Clojure on Aug 29, 2014. They did apply cleanly before that day.</p>
<p>I have not checked how easy or difficult it might be to update this patch. See section "Updating Stale Patches" on this wiki page for some tips on updating patches: <a href="http://dev.clojure.org/display/community/Developing+Patches">http://dev.clojure.org/display/community/Developing+Patches</a></p>
<p>Alf, it can help avoid confusion if different patches have different file names. JIRA lets you create multiple attachments with the same name, but I wouldn't recommend it.</p><p>It was easy to fix the patch. Uploaded the new patch <a href="http://dev.clojure.org/jira/secure/attachment/13291/v3_0001-CLJ-1456-counting-forms-to-catch-malformed-throw-for.patch">v3_0001-CLJ-1456-counting-forms-to-catch-malformed-throw-for.patch</a>, which applies cleanly to the current master.</p><p>Alf, while JIRA can handle multiple attachments for the same ticket with the same name, it can get confusing for people trying to determine which one with the same name is meant. Could you remove or rename one of your identically-named attachments? Instructions for deleting patches are in the "Removing patches" section on this wiki page: <a href="http://dev.clojure.org/display/community/Developing+Patches">http://dev.clojure.org/display/community/Developing+Patches</a></p><p>Removed both obsolete attachments. So shouldn't be confusing any more <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p>ApprovalTriagedGlobal RankPatchCode and Test[CLJ-1440] Unable to exclude clojure.lang.Compiler using :refer-clojurehttp://dev.clojure.org/jira/browse/CLJ-1440
Clojure<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns io.aviso.twixt.js-minification
<span class="code-quote">"Provides support <span class="code-keyword">for</span> JavaScript minification using the Google Closure compiler."</span>
(:refer-clojure :exclude [<span class="code-object">Compiler</span>])
(:<span class="code-keyword">import</span> (com.google.javascript.jscomp CompilerOptions ClosureCodingConvention DiagnosticGroups CheckLevel
SourceFile Result <span class="code-object">Compiler</span>))
(:require [clojure.java.io :as io]
[io.aviso.twixt.utils :as utils]
[io.aviso.tracker :as t]
[clojure.string :as str]))</pre>
</div></div>
<p>Results in:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>clojure.lang.Compiler$CompilerException: java.lang.IllegalStateException: Compiler already refers to: class clojure.lang.Compiler in namespace: io.aviso.twixt.js-minification, compiling:(/Users/hlship/workspaces/annadale/twixt/src/io/aviso/twixt/js_minification.clj:1:1)
java.lang.IllegalStateException: Compiler already refers to: class clojure.lang.Compiler in namespace: io.aviso.twixt.js-minification
clojure.lang.Namespace.referenceClass Namespace.java: 140
clojure.lang.Namespace.importClass Namespace.java: 158
clojure.lang.Namespace.importClass Namespace.java: 164
io.aviso.twixt.js-minification/eval4104/loading--auto-- js_minification.clj: 1
io.aviso.twixt.js-minification/eval4104 js_minification.clj: 1
clojure.lang.Compiler.eval Compiler.java: 6703
clojure.lang.Compiler.eval Compiler.java: 6692
clojure.lang.Compiler.load Compiler.java: 7130
io.aviso.twixt.js-minification/eval4100 form-init4106199735960171933.clj: 1
clojure.lang.Compiler.eval Compiler.java: 6703
clojure.lang.Compiler.eval Compiler.java: 6666
clojure.core/eval core.clj: 2927
clojure.main/repl/read-eval-print/fn main.clj: 239
clojure.main/repl/read-eval-print main.clj: 239
clojure.main/repl/fn main.clj: 257
clojure.main/repl main.clj: 257
clojure.lang.RestFn.invoke RestFn.java: 1096
clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn interruptible_eval.clj: 56
clojure.lang.AFn.applyToHelper AFn.java: 152
clojure.lang.AFn.applyTo AFn.java: 144
clojure.core/apply core.clj: 624
clojure.core/with-bindings* core.clj: 1862
clojure.lang.RestFn.invoke RestFn.java: 425
clojure.tools.nrepl.middleware.interruptible-eval/evaluate interruptible_eval.clj: 41
clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn interruptible_eval.clj: 171
clojure.core/comp/fn core.clj: 2402
clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn interruptible_eval.clj: 138
clojure.lang.AFn.run AFn.java: 22
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java: 1145
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java: 615
java.lang.Thread.run Thread.java: 724
</pre>
</div></div>CLJ-1440Unable to exclude clojure.lang.Compiler using :refer-clojureDefectMajorClosedDeclinedUnassignedHoward Lewis ShipcompilerinteropFri, 6 Jun 2014 16:31:55 -0500Sat, 7 Jun 2014 11:34:29 -0500Sat, 7 Jun 2014 11:34:29 -0500Release 1.600<p>refer and thus refer-clojure only works for Vars.<br/>
a workaround is:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns ..)
(ns-unmap *ns* '<span class="code-object">Compiler</span>)
(<span class="code-keyword">import</span> 'com.google.javascript.jscomp.<span class="code-object">Compiler</span>)</pre>
</div></div><p>Ditto what Nicola said. Or just fully-qualify.</p>Global Rank[CLJ-1422] Recur around try boxes primitiveshttp://dev.clojure.org/jira/browse/CLJ-1422
Clojure<p>Primitive function and recur variables can't pass through a (try) cleanly; they're boxed to Object instead. This causes reflection warnings for fns or loops that use primitive types.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (set! *warn-on-reflection* <span class="code-keyword">true</span>)
<span class="code-keyword">true</span>
user=&gt; (fn [] (loop [t 0] (recur t)))
#&lt;user$eval676$fn__677 user$eval676$fn__677@3d80023a&gt;
user=&gt; (fn [] (loop [t 0] (recur (<span class="code-keyword">try</span> t))))
NO_SOURCE_FILE:1 recur arg <span class="code-keyword">for</span> primitive local: t is not matching primitive, had: <span class="code-object">Object</span>, needed: <span class="code-object">long</span>
Auto-boxing loop arg: t
#&lt;user$eval680$fn__681 user$eval680$fn__681@5419323a&gt;
user=&gt; (fn [^<span class="code-object">long</span> x] (recur (<span class="code-keyword">try</span> x)))
NO_SOURCE_FILE:1 recur arg <span class="code-keyword">for</span> primitive local: x is not matching primitive, had: <span class="code-object">Object</span>, needed: <span class="code-object">long</span>
CompilerException java.lang.IllegalArgumentException: recur arg <span class="code-keyword">for</span> primitive local: x is not matching primitive, had: <span class="code-object">Object</span>, needed: <span class="code-object">long</span>, compiling:(NO_SOURCE_PATH:1:1)</pre>
</div></div>CLJ-1422Recur around try boxes primitivesEnhancementMinorOpenUnresolvedUnassignedKyle KingsburycompilerperformancetypehintsWed, 14 May 2014 20:32:20 -0500Mon, 28 Jul 2014 17:22:12 -0500Release 1.5Release 1.601<p>Without commenting on the most desirable behavior, the following code does not cause reflection warnings:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>user=&gt; (set! *warn-on-reflection* true)
true
user=&gt; (fn [] (loop [t 0] (recur (long (try t)))))
#&lt;user$eval673$fn__674 user$eval673$fn__674@4e56c411&gt;
</pre>
</div></div><p>Similar ticket <a href="http://dev.clojure.org/jira/browse/CLJ-701">http://dev.clojure.org/jira/browse/CLJ-701</a></p><p>try/catch in the compiler only implements Expr, not MaybePrimitiveExpr, looking at extending TryExpr with MaybePrimitiveExpr it seems simple enough, but it turns out recur analyzes it's arguments in the statement context, which causes (try ...) to essentially wrap itself in a function like ((fn [] (try ...))), at which point it is an invokeexpr which is much harder to add maybeprimitiveexpr too and it reduces to the same case as <a href="http://dev.clojure.org/jira/browse/CLJ-701" title="Compiler loses &#39;loop&#39;s return type in some cases">CLJ-701</a></p><p><a href="http://dev.clojure.org/jira/browse/CLJ-701">http://dev.clojure.org/jira/browse/CLJ-701</a> has a patch that I think solves this</p><p>Should I dupe this to <a href="http://dev.clojure.org/jira/browse/CLJ-701" title="Compiler loses &#39;loop&#39;s return type in some cases">CLJ-701</a>? </p><p>if you want the fixes for try out of the return context to be part of <a href="http://dev.clojure.org/jira/browse/CLJ-701" title="Compiler loses &#39;loop&#39;s return type in some cases">CLJ-701</a> then yes it is a dupe, if you are unsure or would prefer 701 to stay more focused (my patch may not be acceptable, or may be too large and doing too much) then no it wouldn't be a dupe. I sort of took it on myself to solve both in the patch on <a href="http://dev.clojure.org/jira/browse/CLJ-701" title="Compiler loses &#39;loop&#39;s return type in some cases">CLJ-701</a> because I came to <a href="http://dev.clojure.org/jira/browse/CLJ-701" title="Compiler loses &#39;loop&#39;s return type in some cases">CLJ-701</a> via Nicola's comment here, and the same compiler machinery can be used for both.</p>
<p>I think the status is pending on the status of <a href="http://dev.clojure.org/jira/browse/CLJ-701" title="Compiler loses &#39;loop&#39;s return type in some cases">CLJ-701</a>.</p>Global Rank[CLJ-1411] Special symbols can be shadowed inconsistentlyhttp://dev.clojure.org/jira/browse/CLJ-1411
Clojure<p>The compiler does not complain about let binding (or def-ing) special symbols, but the binding only works if not used at the beginning of a list:</p>
<p>These work:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(let [<span class="code-keyword">try</span> :a]
<span class="code-keyword">try</span>)
=&gt; :a
(let [<span class="code-keyword">try</span> (constantly :a)]
(apply <span class="code-keyword">try</span> :b))
=&gt; :a</pre>
</div></div>
<p>This doesn't work:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(let [<span class="code-keyword">try</span> (constantly :a)]
(<span class="code-keyword">try</span> :b))
=&gt; :b</pre>
</div></div>
<p>This is true for all special symbols, not just publicly exposed ones like try and new, but also internal ones like fn*.</p>
<p>I would expect consistent behaviour: either the compiler does not permit shadowing special symbols at all, or shadowing them works in all cases. </p>CLJ-1411Special symbols can be shadowed inconsistentlyDefectMinorOpenUnresolvedUnassignedVolkert Oakley JurgenscompilerMon, 28 Apr 2014 03:11:37 -0500Tue, 29 Apr 2014 05:58:36 -0500Release 1.603<p>I don't think that shadowing special symbols is a good idea, but probably having all the special symbols namespace qualified (clojure.core/import* is the only one ns-qualified atm) along with checking for the symbol in the locals env first and fallbacking to the special symbols map after that, would probably help in those scenarios</p><p>I think that shadowing special symbols is a bad idea. If that was possible, we'd have to change most macros in clojure.core to make them safe (i.e. explicitly add a namespace to each special symbol usage). And how would we handle special symbols that are not just implementation specific, like <em>try</em> and <em>new</em>? Every 3rd party macro that uses those might become unsafe.</p>
<p>My personal preference would be to prohibit the shadowing of special symbols.</p><p>That won't be the case since what I'm proposing includes making syntax-quote aware of the namespaced special symbols.<br/>
`def would expand to 'clojure.core/def for example.</p><p>That's true, but macros don't have to use the syntax quote. See for example the definition of <em>when</em>.</p>Global Rank[CLJ-1407] Recur mismatch might cause multiple evaluationhttp://dev.clojure.org/jira/browse/CLJ-1407
Clojure<p>Since mismatching recurs cause the loop body to be re-analyzed, macroexpansion in the loop body might happen more than once, causing any side effects that happen during macroexpansion to be evaluated potentially multiple times</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">Clojure 1.7.0-master-SNAPSHOT
user=&gt; (defmacro x [] (println <span class="code-quote">"foo"</span>))
#'user/x
user=&gt; (fn [] (loop [y 1] (x) (recur (<span class="code-object">Integer</span>. 1))))
foo
foo
#&lt;user$eval6$fn__7 user$eval6$fn__7@71687585&gt;</pre>
</div></div>CLJ-1407Recur mismatch might cause multiple evaluationDefectMinorOpenUnresolvedUnassignedNicola MomettocompilermacroThu, 17 Apr 2014 17:30:33 -0500Thu, 17 Apr 2014 21:54:32 -050001<p>This is not a question about whether the behavior in the description is a bug or not, but rather a curiosity about how often people write macros that have side effects at macroexpansion time. I think the following in Clojure itself do, but there may be others:</p>
<ul>
<li>gen-class, and also ns because it uses gen-class</li>
<li>gen-interface, and also definterface because it uses gen-interface</li>
<li>clojure.core/compile-if (private) calls eval on its expr arg, but as used now doesn't cause macroexpansion-time side effects</li>
<li>doc seems to have one case that prints at macroexpansion time</li>
<li>I am not sure whether defprotocol or deftype have macroexpansion time side effects, or whether they are limited to run time</li>
</ul>
<p>Andy, I don't think there are that many macros that side-effect at macroexpansion time and I haven't discovered this bug in real code but while thinking about how loop locals invalidation was implemented in Compiler.java.</p>
<p>Because there are a really a small number of side-effecting macros, this is unlikely to cause problems in real code, so I changed the priority to minor.</p>Global Rank[CLJ-1401] CompilerException / IllegalStateException when reloading namespaceshttp://dev.clojure.org/jira/browse/CLJ-1401
Clojure<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user&gt; (ns op)
nil
op&gt; (defn * [a b] (clojure.core/* a b))
WARNING: * already refers to: #'clojure.core/* in namespace: op, being replaced by: #'op/*
#'op/*
op&gt; (ns use-op (:require [op :refer :all]))
WARNING: * already refers to: #'clojure.core/* in namespace: use-op, being replaced by: #'op/*
nil
use-op&gt; (ns use-op (:require [op :refer :all]))
IllegalStateException * already refers to: #'op/* in namespace: use-op clojure.lang.Namespace.warnOrFailOnReplace (Namespace.java:88)
use-op&gt; (clojure.repl/pst *e)
IllegalStateException * already refers to: #'op/* in namespace: use-op
clojure.lang.Namespace.warnOrFailOnReplace (Namespace.java:88)
clojure.lang.Namespace.reference (Namespace.java:110)
clojure.lang.Namespace.refer (Namespace.java:168)
clojure.core/refer (core.clj:3920)
use-op/eval2402/loading--4958--auto----2403 (NO_SOURCE_FILE:1)
use-op/eval2402 (NO_SOURCE_FILE:1)
clojure.lang.<span class="code-object">Compiler</span>.eval (<span class="code-object">Compiler</span>.java:6703)
clojure.lang.<span class="code-object">Compiler</span>.eval (<span class="code-object">Compiler</span>.java:6692)
clojure.lang.<span class="code-object">Compiler</span>.eval (<span class="code-object">Compiler</span>.java:6666)
clojure.core/eval (core.clj:2927)
clojure.main/repl/read-eval-print--6625/fn--6628 (main.clj:239)
clojure.main/repl/read-eval-print--6625 (main.clj:239)</pre>
</div></div>
<p>I would expect (at worst) a similar warning to the initial namespace loading, rather than an exception here.</p>CLJ-1401CompilerException / IllegalStateException when reloading namespacesDefectMinorClosedDuplicateUnassignedMike AndersoncompilererrormsgsThu, 10 Apr 2014 22:39:51 -0500Mon, 3 Nov 2014 10:24:17 -0600Mon, 3 Nov 2014 10:24:17 -0600Release 1.602<p>Could you put together a better reproducible test case for this that does not depend on core.matrix? Also, please include the (pst *e) when it occurs.</p><p>I have tried the smallest possible Leiningen project I could think of that would cause the warnings about redefinitions, to see if I could get the exception to occur. 'lein new try1' to create the skeleton project, then edit src/try1/core.clj to contain only the following function definitions:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(defn merge
<span class="code-quote">"This definition of merge replaces clojure.core/merge"</span>
[x y]
(- x y))
(defn *
[x y]
(* x y))</pre>
</div></div>
<p>Then start a REPL with 'lein repl', and I see this behavior:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (require '[try1.core :as c])
WARNING: merge already refers to: #'clojure.core/merge in namespace: try1.core, being replaced by: #'try1.core/merge
WARNING: * already refers to: #'clojure.core/* in namespace: try1.core, being replaced by: #'try1.core/*
nil
user=&gt; (require '[try1.core :as c] )
nil
user=&gt; (require '[try1.core :as c] :reload)
WARNING: merge already refers to: #'clojure.core/merge in namespace: try1.core, being replaced by: #'try1.core/merge
WARNING: * already refers to: #'clojure.core/* in namespace: try1.core, being replaced by: #'try1.core/*
nil</pre>
</div></div>
<p>Ths all looks like behavior as I would expect, and I did not see the exception that Mike reports.</p>
<p>It seems that either Ctrl+Alt+L in Counterclockwise does something different than (require ... :reload), or there is something different about Mike's namespace in addition to redefining names in clojure.core that is causing the problem.</p><p>Marking this as NR for now - would be happy to see it reopened with an easily reproducible test case.</p><p>To reproduce:</p>
<p>(ns op)<br/>
(defn * <span class="error">&#91;a b&#93;</span> (clojure.core/* a b)) ;; gives warning<br/>
(ns use-op (:require <span class="error">&#91;op :refer :all&#93;</span>)) ;; gives warning<br/>
(ns use-op (:require <span class="error">&#91;op :refer :all&#93;</span>)) ;; gives error!</p>
<p>I believe Counterclockwise is simply loading the namespace again with CTRL-Alt+L, which is causing the ns form to be re-executed.</p>
<p>The docstring implies that ns can be used multiple times ("Sets <b>ns</b> to the namespace named by name (unevaluated), creating it if needed") so I would certainly expect multiple invocations of ns to be a no-op</p>
<p>Duped in <a href="http://dev.clojure.org/jira/browse/CLJ-1578" title="1.7.0-alpha3 breakage due to symbol conflicts"><del>CLJ-1578</del></a>.</p>Global Rank[CLJ-1400] Error "Can't refer to qualified var that doesn't exist" should name the bad symbolhttp://dev.clojure.org/jira/browse/CLJ-1400
Clojure<p>Def of var with a ns that doesn't exist will yield this error:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user&gt; (def foo/bar 1)
CompilerException java.lang.RuntimeException: Can't refer to qualified <span class="code-keyword">var</span> that doesn't exist, compiling:(NO_SOURCE_PATH:1:1)</pre>
</div></div>
<p><b>Cause:</b> Compiler.lookupVar() returns null if the ns in a qualified var does not exist yet. </p>
<p><b>Proposed:</b> The error message would be improved by naming the symbol and throwing a CompilerException with file/line/col info. It's not obvious, but this may be the only case where this error occurs. If so, the error message could be more specific that the ns is the part that doesn't exist.</p>
<p><b>Patch:</b> clj-1400-4.diff</p>
<p><b>Screened by:</b> Alex Miller</p>OS XCLJ-1400Error "Can't refer to qualified var that doesn't exist" should name the bad symbolEnhancementMinorOpenUnresolvedUnassignedHoward Lewis ShipCompilererrormsgsWed, 9 Apr 2014 13:10:40 -0500Tue, 7 Oct 2014 12:34:20 -0500Release 1.5Release 1.802<p>This looks to me like relatively low hanging fruit unless I'm missing something; assigning to myself.</p><p>Patch <tt>clj-1400-1.diff</tt> to <tt>Compiler.java</tt>.</p>
<p>With this patch the example would now look like:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user&gt; (def foo/bar 1)
CompilerException java.lang.RuntimeException: Qualified symbol foo/bar refers to nonexistent namespace: foo, compiling:(NO_SOURCE_PATH:1:1)</pre>
</div></div>
<p>I'm not sure the <tt>if(namesStaticMember(sym))</tt> <span class="error">&#91;see below&#93;</span>, and the 2nd branch, is even necessary. Just by inspection I suspect it is not.</p>
<p><span class="error">&#91;footnote&#93;</span></p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java"><span class="code-keyword">public</span> <span class="code-keyword">static</span> <span class="code-object">boolean</span> namesStaticMember(Symbol sym){
<span class="code-keyword">return</span> sym.ns != <span class="code-keyword">null</span> &amp;&amp; namespaceFor(sym) == <span class="code-keyword">null</span>;
}</pre>
</div></div><p>patch: code and test</p><p>I tested on an actual source file, and the exception message included the file/line/col info as desired:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; CompilerException java.lang.RuntimeException: Qualified symbol goo/bar refers to nonexistent namespace: goo, compiling:(/home/scott/dev/foo.clj:3:1)</pre>
</div></div><p>Patch clj-1400-1.diff dated Jun 26 2014 no longer applied cleanly to latest master after some commits were made to Clojure on Aug 29, 2014. It did apply cleanly before that day.</p>
<p>I have not checked how easy or difficult it might be to update this patch. See section "Updating Stale Patches" on this wiki page for some tips on updating patches: <a href="http://dev.clojure.org/display/community/Developing+Patches">http://dev.clojure.org/display/community/Developing+Patches</a></p><p>Attached is an updated patch: "clj-1400-2.diff". I removed the stale patch.</p><p>Few comments to address:</p>
<ul class="alternate" type="square">
<li>Compiler diff was using spaces, not tabs, which makes it harder to diff. I attached a -3.diff that fixes this.</li>
<li>the call to namesStaticMember seems weird. The name of that method is confusing for this use. Beyond that, I think it's doing more than you need. That method is going to attempt resolve the qualified name in terms of the current ns, but I think you don't even want to do that. Rather you just want to know if the sym has a ns (sym.ns != null) - isn't that enough?</li>
<li>In what case will the other error "Var doesn't exist" occur? In other words, in what case will lookupVar not succeed in creating a new var here? If there is no such case, then remove this case. If there is such a case, then add a test.</li>
</ul>
<p>Agree with all three of your bullets. Attached is an updated patch, clj-1400-4.diff. </p>
<ul>
<li>I used tabs in Compiler.java</li>
<li>After close inspection of call to lookupVar(...), I believe null is returned only in the case of exactly this ticket (the symbol having a non-null namespace which has not been loaded yet). So I've taken out the conditional and the 2nd branch.</li>
<li>(Test is unchanged)</li>
</ul>
<p>(properly named patch)</p><p>You could throw a CompilerException with the location of the problem instead (as the ticket description suggests).</p><p>Sorry, I should've mentioned because this wasn't obvious to me either (and in fact I forgot until just now): the RuntimeException is already caught and wrapped in a CompilerException. </p>
<p>I'm not sure which try-catch block within Compiler.java this is happening in, there are multiple. But you can see in the output that the exception is a CompilerException and the file|line|col info is there:</p>
<p>In the Repl...</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>user&gt; (def foo/bar 1)
CompilerException java.lang.RuntimeException: Qualified symbol foo/bar refers to nonexistent namespace: foo, compiling:(NO_SOURCE_PATH:1:1)
</pre>
</div></div>
<p>...or in a source file </p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>user=&gt; CompilerException java.lang.RuntimeException: Qualified symbol goo/bar refers to nonexistent namespace: goo, compiling:(/home/scott/dev/foo.clj:3:1)
</pre>
</div></div>
<p>Also, at the point at which the RuntimeException of this patch is being thrown, the <tt>source</tt> <tt>line</tt> and <tt>col</tt> params to CompilerException are not available, or at least not afaict.</p><p>I'll follow up on this patch later - Rich thought it was making too many assumptions. I think we validated many of those but need to double-check those.</p>ApprovalIncompleteGlobal RankPatchCode and Test[CLJ-1399] missing field munging when recreating deftypes serialized in to byte codehttp://dev.clojure.org/jira/browse/CLJ-1399
Clojure<p>to embed deftypes in the bytecode the compiler emits the value of each field, then emits a call to the deftypes underlying class's constructor.</p>
<p>to get a list of fields the compiler calls .getBasis.</p>
<p>the getBasis fields are the "clojure" level field names of the deftype, which the actual "jvm" level field names have been munged (replacing - with _, etc), so the compiler tries to generate code to set values on non-existent fields</p>
<p><a href="https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L4579">https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L4579</a></p>
<p><a href="https://www.refheap.com/70731">https://www.refheap.com/70731</a></p>
<p>you can work around this by using field names that don't require munging. a solution might be just calling munge in the emission of field sets of ITypes</p>CLJ-1399missing field munging when recreating deftypes serialized in to byte codeDefectMajorOpenUnresolvedUnassignedKevin DowneycompilerdeftypeWed, 2 Apr 2014 16:13:25 -0500Thu, 19 Feb 2015 12:08:07 -060024<p>reproducing case</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>$ rlwrap java -server -Xmx1G -Xms1G -jar /Users/hiredman/src/clojure/target/clojure-1.6.0-master-SNAPSHOT.jar
Clojure 1.6.0-master-SNAPSHOT
user=&gt; (deftype Foo [hello-world])
user.Foo
user=&gt; (alter-var-root #'default-data-readers assoc 'foo (fn [x] (Foo. x)))
{foo #&lt;user$eval6$fn__7 user$eval6$fn__7@2f953efd&gt;, inst #'clojure.instant/read-instant-date, uuid #'clojure.uuid/default-uuid-reader}
user=&gt; #foo "1"
CompilerException java.lang.IllegalArgumentException: No matching field found: hello-world for class user.Foo, compiling:(NO_SOURCE_PATH:0:0)
user=&gt;
</pre>
</div></div><p>this patch fixes the issue on the latest master for me</p><p>FWIW, this was precipitated by real experience (I think I created the refheap paste). The workaround is easy (don't use dashes in field names of deftypes you want to return from data reader functions), but I wouldn't expect anyone to guess that that wasn't already oversensitized to munging edge cases.</p>ApprovalTriagedGlobal RankPatchCode[CLJ-1392] AOT vs "var already exists warning" results in NPEhttp://dev.clojure.org/jira/browse/CLJ-1392
Clojure<p>I just saw this thread on the cascalog list: <a href="https://groups.google.com/forum/#!topic/cascalog-user/Pe5QIpmU0vA">https://groups.google.com/forum/#!topic/cascalog-user/Pe5QIpmU0vA</a></p>
<p>Apparently the "WARNING: some? already refers to: #'clojure.core/some? in namespace: jackknife.seq, being replaced by: #'jackknife.seq/some?" results in a NullPointerException when trying to AOT a namespace that results in the message being output.</p>
<p>Reproducer:</p>
<p>1) lein new aotFail<br/>
2) Edit project.clj <br/>
;add as appropriate<br/>
:aot :all<br/>
:dependencies [<span class="error">&#91;org.clojure/clojure &quot;1.6.0&quot;&#93;</span><br/>
<span class="error">&#91;cascalog &quot;2.0.0&quot;&#93;</span>]<br/>
2) Add "(:use cascalog.api)" to the ns block of src/aotFail/core.clj<br/>
3) lein compile</p>
<p>Output:</p>
<p>Compiling aotFail.core<br/>
WARNING: some? already refers to: #'clojure.core/some? in namespace: jackknife.seq, being replaced by: #'jackknife.seq/some?<br/>
Exception in thread "main" java.lang.NullPointerException, compiling:(api.clj:1:1)<br/>
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3558)<br/>
at clojure.lang.Compiler.compile1(Compiler.java:7226)<br/>
at clojure.lang.Compiler.compile1(Compiler.java:7216)<br/>
at clojure.lang.Compiler.compile(Compiler.java:7292)<br/>
at clojure.lang.RT.compile(RT.java:398)<br/>
at clojure.lang.RT.load(RT.java:438)<br/>
at clojure.lang.RT.load(RT.java:411)<br/>
at clojure.core$load$fn__5066.invoke(core.clj:5641)<br/>
at clojure.core$load.doInvoke(core.clj:5640)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:408)<br/>
at clojure.core$load_one.invoke(core.clj:5446)<br/>
at clojure.core$load_lib$fn__5015.invoke(core.clj:5486)<br/>
at clojure.core$load_lib.doInvoke(core.clj:5485)<br/>
at clojure.lang.RestFn.applyTo(RestFn.java:142)<br/>
at clojure.core$apply.invoke(core.clj:626)<br/>
at clojure.core$load_libs.doInvoke(core.clj:5528)<br/>
at clojure.lang.RestFn.applyTo(RestFn.java:137)<br/>
at clojure.core$apply.invoke(core.clj:628)<br/>
at clojure.core$use.doInvoke(core.clj:5618)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:408)<br/>
at cascalog2.core$loading_<em>4958</em><em>auto</em>_.invoke(core.clj:1)<br/>
at clojure.lang.AFn.applyToHelper(AFn.java:152)<br/>
at clojure.lang.AFn.applyTo(AFn.java:144)<br/>
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3553)<br/>
at clojure.lang.Compiler.compile1(Compiler.java:7226)<br/>
at clojure.lang.Compiler.compile1(Compiler.java:7216)<br/>
at clojure.lang.Compiler.compile(Compiler.java:7292)<br/>
at clojure.lang.RT.compile(RT.java:398)<br/>
at clojure.lang.RT.load(RT.java:438)<br/>
at clojure.lang.RT.load(RT.java:411)<br/>
at clojure.core$load$fn__5066.invoke(core.clj:5641)<br/>
at clojure.core$load.doInvoke(core.clj:5640)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:408)<br/>
at clojure.core$load_one.invoke(core.clj:5446)<br/>
at clojure.core$compile$fn__5071.invoke(core.clj:5652)<br/>
at clojure.core$compile.invoke(core.clj:5651)<br/>
at user$eval19.invoke(form-init2092370125048380878.clj:1)<br/>
at clojure.lang.Compiler.eval(Compiler.java:6703)<br/>
at clojure.lang.Compiler.eval(Compiler.java:6693)<br/>
at clojure.lang.Compiler.load(Compiler.java:7130)<br/>
at clojure.lang.Compiler.loadFile(Compiler.java:7086)<br/>
at clojure.main$load_script.invoke(main.clj:274)<br/>
at clojure.main$init_opt.invoke(main.clj:279)<br/>
at clojure.main$initialize.invoke(main.clj:307)<br/>
at clojure.main$null_opt.invoke(main.clj:342)<br/>
at clojure.main$main.doInvoke(main.clj:420)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:421)<br/>
at clojure.lang.Var.invoke(Var.java:383)<br/>
at clojure.lang.AFn.applyToHelper(AFn.java:156)<br/>
at clojure.lang.Var.applyTo(Var.java:700)<br/>
at clojure.main.main(main.java:37)<br/>
Caused by: java.lang.NullPointerException<br/>
at clojure.lang.Compiler$ObjExpr.emitVar(Compiler.java:4944)<br/>
at clojure.lang.Compiler$DefExpr.emit(Compiler.java:437)<br/>
at clojure.lang.Compiler.compile1(Compiler.java:7225)<br/>
at clojure.lang.Compiler.compile(Compiler.java:7292)<br/>
at clojure.lang.RT.compile(RT.java:398)<br/>
at clojure.lang.RT.load(RT.java:438)<br/>
at clojure.lang.RT.load(RT.java:411)<br/>
at clojure.core$load$fn__5066.invoke(core.clj:5641)<br/>
at clojure.core$load.doInvoke(core.clj:5640)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:408)<br/>
at clojure.core$load_one.invoke(core.clj:5446)<br/>
at clojure.core$load_lib$fn__5015.invoke(core.clj:5486)<br/>
at clojure.core$load_lib.doInvoke(core.clj:5485)<br/>
at clojure.lang.RestFn.applyTo(RestFn.java:142)<br/>
at clojure.core$apply.invoke(core.clj:626)<br/>
at clojure.core$load_libs.doInvoke(core.clj:5524)<br/>
at clojure.lang.RestFn.applyTo(RestFn.java:137)<br/>
at clojure.core$apply.invoke(core.clj:628)<br/>
at clojure.core$use.doInvoke(core.clj:5618)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:408)<br/>
at cascalog.api$loading_<em>4958</em><em>auto</em>_.invoke(api.clj:1)<br/>
at clojure.lang.AFn.applyToHelper(AFn.java:152)<br/>
at clojure.lang.AFn.applyTo(AFn.java:144)<br/>
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3553)</p>CLJ-1392AOT vs "var already exists warning" results in NPEDefectMajorClosedDuplicateUnassignedAaron CohenaotcompilerWed, 26 Mar 2014 11:58:29 -0500Wed, 26 Mar 2014 12:42:56 -0500Wed, 26 Mar 2014 12:42:56 -0500Release 1.601<p>Duplicate of <a href="http://dev.clojure.org/jira/browse/CLJ-1241">http://dev.clojure.org/jira/browse/CLJ-1241</a></p>Global Rank[CLJ-1351] Clojure emits an unused "swapThunk" method for functions with keyword callsiteshttp://dev.clojure.org/jira/browse/CLJ-1351
Clojure<p>This method is no longer used, I did a quick git blame and it look like it was used for an earlier implementation of keyword callsites and forgot to be removed in this commit <a href="https://github.com/clojure/clojure/commit/c7af275d4ee33cdc1794c8df8fa1e6d39039ac84">https://github.com/clojure/clojure/commit/c7af275d4ee33cdc1794c8df8fa1e6d39039ac84</a></p>
<p>Removing this should reduce a bit the size of compile functions.</p>CLJ-1351Clojure emits an unused "swapThunk" method for functions with keyword callsitesEnhancementMajorOpenUnresolvedUnassignedNicola MomettocompilerperformanceFri, 14 Feb 2014 14:05:23 -0600Sun, 28 Dec 2014 11:11:46 -060000ApprovalTriagedGlobal RankPatchCode[CLJ-1342] Byte comparison boxes both bytes and converts to longs to compare (which is slow)http://dev.clojure.org/jira/browse/CLJ-1342
Clojure<p>This came up in a much more complicated example but consider a case like this:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(defn simple []
(let [b (<span class="code-object">byte</span>-array [(<span class="code-object">byte</span> 0)])
m (<span class="code-object">byte</span> 0)]
(= m (aget b 0))))</pre>
</div></div>
<p>In the compiled bytecode, both m and (aget b 0) are known to be bytes, but both are boxed using Byte.valueOf(), then cast using RT.uncheckedLongCast() and finally compared as longs:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">26: iload_2
27: invokestatic #69 <span class="code-comment">// Method java/lang/<span class="code-object">Byte</span>.valueOf:(B)Ljava/lang/<span class="code-object">Byte</span>;
</span> 30: checkcast #81 <span class="code-comment">// class java/lang/<span class="code-object">Number</span>
</span> 33: invokestatic #85 <span class="code-comment">// Method clojure/lang/RT.uncheckedLongCast:(Ljava/lang/<span class="code-object">Object</span>;)J</span></pre>
</div></div>
<p>In a tight loop manipulating and matching against byte arrays, this boxing is significant for performance. </p>
<p>Attached is a test that demonstrates the performance difference between the byte[] and long[] performance to get an idea of the difference.</p>CLJ-1342Byte comparison boxes both bytes and converts to longs to compare (which is slow)EnhancementMinorOpenUnresolvedUnassignedAlex MillercompilerThu, 6 Feb 2014 17:10:24 -0600Thu, 6 Feb 2014 21:39:45 -060012<p>The description states that Util.equiv() has a byte/byte comparison variant but it doesn't look like it actually exists.</p><p>By the way, tools.emitter.jvm uses i2l to cast the byte to a long instead of boxing &amp;&amp; unboxing to a long</p><p>Thanks Nicola - I must have confused it with the boolean/boolean version.</p>Global Rank[CLJ-1330] Class name clash between top-level functions and defn'ed oneshttp://dev.clojure.org/jira/browse/CLJ-1330
Clojure<p>Named anonymous <tt>fn</tt>'s are not guaranteed to have unique class names when AOT-compiled.</p>
<p>For example:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(defn g [])
(def xx (fn g []))</pre>
</div></div>
<p>When AOT-compiled both functions will emit <tt>user$g.class</tt>, the latter overwriting the former.</p>
<p><b>Impact:</b> this affects apps like Cursive, which has been using a patched version of Clojure to get around this issue for quite a while.</p>
<p><b>Demonstration script:</b> demo1.clj</p>
<p><b>Patch:</b> 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a>-v3.patch (already applied, see below)</p>
<p><b>Approach:</b> Generate unique class names for named <tt>fn</tt>'s the same way as for unnamed anonymous <tt>fn</tt>'s. <br/>
The patch contains an additional enhancement to include the name of the local binding in the class name.</p>
<p>Comparison between pre and post patch naming scheme (N denotes unique number):</p>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'>code</th>
<th class='confluenceTh'>before</th>
<th class='confluenceTh'>after</th>
<th class='confluenceTh'>note</th>
</tr>
<tr>
<td class='confluenceTd'><font color="red">(defn a [])</font></td>
<td class='confluenceTd'>user$a</td>
<td class='confluenceTd'>user$a</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'><font color="red">(fn [])</font></td>
<td class='confluenceTd'>user$evalN$fn__N</td>
<td class='confluenceTd'>user$evalN$fn__N</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'><font color="red">(fn a [])</font></td>
<td class='confluenceTd'>user$evalN$a__N</td>
<td class='confluenceTd'>user$evaN$a__N</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'>(let <span class="error">&#91;a <font color="red">(fn [])</font>&#93;</span> a)</td>
<td class='confluenceTd'>user$evalN$a__N</td>
<td class='confluenceTd'>user$evalN$a__N</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'>(let <span class="error">&#91;a <font color="red">(fn x [])</font>&#93;</span> a)</td>
<td class='confluenceTd'>user$eval1N$x__N</td>
<td class='confluenceTd'>user$evalN$a_<em>x</em>_N</td>
<td class='confluenceTd'>IMPROVED - contains local binding name</td>
</tr>
<tr>
<td class='confluenceTd'>(def a <font color="red">(fn [])</font>)</td>
<td class='confluenceTd'>user$a</td>
<td class='confluenceTd'>user$a</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'>(def a <font color="red">(fn x [])</font>)</td>
<td class='confluenceTd'>user$x</td>
<td class='confluenceTd'>user$a_<em>x</em>_N</td>
<td class='confluenceTd'>FIXED conflict with <tt>(defn x [])</tt></td>
</tr>
<tr>
<td class='confluenceTd'>(def ^{:foo <font color="red">(fn [])</font>} a)</td>
<td class='confluenceTd'>user$fn__N</td>
<td class='confluenceTd'>user$fn__N</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'>(def ^{:foo <font color="red">(fn a [])</font>} a)</td>
<td class='confluenceTd'>user$a</td>
<td class='confluenceTd'>user$a__N</td>
<td class='confluenceTd'>FIXED conflict with <tt>(defn a [])</tt></td>
</tr>
<tr>
<td class='confluenceTd'>(def a (fn [] <font color="red">(fn [])</font>))</td>
<td class='confluenceTd'>user$a$fn__N</td>
<td class='confluenceTd'>user$a$fn__N</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'>(def a (fn [] <font color="red">(fn x [])</font>))</td>
<td class='confluenceTd'>user$a$x__N</td>
<td class='confluenceTd'>user$a$x__N</td>
<td class='confluenceTd'>same</td>
</tr>
</tbody></table>
<p><b>See also:</b> This patch also fixes the issue reported in <a href="http://dev.clojure.org/jira/browse/CLJ-1227" title="Definline functions do not work as higher-order functions when AOT compiled"><del>CLJ-1227</del></a>.</p>
<p><b>Screened by:</b> Alex Miller - I am not sure whether the local binding name enhancement is worth doing. It improves debugging of which anonymous class you're talking about but has the downsides of increasing class name (and file name) length.</p>
<p><b>REOPENED Patch:</b> 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a>-v3-no-locals-improv.patch<br/>
<b>REOPENED UPDATE:</b> The local improvement version of this patch increases class names (and thus .class names) and we've had several reports of exceeding file system limits (~143 chars) - see comments for details. Thus, we should rollback the prior patch (0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a>&#45;v3.patch) and apply the version without this enhancement. The replacement patch (0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a>-v3-no-locals-improv.patch) does not have the same effect on class name length.</p>CLJ-1330Class name clash between top-level functions and defn'ed onesDefectCriticalClosedCompletedUnassignedNicola MomettoaotcompilerWed, 22 Jan 2014 08:08:16 -0600Fri, 14 Nov 2014 10:08:16 -0600Fri, 14 Nov 2014 10:08:16 -0600Release 1.7811<p>This seems like the reason why jvm.tools.analyzer cannot analyze clojure.core. On analyzing a definline, there is an "attempted duplicate class definition" error.</p>
<p>This doesn't really matter, but I thought it may or may not be useful information to someone.</p><p>Attached a fix.</p>
<p>This also fixes AOT compiling of code like:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(def x (fn foo []))
(fn foo [])</pre>
</div></div><p>Cleaned up patch</p><p>It looks like the patch changes indentation of some of the code - can you fix that?</p><p>Updated patch without whitespace changes</p><p>Thanks, that's helpful.</p><p>There is consensus that this is a problem, however this is an area of the code with broad impacts as it deals with how classes are named. To that end, there is some work that needs to be done in understanding the impacts before we can consider it.</p>
<p>Some questions we would like to answer:</p>
<p>1) According to Rich, naming of (fn x []) function classes used to work in the manner of this patch - with generated names. Some code archaeology needs to be done on why that was changed and whether the change to the current behavior was addressing problems that we are likely to run into.</p>
<p>2) Are there issues with recursive functions? Are there impacts either in AOT or non-AOT use cases? Need some tests.</p>
<p>3) Are there issues with dynamic redefinition of functions? With the static naming scheme, redefinition causes a new class of the same name which can be picked up by reload of classes compiled to the old definition. With the dynamic naming scheme, redefinition will create a differently named class so old classes can never pick up a redefinition. Is this a problem? What are the impacts with and without AOT? Need some tests.</p><p>Looks like the current behaviour has been such since <a href="https://github.com/clojure/clojure/commit/4651e60808bb459355a3a5d0d649c4697c672e28">https://github.com/clojure/clojure/commit/4651e60808bb459355a3a5d0d649c4697c672e28</a></p>
<p>My guess is that Rich simply forgot to consider the (def y (fn x [] ..)) case.</p>
<p>Regarding 2 and 3, the dynamic naming scheme is no different than what happens for anonymous functions so I don't see how this could cause any issue.</p>
<p>Recursion on the fn arg is simply a call to .invoke on "this", it's classname unaware.</p>
<p>I can add some tests to test that <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(def y (fn x [] 1))</pre>
</div></div> and <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(fn x [] 2)</pre>
</div></div> compile to different classnames but other than that I don't see what should be tested.</p><p>incomplete pending the answers to Alex Miller's questions in the comments</p><p>I believe I already answered his questions, I'll try to be a bit more explicit:<br/>
I tracked the relevant commit from Rich which added the dynamic naming behaviour <a href="https://github.com/clojure/clojure/commit/4651e60808bb459355a3a5d0d649c4697c672e28#diff-f17f860d14163523f1e1308ece478ddbL3081">https://github.com/clojure/clojure/commit/4651e60808bb459355a3a5d0d649c4697c672e28#diff-f17f860d14163523f1e1308ece478ddbL3081</a> which clearly shows that this bug was present since then so.</p>
<p>Regarding redefinitions or recursive functions, both of those operations never take in account the generated fn name so they are unaffected.</p><p>Summarizing some cases here from before/after the patch:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>1) top-level fn (always has name)
1.6 - namespace$name
patch - namespace$name
2) non-top-level fn with name
1.6 - namespace$name (collides with prior case)
patch - namespace$topname__x__name &lt;-- CHANGED
3) anonymous fn (no name)
1.6 - namespace$name$fn__x
patch - namespace$name$fn__x
4) top-level anonymous fn (no name, not at all useful :)
1.6 - namespace$fn__x
patch - namespace$fn__x
</pre>
</div></div>
<p>The key problem is that the first 2 cases produce the identical class name on 1.6. The patch alters the non-top-level named fn so there is no conflict. </p>
<p>Prior to the referenced old commit, I believe cases 1 and 2 would both produce namespace$name__x (where x is unique) so they would not collide. The change was made to prevent the top-level name from changing ("don't append numbers on top-level fn class names"). While the similar change was made on non-top-level fn names, I do not think it needed to be.</p>
<p>I've thought through (and tried) a bunch of the implications of this with the help of Nicola's comments above and I do not see an issue with any of the things I've considered. From a binary compatibility point of view with existing AOT code, old code compiled together should be self-consistent and continue to work. New compiled code will also be consistent. I can't think of a way that new code would expect to know the old name of a non-top-level function such that there could be an issue. </p>
<p>One question - why change the code such that the new class name is namespace$name$topname_&#95;x_&#95;name instead of namespace$name$topname_name_&#95;x (or something else?). And relatedly, while the diff is small, could we refactor a couple more lines to make the intent and cases clearer?</p>
<p>I am 90% ok with this patch but want a little thought into that question before I mark screened. </p><p>Alex, the attached patch munges into ns$topname_&#95;name_&#95;x, not into ns$topname_&#95;x_&#95;name.</p><p>The attached patch 0001-Fix-<a href="http://dev.clojure.org/jira/browse/CLJ-1330" title="Class name clash between top-level functions and defn&#39;ed ones"><del>CLJ-1330</del></a><del>refactored.patch contains the same fix from 0001-Fix</del><a href="http://dev.clojure.org/jira/browse/CLJ-1330" title="Class name clash between top-level functions and defn&#39;ed ones"><del>CLJ-1330</del></a>-make-top-level-named-functions-classnam.patch but also refactors the code that deals with fn name munging</p><p>Hmmm.. I will double-check. That's not why I recall seeing when I did AOT.</p><p>New patch 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a>-v2.patch improves the fn naming scheme a lot.<br/>
I've threw together a number of test cases that show the improvement + bug fixes:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (fn [])
;; pre:
#&lt;user$eval1$fn__2 user$eval1$fn__2@4e13aa4e&gt;
;; post: (no change)
#&lt;user$eval1$fn__3 user$eval1$fn__3@3c92218c&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (fn a [])
;; pre:
#&lt;user$eval5$a__6 user$eval5$a__6@6946a317&gt;
;; post: (no change)
#&lt;user$eval6$a__8 user$eval6$a__8@6f85c59c&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (let [a (fn [])] a)
;; pre:
#&lt;user$eval9$a__10 user$eval9$a__10@15fdf894&gt;
;; post: (no change)
#&lt;user$eval11$a__13 user$eval11$a__13@4d051922&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (let [a (fn x [])] a)
;; pre: (only contains the name of the fn)
#&lt;user$eval17$x__18 user$eval17$x__18@7f0cd67f&gt;
;; post: (contains the name of the local aswell as the name of the fn
#&lt;user$eval21$a__x__23 user$eval21$a__x__23@528ef256&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def a (fn [])) a
#'user/a
;; pre:
#&lt;user$a user$a@33e1ccbc&gt;
;; post: (no change)
#&lt;user$a user$a@6bef63f9&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def a (fn x [])) a
#'user/a
;; pre: (BUG!)
#&lt;user$x user$x@59a04a1b&gt;
;; post: (bug fixed)
#&lt;user$a__x__28 user$a__x__28@5f0bebef&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def ^{:foo (fn [])} a) (-&gt; (meta #'a) :foo)
#'user/a
;; pre:
#&lt;user$fn__23 user$fn__23@d9c21c6&gt;
;; post: (no change)
#&lt;user$fn__30 user$fn__30@4cf0f2eb&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def ^{:foo (fn a [])} a) (-&gt; (meta #'a) :foo)
#'user/a
;; pre: (BUG!)
#&lt;user$a user$a@420dd874&gt;
;; post: (bug fixed)
#&lt;user$a__35 user$a__35@37ff95a9&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def a (fn [] (fn []))) (a)
#'user/a
;; pre:
#&lt;user$a$fn__30 user$a$fn__30@6f57be76&gt;
;; post: (no change)
#&lt;user$a$fn__41 user$a$fn__41@fd34eac&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def a (fn [] (fn x []))) (a)
#'user/a
;; pre:
#&lt;user$a$x__35 user$a$x__35@79930089&gt;
;; post: (no change)
#&lt;user$a$x__48 user$a$x__48@6fc334de&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (let [x (fn [])] (def a (fn [] x))) a (a)
#'user/a
;; pre:
#&lt;user$eval40$a__43 user$eval40$a__43@6db1694e&gt;
#&lt;user$eval40$x__41 user$eval40$x__41@20bd16bb&gt;
;; post (no change)
#&lt;user$eval54$a__58 user$eval54$a__58@7c721de&gt;
#&lt;user$eval54$x__56 user$eval54$x__56@43f7b41b&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (let [x (fn a [])] (def a (fn [] x))) (a)
#'user/a
;; pre: (the local binding name doesn't appear in the class name)
#&lt;user$eval48$a__49 user$eval48$a__49@75d6d1d4&gt;
;; post: (the local binding name is included in the class name)
#&lt;user$eval64$x__a__66 user$eval64$x__a__66@460d4&gt;</pre>
</div></div>
<p>As you can see, this last patch not only fixes the two bugs, but also improves fn naming in let contexts by preserving the name of the local binding in the class name, this I believe will be a great improvement in the understandability of stacktraces.</p><p>The patch should be changed to not create suffix if it's not going to be used. Please update the patch to inline that into each branch <tt>name = nm.name + "__" + RT.nextID();</tt>.</p>
<p>I am unsure whether the "enhancement" part of this patch goes too far. I think it does provide some improvements in debugging but those seem small to me. I am somewhat concerned about greatly increasing the name of the class for nested locals thus making it <b>harder</b> to read stack traces. There is a large limit to class name size of 16 bits (what you can put in the constant table) but class names also map to file names and there have historically been issues on some older Windows architectures with file size limits - we are increasing the risk of running into problems with this. Small risks. I am ok with passing this on to Rich though and he can decide whether to kick that part back or not.</p><p>0001&#45;CLJ&#45;1093&#45;v3.patch is identical to 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a>-v2.patch except it doesn't call RT.nextID() when not necessary, as per Alex's request</p>
<p>Alex, if this is ok please change the "Patch:" field in the description, I won't do that myself since this ticket is now screened</p><p>Addressing the screening comment by Alex Miller, I've attached an alternative patch "0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a><del>v3-no-locals-improv.patch" which is identical to "0001</del><a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a>-v3.patch" except it doesn't include the local binding name enhancement, so that it can be picked in case Rich decides that that improvement is out of scope for this ticket.</p><p>I've reopened this issue based on early reports of breakage due to long file names. </p>
<p>Two reports:<br/>
<a href="https://groups.google.com/d/msg/clojure-dev/hnkJb9_il_M/4e5smM6mVlIJ">https://groups.google.com/d/msg/clojure-dev/hnkJb9_il_M/4e5smM6mVlIJ</a><br/>
<a href="https://groups.google.com/d/msg/clojure/hnkJb9_il_M/QOaTdCo5wmkJ">https://groups.google.com/d/msg/clojure/hnkJb9_il_M/QOaTdCo5wmkJ</a></p>
<p>Here's an example of a class name that is too long on Ubuntu 12.04 LTS 64bit / Java8 - reported max file size is 143 chars: </p>
<p><a href="https://github.com/ska2342/nested-fn-breaks-clojure-17/blob/master/src/nested_fn_breaks_clojure_17/core.clj">https://github.com/ska2342/nested-fn-breaks-clojure-17/blob/master/src/nested_fn_breaks_clojure_17/core.clj</a></p>
<p>With 1.6.0: (95 chars)<br/>
core$this_function_breaks_with_clojure_1_7$my_anonymous_function_<em>18$iter</em><em>19</em><em>23$fn</em><em>24$fn</em>_25</p>
<p>With 1.7.0-alpha3: (144 chars)<br/>
core$this_function_breaks_with_clojure_1_7$my_anonymous_function_<em>my_anonymous_function</em><em>19$iter</em><em>4951</em><em>auto</em>__<em>iter</em><em>20</em><em>24$fn</em><em>25$fn</em>_26.class</p>
<p>With the alternate patch here, the name would be: (95 chars)<br/>
core$this_function_breaks_with_clojure_1_7$my_anonymous_function_<em>19$iter</em><em>20</em><em>24$fn</em><em>25$fn</em>_26</p>
<p>patch "0001&#45;CLJ&#45;1330&#45;remove&#45;local&#45;binding&#45;name&#45;enhancement.patch" has the same effect of reverting f149260c14a75367dc9eba91cbe9b78110113566 and applying "0001&#45;CLJ&#45;1093&#45;v3&#45;no&#45;locals&#45;improv.patch" in case this is preferable</p><p>The tiny and unusual max file size of 143 is standard in the Ubuntu 12.04 crypto container for the home directory. You can get it for any directory with 'getconf NAME_MAX /path/to/dir'.</p>
<p>My initial problem (other than the file to reproduce on github) was triggered by the fns in a for-expression. Don't know if that makes any difference for you. </p>ApprovalOkGlobal RankPatchCode[CLJ-1315] Don't initialize classes when importing themhttp://dev.clojure.org/jira/browse/CLJ-1315
Clojure<p><b>Problem:</b> When classes are imported in Clojure, the class is loaded using Class.forName(), which causes its static initialisers to be executed. This differs from Java where compilation does not cause classes to be loaded. </p>
<p><b>Motivation:</b> In many cases when those classes are normally loaded by Java code during execution of a framework of some kind (IntelliJ in my case, and RoboVM is another culprit mentioned in that thread) the initialiser expects some infrastructure to be in place and will fail when it's not. This means that it's impossible to AOT compile namespaces importing these classes, which is a fairly serious limitation.</p>
<p><b>Approach:</b> Modify ImportExpr to call RT.classForNameNonLoading() instead of Class.forName(), which will load the class but not initialise it. This change causes the Clojure test suite to fail, since clojure.test-clojure.genclass imports a gen-class'ed class which no longer loads its namespace on initialisation. I'm not sure if this is considered an incorrect use of such a class (IIRC with records it's required to import the class and require its namespace), but given that it's in the Clojure test case it's reasonable to assume that this fix would be a breaking change for more code out there. This test failure is also corrected in the attached patch.</p>
<p><b>Patch:</b> 0001-Don-t-initialize-classes-during-import.patch</p>
<p><b>Screened by:</b> Alex Miller - I have tested many open source Clojure projects with this change (particularly seeking out large, complicated, or known users of genclass/deftype/etc) and have found no projects adversely impacted. I know that Cursive has been running with this modification for a long time with no known issues. I am ok with unconditionally enabling this change (re the comment below). The impact is described in more detail in the suggested changelog diff in the comments below.</p>
<p><b>Alternative:</b> This patch enables the change unconditionally, but depending on the extent of breakage it causes, it might need to be enabled with a configuration flag. I propose we make it unconditional in an early 1.7 beta and monitor the fall-out.</p>
<p><b>Background:</b> This issue has been discussed in the following threads<br/>
<a href="https://groups.google.com/d/topic/clojure/tWSEsOk_pM4/discussion">https://groups.google.com/d/topic/clojure/tWSEsOk_pM4/discussion</a><br/>
<a href="https://groups.google.com/forum/#!topic/clojure-dev/qSSI9Z-Thc0">https://groups.google.com/forum/#!topic/clojure-dev/qSSI9Z-Thc0</a></p>CLJ-1315Don't initialize classes when importing themEnhancementCriticalClosedCompletedUnassignedAaron CohenaotcompilerinteropSat, 28 Dec 2013 21:12:49 -0600Tue, 7 Oct 2014 12:32:40 -0500Tue, 7 Oct 2014 12:32:40 -0500Release 1.1Release 1.2Release 1.3Release 1.4Release 1.5Release 1.796<p>From original post:</p>
<p>This issue was originally reported by Zach Oakes and Colin Fleming and this patch was also tested by Colin.</p>
<p>I'm duplicating here my suggested release notes for this issue, which includes my current thoughts on potential breakage (it's also in the commit message of the patch):</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre> "import" no longer causes the imported class to be initialized. This
change better matches Java's import behavior and allows the importing of
classes that do significant work at initialization time which may fail.
This semantics change is not expected to effect most code, but certain
code may have depended on behavior that is no longer true.
1) importing a Class defined via gen-class no longer causes its defining
namespace to be loaded, loading is now deferred until first reference. If
immediate loading of the namespace is needed, "require" it directly.
2) Some code may have depended on import to initialize the class before it
was used. It may now be necessary to manually call (Class/forName
"org.example.Class") when initialization is needed. In most cases, this
should not be necessary because the Class will be initialized
automatically before first use.
</pre>
</div></div><p>I'm not sure if this should also be fixed, but it would be nice if you could emit the code for a proxy of one of these non-initialized classes without forcing initialization. For example, the following raises an exception (I'm using Java 8):</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">Clojure 1.6.0
user=&gt; (def cname <span class="code-quote">"javafx.scene.control.ListCell"</span>)
#'user/cname
user=&gt; (let [cls (<span class="code-object">Class</span>/forName cname <span class="code-keyword">false</span> (clojure.lang.RT/baseLoader))] (.importClass *ns* cls))
javafx.scene.control.ListCell
user=&gt; (defn fails [] (proxy [ListCell] [] (updateItem [item empty] (proxy-<span class="code-keyword">super</span> item empty))))
CompilerException java.lang.ExceptionInInitializerError, compiling:(NO_SOURCE_PATH:3:16)</pre>
</div></div>
<p>The exception was ultimately caused by "IllegalStateException Toolkit not initialized", which javafx throws if you attempt to initialize a Control class outside of <tt>Application.launch</tt>.</p>
ApprovalOkGlobal RankPatchCode[CLJ-1309] Bindings after :as in list destructuring should throw errorhttp://dev.clojure.org/jira/browse/CLJ-1309
Clojure<p>If you try to define a vector binding with anything at all after an :as parameter, you do not get a compiler error, and the binding is silently swallowed:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user&gt; ((fn [[:as y z]] y) [1 2])
[1 2]</pre>
</div></div>
<p>If you try to actually use the binding, there will be a compiler error (the compiler will complain that there's no binding for the symbol), but the actual error has already happened, and should be reported earlier.</p>CLJ-1309Bindings after :as in list destructuring should throw errorEnhancementMinorOpenUnresolvedUnassignedben wolfsonCompilererrormsgsThu, 19 Dec 2013 18:25:17 -0600Wed, 5 Feb 2014 12:00:16 -0600Release 1.4Release 1.501Global Rank[CLJ-1301] case expression fails to match a BigDecimal http://dev.clojure.org/jira/browse/CLJ-1301
Clojure<p>In 1.5.1 (anywhere before the <a href="http://dev.clojure.org/jira/browse/CLJ-1118" title="inconsistent numeric comparison semantics between BigDecimal and other Numerics"><del>CLJ-1118</del></a> patch), this is the behavior on BigDecimal case matching:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (defn t [v] (<span class="code-keyword">case</span> v 1 <span class="code-quote">"<span class="code-object">Long</span>"</span> 1.0M <span class="code-quote">"BigDecimal"</span> <span class="code-quote">"none"</span>))
#'user/t
user=&gt; (map t [1 1.0M 1.00M])
(<span class="code-quote">"<span class="code-object">Long</span>"</span> <span class="code-quote">"BigDecimal"</span> <span class="code-quote">"none"</span>)</pre>
</div></div>
<p>In 1.6 the behavior (post <a href="http://dev.clojure.org/jira/browse/CLJ-1118" title="inconsistent numeric comparison semantics between BigDecimal and other Numerics"><del>CLJ-1118</del></a> patch) has changed:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (defn t [v] (<span class="code-keyword">case</span> v 1 <span class="code-quote">"<span class="code-object">Long</span>"</span> 1.0M <span class="code-quote">"BigDecimal"</span> <span class="code-quote">"none"</span>))
#'user/t
user=&gt; (map t [1 1.0M 1.00M])
(<span class="code-quote">"<span class="code-object">Long</span>"</span> <span class="code-quote">"none"</span> <span class="code-quote">"none"</span>)</pre>
</div></div>
<p>In 1.6 after <a href="http://dev.clojure.org/jira/browse/CLJ-1118" title="inconsistent numeric comparison semantics between BigDecimal and other Numerics"><del>CLJ-1118</del></a>, I expect to see: ("Long" "BigDecimal" "BigDecimal") as they now have the same hash and hasheq.</p>
<p><b>Cause:</b> The case constants are hashed in the clojure.core/case macro using clojure.core/hash which calls clojure.lang.util/hasheq(). In Compiler.emitExprForHashes(), a call to clojure.lang.Util/hash(). In Clojure 1.5 these hash values are the same (hash of 1.0M == hasheq of 1.0M == 311). In Clojure 1.6, they are different (hash of 1.0M = 311, hasheq of 1.0M = 31).</p>
<p>In any cases where Java's hashCode and Clojure's hasheq return different values, the case statement can fail to do the correct thing.</p>
<p><b>Approach:</b> Change Compiler.java to use clojure.lang.Util hasheq() to match the case macro use of clojure.core/hash (which calls clojure.lang.Util.hasheq()).</p>
<p><b>Patch:</b> clj-1301-1.diff</p>
<p><b>Screened by:</b> </p>CLJ-1301case expression fails to match a BigDecimal DefectBlockerClosedCompletedUnassignedAndy FingerhutCompilerSat, 23 Nov 2013 16:36:01 -0600Sun, 26 Jan 2014 09:03:37 -0600Sun, 26 Jan 2014 09:03:37 -0600Release 1.5Release 1.6Release 1.623<p>Patch clj-1301-1.diff modifies Compiler.java so that case* statements use hasheq on the test expression value, rather than Java's hashCode. It also adds a test case that currently fails with latest Clojure master, but passes with the patch.</p><p>This bug is also the root cause for the recent failures of tests for the test.generative library.</p><p>Putting in 1.6 release per Rich.</p><p>Andy, I talked to Rich and the conclusion was that we should make the opposite change here such that the case macro should route to the Java hashcode version clojure.lang.util.hash() and the Compiler should be left as is. Can you update the patch?</p><p>And in case you were wondering, the reason is that the Java hashcode is generally faster (case is all about speed) and there are easy opportunities for you to properly cast your expression and/or case constants (where as the situations with collections where boxing is difficult to fix generically, that is not true).</p><p>Alex, unless I am missing something, changing case to use Java's hashCode() would also require changing its current equality comparison from Clojure = (aka equiv()) to something consistent with hashCode(), which I think must be Java's equals().</p>
<p>Such a change would mean that all of the things that are = but not equals() will not match each other in a case statement, e.g. a case value of (Integer. 5) will not match a (Long. 5) value to compare against in a case branch.</p>
<p>Is that really what is desired here? I almost hesitate to create such a patch, for fear it might be committed <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p><p>Based on discussion comments, move back to Incomplete until we resolve.</p><p>Added better example demonstrating the problem (the specific problem exposed by <a href="http://dev.clojure.org/jira/browse/CLJ-1118" title="inconsistent numeric comparison semantics between BigDecimal and other Numerics"><del>CLJ-1118</del></a>). </p><p>Simplified examples.</p><p>Re Andy's comments above, I walked down that path a bit and built such a patch, however we currently have tests in clojure.test-clojure.control:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(testing <span class="code-quote">"test number equivalence"</span>
(is (= :1 (<span class="code-keyword">case</span> 1N 1 :1 :<span class="code-keyword">else</span>))))</pre>
</div></div>
<p>which clearly seems to expect Clojure equiv() behavior over Java equals() behavior in case constant matching. So either that is a bad test or this is not a viable approach (it also suggests we could break existing code with this change). </p><p>One could consider having the default behavior of case to use hasheq and clojure.core/= everywhere, but add a 'fast' option to use hashCode and Java equals.</p><p>Alternative patch in the direction of using hashcode/equals instead of hasheq/equiv. Note that this test causes some test failures. This is not yet a candidate patch - further work needs to be done in evaluating this path.</p>ApprovalIncompleteGlobal RankPatchCode and Test[CLJ-1297] try to catch using - instead of _ in filenames so the compiler can give a better error message for people who don't know that you need to use _ in file nameshttp://dev.clojure.org/jira/browse/CLJ-1297
Clojure<p><b>Problem:</b> Clojure requires the files that back a namespace that has dashes in it to have the dashes replaced with underscores on the filesystem (ie a.b_c.clj for namespace a.b-c). If you require a file that has been mistakenly saved as b-c.clj instead, you will get an error message:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">Exception in thread <span class="code-quote">"main"</span> java.io.FileNotFoundException: Could not locate a/b_c__init.class or a/b_c.clj on classpath:
...</pre>
</div></div>
<p><b>Proposed:</b><br/>
Fix the bad ending colon in this sentence and add a second sentence only when the file name has an _ in it: "Please check that namespaces with dashes use underscores in the Clojure file name."</p>
<p><b>Patch:</b> clj-1297-v5.patch</p>
<p><b>Screened by:</b> Alex Miller</p>CLJ-1297try to catch using - instead of _ in filenames so the compiler can give a better error message for people who don't know that you need to use _ in file namesEnhancementMajorClosedCompletedUnassignedKevin DowneycompilererrormsgsTue, 19 Nov 2013 16:09:39 -0600Tue, 7 Oct 2014 12:32:40 -0500Tue, 7 Oct 2014 12:32:40 -0500Release 1.7116<p>A perhaps even better solution would be to simply allow the use of dashes in *.clj<span class="error">&#91;s&#93;</span> filenames. I can't imagine the extra disk access per-namespace would be a huge performance burden, and (since dashes aren't allowed currently) I don't think there would be any issues with backwards compatibility.</p><p>It's worth mentioning the combinatorial explosion for namespaces with multiple dashes &#8211; if I <tt>(require 'foo-bar.baz-bang)</tt>, should clojure search for all four possible filenames? Does the jvm have a way to search for files by regex or similar to avoid nasty degenerate cases (like <tt>(require 'foo-------------)</tt>)?</p><p>According to the docs, the FileSystem class's "getPathMatcher" method accepts path globs, so you'd merely have to replace each instance of "<tt>&#45;</tt>" or "<tt>&#95;</tt>" with "{<tt>-,_</tt>}". Actual runtime characteristics would likely depend on the underlying filesystem's implementation.</p><p>I don't think the FileSystem stuff applies when looking up classes on the classpath. Note that Java class names cannot contain "-". </p><p>According to the spec, Java class names can't contain dashes (though IIRC OpenJDK and Oracle's JDK accept them anyway) but the requirement that Clojure source files have names which align with their AOT'd class file eqivalents is something we've imposed upon ourselves. Introducing the disconnect between .clj files and .class files makes way more sense than disconnecting namespaces and .clj files, but arguably it's too late to fix that mistake.</p>
<p>In any case a check for dashed files (resulting only in a more informative compiler error, not a more permissive compiler) which only triggers when a .clj file cannot be found imposes zero overhead in the case where things are already working.</p><p>As Clojure seems to be idiomatic to have sometimes-dashed-namespace-and-function-names as opposed to the ubiquitous camelCaseFunctionNames in java ... I agree to have the compiler automagically handle 'knowing' to look in dir_struct AND dir-struct for requisite files. </p>
<p>or at the least print out a nice message explaining the quirk when files "can't" be found ... WHEN there are dashes and underscores involved... anything to aid in helping things "just work" as one would think they're supposed to.</p><p>I would have saved a few hours as well.</p><p>This patch changes clojure.core/load such that:</p>
<ul class="alternate" type="square">
<li>When loading the resource-root of lib throws a FileNotFoundException, the lib is analyzed...</li>
<li>... if the lib was a name that would be munged, it examines the combinatorial explosion of munge candidates and .clj or .class files in the classpath ...</li>
<li>... if any of these candidates exist, it informs the user of the file's existance, and that a change to that filename would lead to that resource being loaded.</li>
<li>... if none of these candidates exist, it throws the original exception.</li>
</ul>
<p>It also modifies clojure.lang.RT to expose the behavior around finding clj or class files from a resource root.</p><p>I do not know whether it handles all of the cases proposed in this discussion, but I encourage folks to check out the filename/namespace consistency checking in the latest Eastwood release (version 0.1.1) to see if it catches the cases they would hope to catch. It does a static check based on the files in a Leiningen project, nothing at run time. <a href="https://github.com/jonase/eastwood">https://github.com/jonase/eastwood</a></p>
<p>Of course changes to Clojure itself to give warnings about such things can still be very useful, since not everyone will be using a 3rd party tool to check for such things.</p><p>Re the screener's note at the top, my preference would be for the simpler approach.</p><p>I see no reason to fish around in the file system at all. Why can't the message simply remind people that underscores are required and to check that they aren't using dashes?</p><p>The tradeoff is that fishing around in the file system means the error message is only shown if the user likely made the relevant mistake, whereas simply showing the error message would (if I understand correctly) mean this reminder gets shown for every require error, which I would estimate happens a whole lot more often than the mistake in question.</p><p>Gary, there is an exception thrown in any case if the load fails. One approach that I am hacking up now is to add to the existing exception's message that maybe they need to replace - chars in file name with _, but <b>only</b> if the name attempted to be loaded has at least one '-' character in it. So no new output, except in an exception already being thrown, and then only if there is at least a possibility that this is the problem.</p><p>clj-1297-v2.patch is similar to the previously proposed patch, but does not touch the file system in any way that wasn't already being done before this patch.</p>
<p>It adds an extra hint to the message of the exception already being thrown if the resource is not found, but only if there is a '-' character in the name that failed to load.</p><p>clj-1297-v3.patch is nearly identical to clj-1297-v2.patch, described in the previous comment, except it eliminates an unnecessary let expression.</p><p>With latest:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (require 'a-b)
FileNotFoundException Could not locate a_b__init.class or a_b.clj on classpath: Perhaps a file you are attempting to load needs - chars in name replaced with _ clojure.core/load-one/fn--5135 (core.clj:5606)</pre>
</div></div>
<p>That looks goofy due to the base message in RT.load(): <tt>throw new FileNotFoundException(String.format("Could not locate %s or %s on classpath: ", classfile, cljfile));</tt></p>
<p>I would like to:<br/>
1) Fix that RT.load() message to not end with a colon: "Could not locate %s or %s on classpath."<br/>
2) Instead of changing load-one, just add the additional sentence in RT.load(). Second optional sentence could be: "Please check that namespaces with dashes use underscores in the Clojure file name."</p>
<p>Final message would then look like: </p>
<p>"Could not locate a_b__init.class or a_b.clj on classpath. Please check that namespaces with dashes use underscores in the Clojure file name."</p><p>Patch clj-1297-v4.patch modifies only RT.load, including an extra message only if the file name in the argument contains a '_' character in it, with the message suggested by Alex Miller in his last comment before this.</p><p>The bad end colon is not fixed and the optional message is not included in the format string.</p><p>clj-1297-v5.patch should be the one. I can only attempt to blame the previous bone-headed failure on lack of sleep, but even then ...</p>ApprovalOkGlobal RankPatchCode[CLJ-1279] Fix confusing macroexpand1 ArityException handlinghttp://dev.clojure.org/jira/browse/CLJ-1279
Clojure<p>macros can give very confusing error messages when they execute a form which generates an ArityException. clojure.lang.Compiler.macroexpand1 assumes that any ArityException comes from the call to the macro itself, which need not be the case. For instance:</p>
<p> user&gt; (do (defmacro f [] (assoc)) (f))<br/>
ArityException Wrong number of args (-2) passed to: core$assoc clojure.lang.Compiler.macroexpand1 (Compiler.java:6488)<br/>
user&gt; (use 'clojure.repl) (pst)<br/>
nil<br/>
ArityException Wrong number of args (-2) passed to: core$assoc<br/>
clojure.lang.Compiler.macroexpand1 (Compiler.java:6488)<br/>
clojure.lang.Compiler.macroexpand (Compiler.java:6544)<br/>
clojure.lang.Compiler.eval (Compiler.java:6618)<br/>
clojure.lang.Compiler.eval (Compiler.java:6624)<br/>
clojure.lang.Compiler.eval (Compiler.java:6597)<br/>
clojure.core/eval (core.clj:2864)<br/>
clojure.main/repl/read-eval-print-<del>6596/fn</del>-6599 (main.clj:260)<br/>
clojure.main/repl/read-eval-print--6596 (main.clj:260)<br/>
clojure.main/repl/fn--6605 (main.clj:278)<br/>
clojure.main/repl (main.clj:278)<br/>
clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn--1251 (interruptible_eval.clj:56)<br/>
clojure.core/apply (core.clj:617)<br/>
nil</p>
<p>Easy enough to see the source of the problem in this case, but because both the number of arguments actually passed is off by two, and the stacktrace element for the call to assoc has been dropped, this shortcut by macroexpand1 can get super confusing.</p>
<p>The attached patch corrects this behavior. E.g.</p>
<p> user=&gt; (do (defmacro f [] (assoc)) (f))<br/>
ArityException Wrong number of args (0) passed to: core$assoc clojure.lang.AFn.throwArity (AFn.java:437)<br/>
user=&gt; (use 'clojure.repl) (pst)<br/>
nil<br/>
ArityException Wrong number of args (0) passed to: core$assoc<br/>
user/f (NO_SOURCE_FILE:1)<br/>
clojure.lang.Var.invoke (Var.java:419)<br/>
clojure.lang.Var.applyTo (Var.java:532)<br/>
clojure.lang.Compiler.macroexpand1 (Compiler.java:6507)<br/>
clojure.lang.Compiler.macroexpand (Compiler.java:6580)<br/>
clojure.lang.Compiler.eval (Compiler.java:6654)<br/>
clojure.lang.Compiler.eval (Compiler.java:6660)<br/>
clojure.lang.Compiler.eval (Compiler.java:6633)<br/>
clojure.core/eval (core.clj:2864)<br/>
clojure.main/repl/read-eval-print-<del>6594/fn</del>-6597 (main.clj:260)<br/>
clojure.main/repl/read-eval-print--6594 (main.clj:260)<br/>
clojure.main/repl/fn--6603 (main.clj:278)<br/>
nil</p>CLJ-1279Fix confusing macroexpand1 ArityException handlingDefectMajorOpenUnresolvedUnassignedAlex CoventryCompilererrormsgsmacroWed, 16 Oct 2013 22:12:55 -0500Wed, 5 Feb 2014 12:25:20 -0600Release 1.632<p>Patch with test</p><p>Amended patch to deal more gracefully with unexpected stack trace structure.</p><p>Also see <a href="http://dev.clojure.org/jira/browse/CLJ-397" title="better error message when calling macros with arity"><del>CLJ-397</del></a> and <a href="http://dev.clojure.org/jira/browse/CLJ-383" title="different arrities of macro don&#39;t work in 1.2 (20100607 build)"><del>CLJ-383</del></a>. </p><p>Thanks, Alex. It would be easy enough to move most of the logic into ArityException, which would be a compromise between Stu's<span class="error">&#91;1&#93;</span> options 1 and 2. Is that worth doing?</p>
<p>Amending clojure.lang.AFn.throwArity to check whether "this" is a macro and adjust the arg count there accordingly might be the simplest way. I can see why Rich prefers all the logic to go into ArityException, but since ArityExceptions are used for things other than macros, I don't see a way to make an honest error message there without groveling the stack trace.</p>
<p><span class="error">&#91;1&#93;</span> <a href="http://dev.clojure.org/jira/browse/CLJ-397?focusedCommentId=24090&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-24090">http://dev.clojure.org/jira/browse/CLJ-397?focusedCommentId=24090&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-24090</a></p><p>I would have to take more time than I have to make an informed opinion but I can say that from a general point of view inspecting StackTraceElements does not seem like the right solution to (almost) any problem. <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/> </p><p>This patch causes Var.setMacro to set instance attribute AFn.macrop to true, so that AFn.throwArity can reduce the number of arguments reported.</p>
<p>I'm not used to negotiating java class hierarchies, so it's possible there's a cleaner way. Since Var.fn() returns an IFn, I added macrop handling methods IFn.setMacro and IFn.isMacro. These then needed to be implemented in Ref and Keyword, as well as AFn (where I wanted them) because they implement the IFn interface but don't inherit from AFn.</p>
<p>The real drawback I see with this approach is the duplicated state, though: ^{:macro true} vs AFn.macrop==true.</p><p>I have not investigated the reason yet, but neither patch applies cleanly after the latest commits to Clojure master on Oct 25 2013. Given that what kinds of solution methods would be acceptable for this issue, it sounds like more thinking and code changes are probably needed anyway before worrying too much about that.</p>Global RankPatchCode and Test[CLJ-1274] Unable to set compiler options via system properties except for AOT compilationhttp://dev.clojure.org/jira/browse/CLJ-1274
Clojure<p>The code that converts JVM system properties into keys under the &#42;compiler-options&#42; var is present only inside the clojure.lang.Compile class. This is a problem when using a debugger inside an IDE and not AOT compiling; specifying <tt>-Dclojure.compiler.disable-locals-clearing=true</tt> has no effect here when it would be most useful!</p>
<p><b>Patch:</b> <a href="http://dev.clojure.org/jira/browse/CLJ-1274" title="Unable to set compiler options via system properties except for AOT compilation"><del>CLJ-1274</del></a>.patch<br/>
<b>Screened by:</b> Stu</p>CLJ-1274Unable to set compiler options via system properties except for AOT compilationDefectMajorClosedCompletedUnassignedHoward Lewis ShipcompilerWed, 2 Oct 2013 16:46:35 -0500Fri, 29 Aug 2014 09:24:13 -0500Fri, 29 Aug 2014 09:24:13 -0500Release 1.5Release 1.731<p>Obviously, that's supposed to be <tt>&#42;compiler-options&#42;</tt>.</p><p>Changes initialization of <tt>*compiler-options&#42;</tt> to occur statically inside Compiler; now available to all forms of Clojure, not just AOT compilation; however, the initial <tt>&#42;compiler-options&#42;</tt> value is now defined as a root binding, rather than a per-thread binding, which has slightly different semantics.</p><p>Patch is straightforward, marking screened.</p>
<p>I am left wondering if other options that are set only in Compile.java ought also to be moved.</p>ApprovalOkGlobal RankPatchCode[CLJ-1263] Allow static compilation of function invocationshttp://dev.clojure.org/jira/browse/CLJ-1263
Clojure<p>This proposal is to allow metadata on functions to prevent a fully dynamic var deref to be used whenever the function is called. </p>
<p>When the function is invoked, JVM "invokevirtual" instruction will be used, which is faster than the current implementation (var deref + IFn cast + invokinterface) and has less restrictions (no need to predefine interfaces to match the function parameters). The JVM is generally able to compile such invokevirtual instructions into extremely efficient code - effectively as fast as pure Java.</p>
<p>This is intended to pave the way to better support for statically compiled, high performance code. In particular, it allow:</p>
<ul class="alternate" type="square">
<li>Supporting arbitrary JVM primitives (float, int, byte, char etc.) as well as just double/long.</li>
<li>Supporting typed return values e.g. "String". This could eliminate many casts and type checks.</li>
<li>Supporting typed reference arguments (e.g. String).</li>
</ul>
<p>Suggested usage:</p>
<p>(defn ^:static foo ^int <span class="error">&#91;^String a ^String b&#93;</span><br/>
(+ (count a) (Integer/parseInt b)))</p>
<p>Existing code / semantics should not be affected</p>CLJ-1263Allow static compilation of function invocationsEnhancementMajorOpenUnresolvedUnassignedMike AndersoncompilerSat, 14 Sep 2013 23:31:10 -0500Thu, 7 Nov 2013 12:48:46 -060025<p>Very nice! That is what would really improve experience with certain tasks. I think it will also make possible to work with primitive arrays without the conversions?</p><p>Hi Alex - which aspect of "work with primitive arrays" are you referring to? This feature would certainly help with passing primitive arguments to/from functions that use primitive arrays. It would also potentially help avoid some casts of primitive array arguments themselves. I don't think it helps in any other way - perhaps a separate issue would be appropriate if there is another thing you are trying to do?</p><p>this issue is confusing, because there was/is a :static feature in clojure(which seems to be disabled in the compiler at the moment) and this proposal doesn't mention the existing work at all.</p>
<p>I also think this proposal is begging the question, there is no discussion of other possible solutions to the performance problem (whatever that is) that this is trying to solve.</p>
<p>the (var.deref()(IFn)).invoke(...) is pretty fundamental to the feel of clojure, in fact the existing :static keyword seems to be disabled in the compiler exactly because it complicates those semantics. so we should have a very clear proposal (not a wishlist) if we want to change that with some very clear wins.</p>
<p>maybe an optimizing clojure compiler would be a better approach.</p><p>Hi Kevin,</p>
<p>This is partly in response to this discussion on Clojure-Dev, where we discussed there are quite a lot of performance issues around the way that Clojure passes arguments currently:<br/>
<a href="https://groups.google.com/d/topic/clojure/H5P25eYKBj4/discussion">https://groups.google.com/d/topic/clojure/H5P25eYKBj4/discussion</a></p>
<p>Also I believe it reinstates the original intention of "^:static": I can't find where this is/was officially documented, but Arthur's answer in this SO question suggests that this was the case:<br/>
<a href="http://stackoverflow.com/questions/7552632/what-does-static-do-in-clojure">http://stackoverflow.com/questions/7552632/what-does-static-do-in-clojure</a></p>
<p>I think the proposal is relatively clear: it's probably the minimal change required to get static/direct (i.e. not via an indirect var reference / IFn) function invocations without affecting any of the semantics of current code.</p>
<p>This is sufficiently important for me that it's preventing me from shifting some performance-critical code to Clojure (even with primitive type hints). e.g. here's a simple case with a small primitive function:</p>
<p>(defn ^long foo <span class="error">&#91;^long x&#93;</span><br/>
(inc x))</p>
<p>(c/quick-bench (dotimes <span class="error">&#91;i 100&#93;</span> (foo i))) ;; c = criterium<br/>
=&gt; Execution time mean : 194.334293 ns</p>
<p>(c/quick-bench (dotimes <span class="error">&#91;i 100&#93;</span> (inc i)))<br/>
=&gt; Execution time mean : 71.539048 ns</p>
<p>i.e. the indirect function invocation is costing us nearly 170% overhead. In Java the equivalent functions perform identically: the overhead is zero because with static function invocation the JVM JIT is able to eliminate all the function call overhead.</p>
<p>In the long term, I agree that a proper optimising compiler would be the best way forward (perhaps Clojure 2.0/CinC can give us this?) but in the meantime I think this is a pragmatic way to improve performance with minimal impact on existing code. Even with an optimising compiler, I think we' would need some way to specifiy the "optimised" semantics rather than the indirect var deref behaviour, and "^:static" seems like a reasonable way to do so (unless anyone has a better idea?)</p>
<p>have you looked at the definition of int and how it uses :inline/definline to avoid the call overhead?</p><p>Good point Kevin - :inline and definline seem like a good approach in many cases (although it's marked as "experimental" - does that mean we can't rely on it to work in future releases?).</p>
<p>This proposal is still somewhat different: the inline solutions and its variants are effectively doing macro expansion to generate code without a function call on the Clojure side. The approach in this proposal would still emit a function call in bytecode, but do so in a way that the JVM can subsequently inline and optimise much more efficiently. Both have their uses, I think?</p>
<p>Commented edited Nov 7 2013 by Andy Fingerhut: Regarding definline marked as experimental, it has been so marked since Clojure 1.0's release, and the plan is to keep it marked that way in the pending Clojure 1.6 release. See discussion thread on <a href="http://dev.clojure.org/jira/browse/CLJ-1281" title="Clojure 1.6 - reconsider what is &quot;alpha&quot; in core"><del>CLJ-1281</del></a>. No plans to remove it that I am aware of.</p><p>my point is your benchmark above is not a comparison of clojure's current deref + cast + invoke vs. invokevirtual, inc is being inlined in to a static method call there</p><p>I've been noodling around this, and it is entirely possible to generate and invoke code in clojure right now without paying the extra deref() cost:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre> (defn ^long fib [^long n]
(case n
0 0
1 1
(+ (fib (dec n))
(fib (dec (dec n))))))
</pre>
</div></div>
<p>can be written as </p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(declare TheR1798)
(definterface I1797
(^long fib_Invoke_1 [^long n]))
(deftype R1798 []
I1797
(^long fib_Invoke_1
[this1799 ^long n]
(case n
0 0
1 1
(+ (.fib_Invoke_1 this1799 (dec n))
(.fib_Invoke_1 this1799 (dec (dec n)))))))
(def TheR1798 (new R1798))
(defn ^long fib [^long n]
(.fib_Invoke_1 TheR1798 n)))
</pre>
</div></div>
<p>now the recursive calls are invokeinterfaces, and the resulting function seems to have mean execution time about 5 times smaller using criterium to bench mark</p>
<p>it is entirely possible to write a macro that translates one in to other, and the weird names in the above are because I have a little proof of concept that does that.</p>
<p>the body of the bytecode for the regular fib function first shown looks something like:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre> public final java.lang.Object invokePrim(long);
Code:
0: lload_1
1: lstore_3
2: lload_3
3: l2i
4: tableswitch { // 0 to 1
0: 28
1: 40
default: 52
}
28: lconst_0
29: lload_3
30: lcmp
31: ifne 52
34: getstatic #33 // Field const__1:Ljava/lang/Object;
37: goto 94
40: lconst_1
41: lload_3
42: lcmp
43: ifne 52
46: getstatic #37 // Field const__3:Ljava/lang/Object;
49: goto 94
52: getstatic #57 // Field const__5:Lclojure/lang/Var;
55: invokevirtual #70 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
58: checkcast #6 // class clojure/lang/IFn$LO
61: lload_1
62: invokestatic #75 // Method clojure/lang/Numbers.dec:(J)J
65: invokeinterface #77, 3 // InterfaceMethod clojure/lang/IFn$LO.invokePrim:(J)Ljava/lang/Object;
70: getstatic #57 // Field const__5:Lclojure/lang/Var;
73: invokevirtual #70 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
76: checkcast #6 // class clojure/lang/IFn$LO
79: lload_1
80: invokestatic #75 // Method clojure/lang/Numbers.dec:(J)J
83: invokestatic #75 // Method clojure/lang/Numbers.dec:(J)J
86: invokeinterface #77, 3 // InterfaceMethod clojure/lang/IFn$LO.invokePrim:(J)Ljava/lang/Object;
91: invokestatic #81 // Method clojure/lang/Numbers.add:(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Number;
94: areturn
LineNumberTable:
line 243: 0
line 244: 2
line 247: 52
line 247: 52
line 247: 61
line 248: 70
line 248: 79
line 248: 79
LocalVariableTable:
Start Length Slot Name Signature
2 92 3 G__301 J
0 94 0 this Ljava/lang/Object;
0 94 1 n J
public java.lang.Object invoke(java.lang.Object);
Code:
0: aload_0
1: aload_1
2: checkcast #89 // class java/lang/Number
5: invokestatic #93 // Method clojure/lang/RT.longCast:(Ljava/lang/Object;)J
8: invokeinterface #77, 3 // InterfaceMethod clojure/lang/IFn$LO.invokePrim:(J)Ljava/lang/Object;
13: areturn
</pre>
</div></div>
<p>the body of the "optimized" version looks like:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre> public long fib_Invoke_1(long);
Code:
0: lload_1
1: lstore_3
2: lload_3
3: l2i
4: tableswitch { // 0 to 1
0: 28
1: 38
default: 48
}
28: lconst_0
29: lload_3
30: lcmp
31: ifne 48
34: lconst_0
35: goto 80
38: lconst_1
39: lload_3
40: lcmp
41: ifne 48
44: lconst_1
45: goto 80
48: aload_0
49: checkcast #6 // class com/thelastcitadel/kernel/I2364
52: lload_1
53: invokestatic #53 // Method clojure/lang/Numbers.dec:(J)J
56: invokeinterface #55, 3 // InterfaceMethod com/thelastcitadel/kernel/I2364.fib_Invoke_1:(J)J
61: aload_0
62: checkcast #6 // class com/thelastcitadel/kernel/I2364
65: lload_1
66: invokestatic #53 // Method clojure/lang/Numbers.dec:(J)J
69: invokestatic #53 // Method clojure/lang/Numbers.dec:(J)J
72: invokeinterface #55, 3 // InterfaceMethod com/thelastcitadel/kernel/I2364.fib_Invoke_1:(J)J
77: invokestatic #59 // Method clojure/lang/Numbers.add:(JJ)J
80: lreturn
LineNumberTable:
line 251: 0
line 251: 2
line 251: 48
line 251: 48
line 251: 52
line 251: 61
line 251: 65
line 251: 65
LocalVariableTable:
Start Length Slot Name Signature
2 78 3 G__2363 J
0 80 0 this Lcom/thelastcitadel/kernel/R2365;
0 80 1 n J
</pre>
</div></div>
<p>so the calls are not invokevirtual (due to the way clojure compiles stuff, you cannot type anything inside a record as being that record's type), but the interface is unique and only has one instance, so I think the jvm's class hierarchy analysis makes short work of that.</p>
<p>if I have time I may try and complete my macro and release it as a library, but given tools.analyzer.jvm someone should be able to do better than my little proof of concept very quickly.</p><p>I don't know if my editing of Mike Anderson's Nov 5 2013 comment is notified to people watching this ticket, so adding a new comment so those interested in definline's experimental status can know to go back and re-read it.</p>Global Rank[CLJ-1256] Support type-hinted overrides of function parametershttp://dev.clojure.org/jira/browse/CLJ-1256
Clojure<p>Problem: in many cases, the Clojure compiler has enough information about the type of a function argument to statically emit maximally efficient code on the JVM (i.e. without instance? checks, type casts or other forms of dynamic polymorphic dispatch). We are currently unable to do so in Clojure, which pushes developers with strong performance requirements to use some unidiomatic or convoluted workarounds.</p>
<p>Proposal is simply to allow functions to take type-hinted overloads of function arguments, e.g.</p>
<p>(defn foo<br/>
(<span class="error">&#91;^double x&#93;</span> (Math/floor x))<br/>
(<span class="error">&#91;^float x&#93;</span> (Math/floor (double x)))<br/>
(<span class="error">&#91;^String s&#93;</span> (count s)))</p>
<p>An "Object" version of the code with the correct arity will always be emitted, which will maintain compatibility with the IFn interface and ensure that the function can still be used in dynamic / interactive contexts. If the "Object" version is not explicitly provided, then it will be generated to use instance? checks that subsequently delegate to the appropriate typed version of the function (or throw an InvalidArgumentException if no match is found).</p>
<p>Matching rules would be the same as Java.</p>
<p>This will be backwards compatible with all existing uses of defn. In particular, it should extend / enhance / supercede the existing handling of primitive functions.</p>
<p>In the future, this technique might be used alongside core.typed to ensure that the most efficient function version is chosen based on type inference.</p>CLJ-1256Support type-hinted overrides of function parametersEnhancementMajorOpenUnresolvedUnassignedMike AndersoncompilerinteroptypehintsFri, 6 Sep 2013 21:37:59 -0500Mon, 9 Sep 2013 13:44:02 -050022Global Rank[CLJ-1241] NPE when AOTing overrided clojure.core functionshttp://dev.clojure.org/jira/browse/CLJ-1241
Clojure<p>When performing AOT compilation on a namespace that overrides a clojure.core function without excluding the original clojure.core function from the ns, you get a NullPointerException. </p>
<p>To reproduce aot compile a namespace like "(ns x) (defn get [])"</p>
<p>For example:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">$ lein <span class="code-keyword">new</span> aot-get
$ cd aot-get
$ sed -i s/foo/get/
$ lein compile :all
WARNING: get already refers to: #'clojure.core/get in namespace: aot-get.core, being replaced by: #'aot-get.core/get
Exception in thread <span class="code-quote">"main"</span> java.lang.NullPointerException
at clojure.lang.<span class="code-object">Compiler</span>$ObjExpr.emitVar(<span class="code-object">Compiler</span>.java:4858)
at clojure.lang.<span class="code-object">Compiler</span>$DefExpr.emit(<span class="code-object">Compiler</span>.java:428)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7152)
at clojure.lang.<span class="code-object">Compiler</span>.compile(<span class="code-object">Compiler</span>.java:7219)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)</pre>
</div></div>
<p><b>Cause:</b> DefExpr.parse does not call registerVar for vars overridding clojure.core ones, thus when AOT compiling the var is not registered in the constant table.</p>
<p><b>Proposed:</b> The attached patch makes DefExpr.parse call registerVar for vars overridding clojure.core ones.</p>
<p><b>Patch:</b> 0001-fix-<a href="http://dev.clojure.org/jira/browse/CLJ-1241" title="NPE when AOTing overrided clojure.core functions"><del>CLJ-1241</del></a>.patch</p>
<p><b>Screened by:</b> Alex Miller</p>CLJ-1241NPE when AOTing overrided clojure.core functionsDefectMajorClosedCompletedUnassignedPhil HagelbergaotcompilerTue, 30 Jul 2013 13:56:10 -0500Fri, 29 Aug 2014 12:00:25 -0500Fri, 29 Aug 2014 12:00:25 -0500Release 1.5Release 1.6Release 1.726<p>DefExpr.parse was not calling registerVar for vars overridding clojure.core ones.</p><p>Verified on Clojure 1.5.1.</p><p>Reproduced with `key` function without `(:refer-clojure :exclude <span class="error">&#91;key&#93;</span>)`</p><p>This doesn't meet triage guidelines - i.e. there is this problem, therefore we will fix it by _____ so it then does _____</p><p>This is still present in the 1.6 release. I think it's mis-classified as low priority.</p><p>See for instance the cascalog mailing list: <a href="https://groups.google.com/forum/#!topic/cascalog-user/Pe5QIpmU0vA">https://groups.google.com/forum/#!topic/cascalog-user/Pe5QIpmU0vA</a></p><p>It may help if someone could clarify Rich's comment.</p>
<p>Does it mean that the ticket <b>should</b> include a plan of the form "therefore we will fix it by _____ so it then does _____", but this ticket doesn't have that?</p>
<p>Or perhaps it means that the ticket <b>should not</b> include a plan of that form, but this ticket does? If so, I don't see it, except perhaps the very last sentence of the description. If that is a problem for vetting a ticket, perhaps we could just delete that sentence and proceed from there?</p>
<p>Something else?</p><p>Andy, I added the two last lines in the description after reading Rich's comment to explain why this bug happens and how the patch I attached works around this.</p>
<p>I don't know if this is what he was asking for though.</p><p>I think Rich meant that a ticket <b>should</b> have a plan of that form but does not. My own take on "triaged" is that it should state actual and expected results demonstrating a problem - I don't think it needs to actually describe the solution (as that can happen later in development). It is entirely possible that Rich and I differ in our interpretation of that. <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/> I will see if I can rework the description a bit to match what I've been doing elsewhere. </p><p>Alex, I have looked through the existing wiki pages on the ticket tracking process, and do not recall seeing anything about this desired aspect of a triaged ticket. Is it already documented somewhere and I missed it? Not that it has to be documented, necessarily, but Rich saying "triage guidelines" makes it sound like a filter he applies that ticket creators and screeners maybe should know about.</p><p>To me, Triage (and Vetting) is all about having good problem statements. For a defect, it is most important to demonstrate the problem (what happens now) and what you expect to happen instead. I do not usually expect there to necessarily be "by ____" in the ticket - to me that is part of working through the solution (although it is typical to have this in an enhancement). This ticket, as it stands now, seems to have both a good problem statement and a good cause/solution statement so seems to exceed Triaging standards afaik.</p>
<p>Two places where I have tried to write about these things in the past are <a href="http://dev.clojure.org/display/community/Creating+Tickets">http://dev.clojure.org/display/community/Creating+Tickets</a> and in the Triage process on the workflow page <a href="http://dev.clojure.org/display/community/JIRA+workflow">http://dev.clojure.org/display/community/JIRA+workflow</a>. </p>ApprovalOkGlobal RankPatchCode[CLJ-1233] Allow ** as a valid symbol name without triggering "not declared dynamic" warningshttp://dev.clojure.org/jira/browse/CLJ-1233
Clojure<p>Currently declaring a symbol ** triggers a compiler warning:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (defn ** [] nil)
Warning: ** not declared dynamic and thus is not dynamically rebindable, but its name suggests otherwise. Please either indicate ^:dynamic ** or change the name. (NO_SOURCE_PATH:1)
#'user/**</pre>
</div></div>
<p>&#42;&#42; is a useful symbol in many domains, e.g. as an exponent function or as a matrix multiplication operator.</p>
<p><b>Cause:</b> This warning checks for a def of a symbol that starts and ends with *.</p>
<p><b>Solution:</b> Change check for name length &gt;2 to skip this particular case.</p>
<p><b>Patch:</b> clj-1233-with-test-v2.diff</p>
<p><b>Screened by:</b> Alex Miller</p>AllCLJ-1233Allow ** as a valid symbol name without triggering "not declared dynamic" warningsDefectTrivialClosedCompletedUnassignedMike AndersonCompilerTue, 23 Jul 2013 05:30:12 -0500Fri, 25 Oct 2013 11:01:35 -0500Fri, 25 Oct 2013 11:01:35 -0500Release 1.5Release 1.601<p>Link to discussion on Clojure-Dev<br/>
<a href="https://groups.google.com/forum/#!topic/clojure-dev/OuTMsZQkxN4">https://groups.google.com/forum/#!topic/clojure-dev/OuTMsZQkxN4</a></p><p>Minimal patch for the ** case only</p><p>Minimal patch would be slightly less minimal with a test. <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p><p>Hmmm... is there a standard/reliable method for testing the presence / non-presence of emitted warnings?</p><p>There are some test helpers in Clojure's test/clojure/test_helper.clj for capturing error messages.</p><p>New patch with a test that includes using "with-err-print-writer" to detect the avoidance of the warning.</p><p>Patch clj-1233-with-test-v2.txt is identical to Mike Anderson's clj-1233-with-test.diff (preserving his authorship), except it avoids git warnings when applying by eliminating trailing whitespace in added lines.</p>ApprovalOkGlobal RankPatchCode and Test[CLJ-1232] Functions with non-qualified return type hints force import of hinted classes when called from other namespacehttp://dev.clojure.org/jira/browse/CLJ-1232
Clojure<p>You can add a type hint to function arglists to indicate the return type of a function like so.</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>user&gt; (import '(java.util List))
java.util.List
user&gt; (defn linkedlist ^List [] (java.util.LinkedList.))
#'user/linkedlist
user&gt; (.size (linkedlist))
0
</pre>
</div></div>
<p>The problem is that now when I call `linkedlist` exactly as above from another namespace, I'll get an exception because java.util.List is not imported in there.</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>user&gt; (in-ns 'user2)
#&lt;Namespace user2&gt;
user2&gt; (refer 'user)
nil
user2&gt; (.size (linkedlist))
CompilerException java.lang.IllegalArgumentException: Unable to resolve classname: List, compiling:(NO_SOURCE_PATH:1:1)
user2&gt; (import '(java.util List)) ;; Too bad, need to import List here, too.
java.util.List
user2&gt; (.size (linkedlist))
0
</pre>
</div></div>
<p>There are two workarounds: You can import the hinted type also in the calling namespace, or you always use fully qualified class names for return type hints. Clearly, the latter is preferable.</p>
<p>But clearly, that's a bug that should be fixed. It's not in analogy to type hints on function parameters which may be simple-named without having any consequences for callers from other namespaces.</p>CLJ-1232Functions with non-qualified return type hints force import of hinted classes when called from other namespaceDefectMajorOpenUnresolvedUnassignedTassilo HorncompilertypehintsThu, 18 Jul 2013 06:06:29 -0500Thu, 2 Oct 2014 03:16:30 -0500Release 1.5Release 1.6Release 1.785<p>To make sure I understand, Nicola, in this ticket you are asking that the Clojure compiler change behavior so that the sample code works correctly with no exceptions, the same way as it would work correctly without exceptions if one of the workarounds were used?</p><p>Hi Andy. Tassilo here, not Nicola. But yes, the example should work as-is. When I'm allowed to use type hints with simple imported class names for arguments, then doing so for return values should work, too. </p><p>Type hints on function params are only consumed by the function definition, i.e. in the same module as the import/alias. Type hints on returns are just metadata, they don't get 'compiled' and if the metadata is not useful to consumers in other namespaces, it's not a useful hint. So, if it's not a type in the auto-imported set (java.lang), it should be fully qualified.</p><p>Based on Rich's comment, this ticket should probably morph into an enhancement request on documentation, probably on <a href="http://clojure.org/java_interop#Java">http://clojure.org/java_interop#Java</a> Interop-Type Hints. </p><p>I would suggest something like the following for a documentation change, after this part of the text on the page Alex links in the previous comment:</p>
<p>For function return values, the type hint can be placed before the arguments vector:</p>
<p>(defn hinted<br/>
(^String [])<br/>
(^Integer <span class="error">&#91;a&#93;</span>)<br/>
(^java.util.List <span class="error">&#91;a &amp; args&#93;</span>))</p>
<p>-&gt; #user/hinted</p>
<p>If the return value type hint is for a class that is outside of java.lang, which is the part auto-imported by Clojure, then it must be a fully qualified class name, e.g. java.util.List, not List.</p><p>I don't understand why we should enforce this complexity to the user.<br/>
Why can't we just make the Compiler (or even defn itself) update all the arglists tags with properly resolved ones? (that's what I'm doing in tools.analyzer.jvm)</p><p>I'm with Nicola here. I also think that defn should resolve the type hint according the imports of the namespace defn is used in.</p><p>Same here, I was bit by this in the past. The current behavior is clearly counterintuitive. </p><p>Attached two patches implementing two different solutions:</p>
<ul>
<li>0001-auto-qualify-arglists-class-names.patch makes the compiler automatically qualify all the tags in the :arglists</li>
<li>0001-throw-on-non-qualified-class-names-that-are-not-auto.patch makes the compiler throw an exception for all public defs whose return tag is a symbol representing a non-qualified class that is not in the auto-import list (approach proposed in IRC by Alex Miller)</li>
</ul>
<p>For what it's worth, I'd prefer the first patch because the second doesn't help in situations where the caller lives in a namespace where the called function's return type hinted class is `ns-unmap`-ed. And there a good reasons for doing that. For example, Process is a java.lang class and Process is a pretty generic name. So in some namespace, I want to define my own Process deftype or defrecord. Without unmapping 'Process first to get rid of the java.lang.Process auto-import, I'd get an exception:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>user&gt; (deftype Process [])
IllegalStateException Process already refers to: class java.lang.Process in namespace: user clojure.lang.Namespace.referenceClass (Namespace.java:140)
</pre>
</div></div>
<p>Now when I call some function from some library that has a `^Process` return type hint (meaning java.lang.Process there), I get the same exception as in my original report.</p>
<p>I can even get into troubles when only using standard Clojure functions because those have `^String` and `^Class` type hints. IMO, Class is also a pretty generic name I should be able to name my custom deftype/defrecord. And I might also want to have a custom String type/record in my astrophysics system.</p><p>Not sure whether the root cause of this behavior is the same as the example in the description or not, but seems a little weird that even for fully qualified Java class names hinting the arg vector, it makes a difference whether it is done with defn or def:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">Clojure 1.6.0
user=&gt; (set! *warn-on-reflection* <span class="code-keyword">true</span>)
<span class="code-keyword">true</span>
user=&gt; (defn f1 ^java.util.LinkedList [coll] (java.util.LinkedList. coll))
#'user/f1
user=&gt; (def f2 (fn ^java.util.LinkedList [coll] (java.util.LinkedList. coll)))
#'user/f2
user=&gt; (.size (f1 [2 3 4]))
3
user=&gt; (.size (f2 [2 3 4]))
Reflection warning, NO_SOURCE_PATH:5:1 - reference to field size can't be resolved.
3</pre>
</div></div><p>Andy, can you file that as a separate ticket?</p><p>Created ticket <a href="http://dev.clojure.org/jira/browse/CLJ-1543" title="Type tags on argument vector appear to help avoid reflection when used with defn, but not with def foo (fn ...)">CLJ-1543</a> for the issue raised in my comment earlier on 30 Sep 2014.</p><p>Tassilo (or anyone), is there a reason to prefer putting the tag on the argument vector in your example? It seems that putting it on the Var name instead avoids this issue:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (clojure-version)
<span class="code-quote">"1.6.0"</span>
user=&gt; (set! *warn-on-reflection* <span class="code-keyword">true</span>)
<span class="code-keyword">true</span>
user=&gt; (<span class="code-keyword">import</span> '(java.util List))
java.util.List
user=&gt; (defn ^List linkedlist [] (java.util.LinkedList.))
#'user/linkedlist
user=&gt; (.size (linkedlist))
0
user=&gt; (ns user2)
nil
user2=&gt; (refer 'user)
nil
user2=&gt; (.size (linkedlist))
0</pre>
</div></div>
<p>I suppose that only allows a single type tag, rather than an independent one for each arity.</p><p>I wasn't aware of the fact that you can put it on the var's name. That's not documented at <a href="http://clojure.org/java_interop#Java">http://clojure.org/java_interop#Java</a> Interop-Type Hints. But IMHO the documented version with putting the tag on the argument vector is more general since it supports different return type hints for the different arity version. In any case, if both forms are permitted then they should be equivalent in the case the function has only one arity.</p>ApprovalTriagedGlobal RankPatchCode[CLJ-1226] set! of a deftype field using field-access syntax causes ClassCastExceptionhttp://dev.clojure.org/jira/browse/CLJ-1226
Clojure<p>Clojure 1.6.0-master-SNAPSHOT</p>
<p>user=&gt; (defprotocol p (f <span class="error">&#91;_&#93;</span>))<br/>
p<br/>
user=&gt; (deftype t <span class="error">&#91;^:unsynchronized-mutable x&#93;</span> p (f <span class="error">&#91;_&#93;</span> (set! (.x _) 1)))<br/>
user.t<br/>
user=&gt; (f (t. 1))<br/>
ClassCastException user.t cannot be cast to compile__stub.user.t user.t (NO_SOURCE_FILE:1</p>
<p>After patch:<br/>
Clojure 1.6.0-master-SNAPSHOT<br/>
user=&gt; (defprotocol p (f <span class="error">&#91;_&#93;</span>))<br/>
p<br/>
user=&gt; (deftype t <span class="error">&#91;^:unsynchronized-mutable x&#93;</span> p (f <span class="error">&#91;_&#93;</span> (set! (.x _) 1)))<br/>
user.t<br/>
user=&gt; (f (t. 1))<br/>
1</p>CLJ-1226set! of a deftype field using field-access syntax causes ClassCastExceptionDefectMajorOpenUnresolvedUnassignedNicola MomettocompilerdeftypeinteropWed, 26 Jun 2013 19:21:01 -0500Sun, 31 Aug 2014 10:36:53 -0500Release 1.700<p>This patch offers a better workaround for <a href="http://dev.clojure.org/jira/browse/CLJ-1075" title="deftype: compiler error on set! for mutable field"><del>CLJ-1075</del></a>, making it possible to write<br/>
(deftype foo <span class="error">&#91;^:unsynchronized-mutable x&#93;</span> MutableX (set-x <span class="error">&#91;this v&#93;</span> (try (set! (.x this) v)) v))</p>Global RankPatchCode and Test[CLJ-1216] Evaling ((fn [do] do) 1) returns nil while ((fn [do] do do) 1) returns 1http://dev.clojure.org/jira/browse/CLJ-1216
Clojure<p>user=&gt; ((fn <span class="error">&#91;do&#93;</span> do) 1)<br/>
nil</p>
<p>user=&gt; ((fn <span class="error">&#91;do&#93;</span> (do do)) 1)<br/>
1</p>
<p>user=&gt; ((fn [] do))<br/>
nil</p>
<p>user=&gt; ((fn [] do do))<br/>
CompilerException java.lang.RuntimeException: Unable to resolve symbol: do in this context, compiling:(NO_SOURCE_PATH:0:0) </p>CLJ-1216Evaling ((fn [do] do) 1) returns nil while ((fn [do] do do) 1) returns 1DefectMinorOpenUnresolvedUnassignedNicola MomettocompilerSun, 9 Jun 2013 16:29:14 -0500Tue, 3 Sep 2013 09:07:23 -050000<p>This patch creates a DoExpr class and makes DoExpr.Parser the DO special form parser.</p>
<p>DoExpr.Parser simply removes the 'do' symbol and delegates to BodyExpr, that was previously done by BodyExpr incorrectly.</p>Global RankPatchCode and Test[CLJ-1214] Compiler runs out of memory on a small snippet of codehttp://dev.clojure.org/jira/browse/CLJ-1214
Clojure<p>Clojure compiler runs out of memory when loading the attached file. Transcript below.</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>$ java -cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar:. clojure.main
Clojure 1.5.1
user=&gt; (load "fubar")
OutOfMemoryError GC overhead limit exceeded [trace missing]
user=&gt;
</pre>
</div></div>
<p>The file contents are:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre> (ns fu.bar)
(defn foo[l] (concat (drop-last l) (repeat (last l))))
(def ^:const bar (foo [#(print "") #(println ";")]))
bar
</pre>
</div></div>
<p>If I remove the metadata on bar, it works. Removal of namespace seems to fix it as well. Pretty strange.</p>
<p>Although I realize this code is not quite kosher, it would be nice to have the compiler deal with it.</p>Linux 3.2.0-39-generic CLJ-1214Compiler runs out of memory on a small snippet of codeDefectMajorClosedDeclinedUnassignedPraki PrakashcompilerFri, 31 May 2013 21:33:04 -0500Fri, 18 Apr 2014 07:08:47 -0500Fri, 18 Apr 2014 07:08:47 -0500Release 1.500<p>If you really do have 'bar' on a line by itself at the end of file fubar.clj, it seems you are asking it to evaluate the value of bar, which by the definitions is an infinite list, and will of course exhaust all available memory if you attempt to evaluate it.</p>
<p>It seems to me the more odd thing is not that it runs out of memory as shown, but that it does <b>not</b> run out of memory when you remove the metadata on bar.</p>
<p>What is the purpose of having 'bar' on a line by itself at the end of the file?</p>
<p>If I try this but remove 'bar' as the last line of the file, loading the file causes no errors regardless of whether there is metadata on bar's definition or not. It is strange that doing (load "fubar") followed by (first fu.bar/bar) seems to go into an infinite loop if the :const is there on bar, but quickly returns the correct answer if the metadata is removed.</p><p>This code snippet is a minimal test case to show that the compiler runs out of memory. What I meant by "it works" was that the compiler doesn't run out of memory and successfully loads the file (or in my real code base, the namespace is compiled).</p>
<p>In my code, I use bar (or whatever the real thing is) as source of sequence of functions. The sole reference to bar is needed to trigger this issue. I believe that bar is not being fully evaluated here and thus no infinite loop. If I try to print it, yes, it will ultimately fail.</p>
<p>Having thought about this a bit more, it seems to me that when bar is annotated with const, the compiler is trying to evaluate the associated expression which exhausts the heap? I have never looked at the compiler source and thus not sure if this is what is happening. If it is, then it seems like one should be really careful when adding metadata.</p>
<p>That still leaves the other question about the namespace requirement to cause memory exhaustion. I quite distinctly recall having to add the namespace when trying to come up with minimal test case to reproduce the bug.</p>
<p>If you think this is really user error, I would accept it <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p><p>It is not just any old metadata that causes this issue, only :const metadata. I tried testing with :const replaced with :dynamic and :private, and there was no problem.</p>
<p>This might shed some light on the issue: <a href="https://github.com/clojure/clojure/blob/master/changes.md#215-const-defs">https://github.com/clojure/clojure/blob/master/changes.md#215-const-defs</a></p>
<p>It appears that ^:const is causing the compiler to evaluate the value at compile time. The value in your example is unbounded, so that can never complete.</p><p>the problem is for complex objects :const results in the pr-str of the object being embedded in the bytecode, a pr-str of an infinite seq is going to blow the heap. I think the limits on the usage of :const are not well defined and instead of falling back on pr-str it should throw a compilation error for constants that are not jvm primitves or strings</p><p>There is support for handling a variety of other useful cases (core Clojure collections for example) as constant objects. And it is useful to allow code that is not constant but evaluates to a constant result (for creating data tables, etc). In the face of that, I'm not sure it's possible to bound this usefully without solving the halting problem. <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/> I'm tempted to just put this issue in the category of "don't do that". </p><p>After sleeping on it, I don't think this is worth working on (if there is anything that could even be done).</p>Global Rank[CLJ-1208] Namespace is not loaded on defrecord class inithttp://dev.clojure.org/jira/browse/CLJ-1208
Clojure<p>As a user of Clojure interop from Java, I want defrecords (and deftypes?) to load their namespaces upon class initialization so that I can simply construct and use AOT'd record classes without manually requiring their namespaces first.</p>
<p>Calling the defrecord's constructor may or may not result in "Attempting to call unbound fn" exceptions, depending on what code has already been run.</p>
<p>This issue has been raised several times over the years, but I could not find an existing ticket for it:</p>
<ul>
<li><a href="https://groups.google.com/d/msg/clojure-dev/4CtSVWcD15A/shpMuyjMpxsJ">https://groups.google.com/d/msg/clojure-dev/4CtSVWcD15A/shpMuyjMpxsJ</a></li>
<li><a href="https://groups.google.com/d/msg/clojure/3DekTQZfDTk/wlssZL7EQWEJ">https://groups.google.com/d/msg/clojure/3DekTQZfDTk/wlssZL7EQWEJ</a></li>
<li><a href="https://groups.google.com/d/msg/clojure/K4gfjrLe8GA/OnB5g4SU0l8J">https://groups.google.com/d/msg/clojure/K4gfjrLe8GA/OnB5g4SU0l8J</a></li>
</ul>
CLJ-1208Namespace is not loaded on defrecord class initEnhancementMajorOpenUnresolvedUnassignedTim McCormackcompilerdefrecorddeftypeFri, 3 May 2013 17:21:54 -0500Thu, 22 Jan 2015 12:59:22 -0600Release 1.855<p>The attached patch approaches this issue by adding a :load-ns options to deftype/defrecord which defaults to false.<br/>
When true, the type/record be compiled with a call to clojure.core/require to its originating namespace in its static initializer.</p>
<p>The patch has two known limitations:</p>
<ul class="alternate" type="square">
<li>clojure.core deftypes/defrecords cannot have :load-ns since we use clojure.core/require to load the namespaces so clojure.core <b>needs</b> to be loaded manually anyway</li>
<li>clojure.lang.Compiler/demunge is used to get the originating namespace from the deftype/defrecord class name, this means that namespaces like foo_bar are not supported since they get demunged into foo-bar. If this is something that needs to be addressed, it shouldn't be too hard to just pass the unmunged namespace name in the opts map.</li>
</ul>
<p>Updated patch fixing a whitespace error and mentionint :load-ns in the docstrings of deftype/defrecord</p>ApprovalVettedGlobal RankPatchCode and Test[CLJ-1184] Evaling #{do ...} or [do ...] is treated as the do special formhttp://dev.clojure.org/jira/browse/CLJ-1184
Clojure<p><b>Problem:</b> Evaluating a persistent collection for which the function <tt>first</tt> returns the symbol <tt>do</tt> leads to that collection being treated as the do special form, even though it may be a vector or even a set. IMHO, the expected result would be to report that <tt>do</tt> cannot be resolved. </p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">[<span class="code-keyword">do</span> 1 2]
;=&gt; 2
#{<span class="code-quote">"hello"</span> <span class="code-quote">"goodbye"</span> <span class="code-keyword">do</span>}
;=&gt; <span class="code-quote">"hello"</span>
; Wat?</pre>
</div></div>
<p><b>Cause:</b> The check for <tt>do</tt> is checking for IPersistentCollection instead of ISeq.</p>
<p><b>Solution:</b> Change the cast (occurs in two places) for the <tt>do</tt> form check from IPersistentCollection to ISeq:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java"><span class="code-keyword">if</span>(form <span class="code-keyword">instanceof</span> IPersistentCollection &amp;&amp; Util.equals(RT.first(form), DO))</pre>
</div></div>
<p>to</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java"><span class="code-keyword">if</span>(form <span class="code-keyword">instanceof</span> ISeq &amp;&amp; Util.equals(RT.first(form), DO))</pre>
</div></div>
<p><b>Current patch:</b> <a href="http://dev.clojure.org/jira/browse/CLJ-1184" title="Evaling #{do ...} or [do ...] is treated as the do special form"><del>CLJ-1184</del></a>-p4.patch</p>
<p><b>Screened by:</b> Alex Miller</p>
CLJ-1184Evaling #{do ...} or [do ...] is treated as the do special formDefectTrivialClosedCompletedUnassignedJiří MaršíkCompilerSat, 16 Mar 2013 08:56:32 -0500Fri, 25 Oct 2013 11:01:36 -0500Fri, 25 Oct 2013 11:01:36 -0500Release 1.5Release 1.602<p>Attached a patch that changes IPersistentCollection to ISeq on the referenced line, and a regression test.</p><p>As found out on #clojure, there are still some cases where the symbol "do" behaves in unexpected ways that this patch doesn't address.</p>
<p>user=&gt; ((fn <span class="error">&#91;do&#93;</span> do) 1)<br/>
nil<br/>
user=&gt; (let <span class="error">&#91;do 1&#93;</span> do)<br/>
nil</p><p>Presumably the same issue is the difference between</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(let [x 5] <span class="code-keyword">do</span> x)</pre>
</div></div>
<p>which returns <tt>5</tt> and</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(let [x 5] <span class="code-keyword">do</span> <span class="code-keyword">do</span> x)</pre>
</div></div>
<p> which gives a compile error.</p><p>This is actually a different bug.<br/>
I created another ticket with patch+test see <a href="http://dev.clojure.org/jira/browse/CLJ-1216">http://dev.clojure.org/jira/browse/CLJ-1216</a></p><p>There is a similar case that shows up in Compiler.compile1() around line 7139. Should this also change? </p>
<p>Whatever case that is, would also be nice to have a test for it too.</p><p>Good catch; this one's a bit trickier to test, since you either have to have a resource on the classpath to compile using clojure.core/compile, or else call the lower-level Compiler/compile directly. To keep the filesystem clean I'm opting for the second one. Path coming shortly.</p><p>Attached a replacement patch with the second fix. For testing I couldn't call Compiler.compile directly without getting an NPE (that I couldn't reproduce at the repl), so instead I opted to write to a file, use clojure.core/compile, and cleanup the files inside the test.</p><p>Presumptuously changing back to its former Vetted state, since the latest patch seems to address the reasons it was marked Incomplete on July 2, 2013.</p><p>Updated patch very slightly to fix a spelling typo.</p><p>Updated description to help it along a bit and marked Screened.</p><p>All 3 of the attached patches no longer apply cleanly to latest master as of Aug 14 2013. This may simply be due to extra tests added to file compilation.clj by the patch for <a href="http://dev.clojure.org/jira/browse/CLJ-1154" title="Compile.java closes out preventing java from reporting exceptions"><del>CLJ-1154</del></a>, which was committed earlier today. If so, it should be pretty straightforward to update the stale patch(es). See the section "Updating Stale Patches" at <a href="http://dev.clojure.org/display/community/Developing+Patches">http://dev.clojure.org/display/community/Developing+Patches</a></p><p>Attached a new patch (p4) that should apply. Also halfway reverted a change that Alex made regarding which files are cleaned up after the test. When I run the tests on my machine with his version, several class files are leftover.</p><p>Screened.</p>ApprovalOkGlobal RankPatchCode and Test[CLJ-1093] Empty PersistentCollections get incorrectly evaluated as their generic clojure counterparthttp://dev.clojure.org/jira/browse/CLJ-1093
Clojure<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user&gt; (defrecord x [])
user.x
user&gt; #user.x[] ;; expect: #user.x{}
{}
user&gt; #user.x{} ;; expect: #user.x{}
{}
user&gt; #clojure.lang.PersistentTreeMap[]
{}
user&gt; (class *1) ;; expect: clojure.lang.PersistentTreeMap
clojure.lang.PersistentArrayMap</pre>
</div></div>
<p><b>Cause:</b> Compiler's ConstantExpr parser returns an EmptyExpr for all empty persistent collections, even if they are of types other than the core collections (for example: records, sorted collections, custom collections). EmptyExpr reports its java class as one the classes - IPersistentList/IPersistentVector/IPersistentMap/IPersistentSet rather than the original type.</p>
<p><b>Proposed:</b> If one of the Persistent* classes, then create EmptyExpr as before, otherwise retain the ConstantExpression of the original collection. <br/>
Since EmptyExpr is a compiler optimization that applies only to some concrete clojure collections, making EmptyExpr dispatch on concrete types rather than on generic interfaces makes the compiler behave as expected</p>
<p><b>Patch:</b> 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a>-v2.patch</p>
<p><b>Screened by:</b></p>CLJ-1093Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpartDefectMajorReopenedUnresolvedUnassignedNicola MomettocollectionscompilerWed, 24 Oct 2012 09:13:56 -0500Mon, 6 Oct 2014 12:29:29 -0500Release 1.4Release 1.5Release 1.854<p>Unable to reproduce this bug on latest version of master. Most likely fixed by some of the recent changes to data literal readers. </p>
<p>Marking Not-Approved. </p><p>Could not reproduce in master. </p><p>I just checked, and the problem still exists for records with no arguments:</p>
<p>Clojure 1.6.0-master-SNAPSHOT<br/>
user=&gt; (defrecord a [])<br/>
user.a<br/>
user=&gt; #user.a[]<br/>
{}</p>
<p>Admittedly it's an edge case and I see little usage for no-arguments records, but I think it should be addressed aswell since the current behaviour is not what one would expect</p><p>Got the following REPL interaction:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">% java -jar ~/.m2/repository/org/clojure/clojure/1.5.0/clojure-1.5.0.jar
user=&gt; (defrecord a [])
user.a
user=&gt; (a.)
#user.a{}
user=&gt; #user.a{}
{}
#user.a[]
{}</pre>
</div></div>
<p>This should be reopened or declined for another reason than reproducability.</p><p>I'm reopening this since the bug is still there.</p>
<p>Patch clj-1093-fix-empty-record-literal-patch-v2.txt dated Mar 13, 2013 is identical to Bronsa's patch 001-fix-empty-record-literal.patch dated Oct 24, 2012, except that it applies cleanly to latest master. I'm not sure why the older patch doesn't but git doesn't like something about it.</p><p>Patch 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a>-fix-empty-records-literal-v2.patch solves more issues than the previous patch that was not evident to me at the time.</p>
<p>Only collections that are either PersistentList or PersistentVector or PersistentHash<span class="error">&#91;Map|Set&#93;</span> or PersistentArrayMap can now be EmptyExpr.<br/>
This is because we don't want every IPersistentCollection to be emitted as a generic one eg.</p>
<p>user=&gt; (class #clojure.lang.PersistentTreeMap[])<br/>
clojure.lang.PersistentArrayMap</p>
<p>Incidentally, this patch also solves <a href="http://dev.clojure.org/jira/browse/CLJ-1187" title="Clojure loses quoted metadata on empty literals"><del>CLJ-1187</del></a><br/>
This patch should be preferred over the one on <a href="http://dev.clojure.org/jira/browse/CLJ-1187" title="Clojure loses quoted metadata on empty literals"><del>CLJ-1187</del></a> since it's more general</p><p>Maybe this is related:</p>
<p>user=&gt; (def x `(quote ~(list 1 (clojure.lang.PersistentTreeMap/create (seq <span class="error">&#91;1 2 3 4&#93;</span>)))))<br/>
#'user/x<br/>
user=&gt; x<br/>
(quote (1 {1 2, 3 4}))<br/>
user=&gt; (class (second (second x)))<br/>
clojure.lang.PersistentTreeMap<br/>
user=&gt; (eval x)<br/>
(1 {1 2, 3 4})<br/>
user=&gt; (class (second (eval x)))<br/>
clojure.lang.PersistentArrayMap</p>
<p>Even if the collection is not evaluated, it is still converted to the generic clojure counterpart.</p><p>In the change for ObjectExpr.emitValue() where you've added PersistentArrayMap to the PersistentHashMap case, should the IPersistentVector case below that be PersistentVector instead, otherwise it would snare a custom IPersistentVector that's not a PersistentVector, right?</p>
<p>This line: "else if(form instanceof ISeq)" at the end of the Compiler diff has different leading tabs which makes the diff slightly more confusing than it could be.</p>
<p>Would be nice to add a test for the sorted map case in the description.</p>
<p>Marking incomplete to address some of these.</p><p>bump</p><p>Attached patch 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a>-fix-empty-collection-literal-evaluation.patch which implements your suggestions.</p>
<p>replacing IPersistentVector with PersistentVector in ObjectExpr.emitValue() exposes a print-dup limitation: it expects every IPersistentCollection to have a static "create" method.</p>
<p>This required special casing for MapEntry and APersistentVector$SubVector</p><p>I updated the patch adding print-dups for APersistentVector$SubVec and other IPersistentVectors rather than special casing them in the compiler</p><p>All of the checks on concrete classes in the Compiler parts of this patch don't sit well with me. I understand how you got to this point and I don't have an alternate recommendation (yet) but all of that just feels like the wrong direction.</p>
<p>We want to be built on abstractions such that internal collections are not special; they should conform to the same expectations as an external collection and both should follow the same paths in the compiler - needing to check for those types is a flag for me that something is amiss.</p>
<p>I am marking Incomplete for now based on these thoughts. </p><p>I've been thinking for a while about this issue and I've come to the conclusion that in my later patches I've been trying to incorporate fixes for 3 different albeit related issues:</p>
<p>1- Clojure transforms all empty IPersistentCollections in their generic Clojure counterpart</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user&gt; (defrecord x [])
user.x
user&gt; #user.x[] ;; expected: #user.x{}
{}
user&gt; #user.x{} ;; expected: #user.x{}
{}
user&gt; #clojure.lang.PersistentTreeMap[]
{}
user&gt; (class *1) ;; expected: clojure.lang.PersistentTreeMap
clojure.lang.PersistentArrayMap</pre>
</div></div>
<p>2- Clojure transforms all the literals of collections implementing the Clojure interfaces (IPersistentList, IPersistentVector ..) that are NOT defined with deftype or defrecord, to their<br/>
generic Clojure counterpart</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (class (eval (sorted-map 1 1)))
clojure.lang.PersistentArrayMap ;; expected: clojure.lang.PersistentTreeMap</pre>
</div></div>
<p>3- print-dup is broken for some Clojure persistent collections</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (print-dup (subvec [1] 0) *out*)
#=(clojure.lang.APersistentVector$SubVector/create [1])
user=&gt; #=(clojure.lang.APersistentVector$SubVector/create [1])
IllegalArgumentException No matching method found: create clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:53)</pre>
</div></div>
<p>I'll keep this ticket regarding issue #1 and open two other tickets for issue #2 and #3</p><p>I've attached a new patch fixing only this issue, the approach is explained in the description</p><p>0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart">CLJ-1093</a>-v2.patch is an updated patch that correctly handles metadata evaluation semantics on empty literals and adds some tests for it</p>ApprovalVettedGlobal RankPatchCode and Test[CLJ-1028] (compile) crashes with NullPointerException if public function 'load' is definedhttp://dev.clojure.org/jira/browse/CLJ-1028
Clojure<p>When performing AOT compilation, if the namespace being compiled or one of the namespaces :required by it defines a public function named 'load', the compiler will crash with a NullPointerException.</p>
<p>The following code demonstrates this:</p>
<p>(ns ca.ancilla.kessler.core (:gen-class)) (defn load <span class="error">&#91;x&#93;</span> x) (defn -main <span class="error">&#91;&amp; args&#93;</span> 0)</p>
<p>When run directly, with 'clojure -m ca.ancilla.kessler.core' or 'clojure ca/ancilla/kessler/core.clj', it runs as expected. When loaded with 'clojure -i' and (compile)d, however, or when automatically compiled by 'lein run', it results in a NullPointerException (the complete stack trace is attached).</p>
<p>This occurs whether or not 'load' or actually called. It does not, however, occur if 'load' is private.</p>Linux, OpenJDK 1.6.0 64bitCLJ-1028(compile) crashes with NullPointerException if public function 'load' is definedDefectMinorClosedDeclinedUnassignedBen KellyCompilerbugFri, 20 Jul 2012 12:40:30 -0500Fri, 20 Jul 2012 22:06:22 -0500Fri, 20 Jul 2012 16:35:47 -0500Release 1.401<p>If you add (:refer-clojure :exclude <span class="error">&#91;load&#93;</span>) to the (ns), it works fine:</p>
<p>(ns ca.ancilla.kessler.core (:refer-clojure :exclude <span class="error">&#91;load&#93;</span>) (:gen-class))<br/>
(defn load <span class="error">&#91;x&#93;</span> x)<br/>
(defn -main <span class="error">&#91;&amp; args&#93;</span> 0)</p>
<p>Thanks to llasram on IRC for discovering this.</p><p>You should not replace functions in clojure.core. This is left legal (with a loud CONSOLE warning) for compatibility, but programs that do it are in error.</p><p>So, just to make sure that I have this right, then...</p>
<p>If I want to create a namespace with a public function that shares a name with a function in clojure.core, the only supported way of doing this is to (:refer-clojure :exclude <span class="error">&#91;list of all such functions&#93;</span>)?</p>
<p>If so, it would be nice if the warning were replaced with an error, rather than having the compiler emit an error and then <b>crash</b>.</p>Global Rank[CLJ-979] Clojure resolves to wrong deftype classes when AOT compiling or reloadinghttp://dev.clojure.org/jira/browse/CLJ-979
Clojure<p>Compiling a class via `deftype` during AOT compilation gives different results for the different constructors. These hashes should be identical.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (binding [*compile-files* <span class="code-keyword">true</span>] (eval '(deftype Abc [])))
user.Abc
user=&gt; (hash Abc)
16446700
user=&gt; (hash (class (-&gt;Abc)))
31966239 ;; should be 16446700</pre>
</div></div>
<p>This also means that whenever there's a stale AOT compiled deftype class in the classpath, that class will be used rather then the JIT compiled one, breaking repl interaction.</p>
<p>Another demonstration of this classloader issue (from <a href="http://dev.clojure.org/jira/browse/CLJ-1495" title="Defining a record with defrecord twice breaks record equality"><del>CLJ-1495</del></a>) when reloading deftypes (no AOT) :</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user&gt; (defrecord Foo [bar])
user.Foo
user&gt; (= (-&gt;Foo 42) #user.Foo{:bar 42}) ;;expect <span class="code-keyword">this</span> to evaluate to <span class="code-keyword">true</span>
<span class="code-keyword">true</span>
user&gt; (defrecord Foo [bar])
user.Foo
user&gt; (= (-&gt;Foo 42) #user.Foo{:bar 42}) ;;expect <span class="code-keyword">this</span> to evaluate to <span class="code-keyword">true</span> also -- but it doesn't!
<span class="code-keyword">false</span>
user&gt;</pre>
</div></div>
<p>This bug also affects AOT compilation of multimethods that dispatch on a class, this affected core.match for years see <a href="http://dev.clojure.org/jira/browse/MATCH-86">http://dev.clojure.org/jira/browse/MATCH-86</a>, <a href="http://dev.clojure.org/jira/browse/MATCH-98">http://dev.clojure.org/jira/browse/MATCH-98</a>. David had to work-around this issue by using a bunch of protocols instead of multimethods.</p>
<p><b>Cause of the bug:</b> currently clojure uses Class.forName to resolve a class from a class name, which ignores the class cache from DynamicClassLoader thus reloading deftypes or mixing AOT compilation at the repl with deftypes breaks, resolving to the wrong class.</p>
<p><b>Approach:</b> the current patch (<a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>-v7.patch) addresses this issue in multiple ways:</p>
<ul class="alternate" type="square">
<li>it makes RT.classForName/classForNameNonLoading look in the class cache before delegating to Class/forName if the current classloader is not a DynamicClassLoader (this incidentally addresses also <a href="http://dev.clojure.org/jira/browse/CLJ-1457" title="once the compiler pops the dynamic classloader from the stack, attempts to read record reader literals will fail"><del>CLJ-1457</del></a>)</li>
<li>it makes clojure use RT.classForName/classForNameNonLoading instead of Class/forName</li>
<li>it overrides Classloader/loadClass so that it's class cache aware &#8211; this method is used by the jvm to load classes</li>
<li>it changes gen-interface to always emit an in-memory interface along with the <span class="error">&#91;optional&#93;</span> on-disk interface so that the in-memory class is always updated.</li>
</ul>
<p><b>Patch:</b> <a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>-v7.patch</p>
<p><b>Screened by:</b> Alex Miller</p>CLJ-979Clojure resolves to wrong deftype classes when AOT compiling or reloadingDefectCriticalClosedCompletedUnassignedEdmund JacksonaotclassloadercompilerThu, 3 May 2012 03:01:37 -0500Wed, 18 Feb 2015 04:21:23 -0600Sat, 10 Jan 2015 09:24:25 -0600Release 1.3Release 1.4Release 1.5Release 1.6Release 1.7Release 1.71410<p>I can't reproduce this under Clojure 1.3 or 1.4, and Leiningen 1.7.1 on either Java 1.7.0-jdk7u4-b21 OpenJDK 64-Bit or Java 1.6.0_31 Java HotSpot 64-Bit. OS is Mac OS X 10.7.</p>
<p>Edmund, how are you running this AOT code? I wrapped your code in a main function and built an uberjar from it.</p><p>Hi Scott,</p>
<p>Interesting. </p>
<p>I have two use cases<br/>
1. AOT compile and call from repl.<br/>
My steps: git clone, lein compile, lein repl, (use 'aots.death), (in-ns 'aots.death), (= (class (Dontwork. nil)) (class (map-&gt;Dontwork {:a 1}))) =&gt; false</p>
<p>2. My original use case, which I've minimised here, is an AOT ns, producing a genclass that is called instantiated from other Java (no main). This produces the same error. I will produce an example of this and post it too.</p><p>Hi Scott,</p>
<p>Here is an example of it failing in the interop case: <a href="https://github.com/ejackson/aotquestion2">https://github.com/ejackson/aotquestion2</a><br/>
The steps I'm following to compile this all up are</p>
<p>git clone git@github.com:ejackson/aotquestion2.git<br/>
cd aotquestion2/cljside/<br/>
lein uberjar<br/>
lein install<br/>
cd ../javaside/<br/>
mvn package<br/>
java -jar ./target/aotquestion-1.0-SNAPSHOT.jar</p>
<p>and it dies with this:</p>
<p>Exception in thread "main" java.lang.ClassCastException: cljside.core.Dontwork cannot be cast to cljside.core.Dontwork<br/>
at cljside.MyClass.makeDontwork(Unknown Source)<br/>
at aotquestion.App.main(App.java:8)</p>
<p>The error message is really confusing (to me, anyway), but I think its the same root problem as for the REPL case.</p>
<p>What do you see when you run the above ?</p><p>Ah, yes, looks like my initial attempt to reproduce was too simplistic. I used your second git repo, and can now confirm that it's failing for me with the same error.</p><p>I looked into this a little further and the AOT generated code looks correct, in the sense that both code paths appear to be returning the same type.</p>
<p>However, I wonder if this is really a ClassLoader issue, whereby two definitions of the same class are being loaded at different times, because that would cause the x.y.Class cannot be cast to x.y.Class exception that we're seeing here.</p><p>This could be related to <a href="http://dev.clojure.org/jira/browse/CLJ-1157" title="Classes generated by gen-class aren&#39;t loadable from remote codebase for mis-implementation of static-initializer">CLJ-1157</a> which deals with a ClassLoader issue with AOT compiled code.</p><p>I've tried <a href="http://dev.clojure.org/jira/secure/attachment/12711/20140121_fix_classloader.diff">this</a> patch attached to <a href="http://dev.clojure.org/jira/browse/CLJ-1157" title="Classes generated by gen-class aren&#39;t loadable from remote codebase for mis-implementation of static-initializer">CLJ-1157</a> and it did not solve this issue.</p><p>This bug seems to be rooted in different behaviour for do/let under compilation. Attached a patch showing these symptoms in the hope it helps people find the cause.</p><p>Just a quick note to confirm that this still seems to be around as of Clojure 1.7.0-alpha2. Don't have any useful input on possible solutions, sorry.</p><p>Duplicates - <a href="http://dev.clojure.org/jira/browse/CLJ-1495" title="Defining a record with defrecord twice breaks record equality"><del>CLJ-1495</del></a>, <a href="http://dev.clojure.org/jira/browse/CLJ-1132" title="For record type Rec, (instance? Rec (map-&gt;Rec {...})) need not return true, though (instance? Rec (Rec. ...)) does."><del>CLJ-1132</del></a></p><p>The attached patch fixes the classloader issues by routing RT.classForName &amp; variants through the DynamicClassLoader class cache before invoking Class.forName</p><p>Re-adding triaged status added by Alex Miller that got accidentaly nuked by a race-condition between my edits to the ticket description and Alex's ones</p><p>0001-<a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>&#45;make-clojure-resolve-to-the-correct-Class-in-v2.patch is the same as 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>-make-clojure-resolve-to-the-correct-Class-in.patch except it unconditionally looks for classes in the class cache of DynamicClassLoader, even if baseLoader() is not a DynamicClassLoader.<br/>
This fixes the bug of <a href="http://dev.clojure.org/jira/browse/CLJ-1457" title="once the compiler pops the dynamic classloader from the stack, attempts to read record reader literals will fail"><del>CLJ-1457</del></a> but might just be a workaround</p><p>Current patch blows up my Clojure build</p>
<p><a href="https://gist.github.com/MichaelBlume/aa26fc715cbbdf711290">https://gist.github.com/MichaelBlume/aa26fc715cbbdf711290</a></p><p>Michael: the current patch builds clojure fine for me, I'll try to reproduce. Which jvm version are you using?</p><p><span class="error">&#91;14:24&#93;</span><span class="error">&#91;michael.blume@tcc-michael-4:~/workspace/clojure((0fc43db...))&#93;</span>$ java -version<br/>
java version "1.8.0_25"<br/>
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)<br/>
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)<br/>
<span class="error">&#91;14:24&#93;</span><span class="error">&#91;michael.blume@tcc-michael-4:~/workspace/clojure((0fc43db...))&#93;</span>$ mvn -version<br/>
Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T13:58:10-07:00)<br/>
Maven home: /usr/local/Cellar/maven/3.2.3/libexec<br/>
Java version: 1.8.0_25, vendor: Oracle Corporation<br/>
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/jre<br/>
Default locale: en_US, platform encoding: UTF-8<br/>
OS name: "mac os x", version: "10.10.1", arch: "x86_64", family: "mac"</p>
<p>build was after I applied the patch to the current master branch of the clojure github repo</p><p>I am seeing a similar compilation error as Michael Blume, with both JDK 1.7 and 1.8 on Mac OS X 10.9.5.</p>
<p>By accident I found that if I take latest Clojure master and do 'mvn package', then apply the patch <a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>.patch dated Dec 11 2014, then do 'mvn package' again without 'mvn clean', it compiles with no errors. If I do 'mvn clean' then 'mvn package' in a patched tree, I get the error every time I've tried.</p><p>The updated patch fixes the LinkageError Andy and Michael were getting. </p>
<p>Andy, Michael, can you confirm?</p><p>Added more testcases to new patch</p><p>Cleaned up the patch from whitespace changes</p><p>I tried latest Clojure master plus patch <a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>-v4.patch, dated 12 Dec 2014, with Mac OS X 10.9.5 + JDK7, and Ubuntu Linux 14.04 with JDKs 6 through 9, and 'mvn clean' followed by 'mvn package' built and passed tests successfully with all of them.</p>
<p>I did notice that some files were created in the test directory that were not cleaned up by the end of the test, which you can use 'git status .' to see. Not sure if that is considered a bad thing for Clojure tests.</p><p>Thanks Andy, I've updated the patch and it now should remove all temporary classes created by the test.<br/>
It's probably not the best way to do it but I couldn't figure out how to do it another way.</p><p>Yep, looks good to me =)</p><p>Thanks first to Nicola for all his work so far on this!</p>
<p>Some feedback:<br/>
1) While the ticket itself isn't bad, I would really like to focus the title and description on a crisp statement of the real problem now that we understand it more. I'd like help on making sure we capture that correctly - how is this for a title: "Uses of Class.forName() miss classes in DynamicClassLoader cache?" ? </p>
<p>Similarly, the description should focus on the problem and show some examples. The defrecord one is good. The first example works for me before the patch and fails after? </p>
<p>2) The crux of this whole thing is the change in loading order in DCL.loadClass() - changing this is a big deal. We really need broader testing with things likely to be affected - off the top of my head: Immutant, Pomegranate, Leiningen, or anything else that monkeys with classloader stuff. Maybe something with Eclipse/OSGi if there is something testable out there.</p>
<p>3) DynamicClassLoader comments:<br/>
a) loadClass(String name) - I believe this is identical to the super impl, so can be removed.<br/>
b) findClass(String name) - now that we are hijacking loadClass(), I'm not sure it's even necessary to implement this or to call super.findClass() - if you get to the super.findClass(), I would expect that to always throw CNFE. Potentially, this method could even be removed (but this might do bad things if there are subclasses of DCL out there in the wild). <br/>
c) loadClass(String name, ...) - instead of calling findClass() and using the CNFE for control flow, you could just directly call findInMemoryClass(), then use a returned null as the decision factor. Again, this is possibly bad if there are DCL subclasses, so I'm on the fence about it. </p>
<p>4) Is the change in gen-interface something that should be a separate ticket? Seems like it could be separable.</p>
<p>5) I don't like the test changes with respect to set up and cleanup. The build already supports compiling a subset of test ns'es (like clojure.test-clojure.genclass.examples). I'd prefer to use that existing mechanism if at all possible. Check the build.xml for the hard-coded ns list. </p>
<p>6) What are the performance implications? I'm not expecting they are significant but we just made a bunch of changes for compilation performance and I'd hate to undo them all. Could findInMemoryClass be smarter about avoiding checks that can't succeed (avoiding "java.*" for example?).</p><p>1) It's not really about Class.forName() specifically, it's about DynamicClassLoader not being class cache aware in the loadClass method. The JVM uses the classloader loadClass method for resolving all kind of class usages,<br/>
including but not limited to Class.forName() (i.e. when loading some bytecode containing a "new" instruction, that class reference will be resolved via a call to loadClass)<br/>
I'll try to make the documentation a bit more clear, the first example is an exhibit of the bugged behaviour, the two calls should output the same hash.</p>
<p>2,4) So, there are 3 approaches to how DynamicClassLoader could go at it:</p>
<ul class="alternate" type="square">
<li>Prefer in-disk classes over in-memory classes, roughly the current approach (sometimes it will pick the in-memory class over the in-disk one causing weird bugs like Foo. and -&gt;Foo constructing different classes), has the<br/>
negative effect of breaking interaction between AOT compilation and JIT loading, which has created all sorts of troubles with redefinig deftypes/defprotocols in repls while having stale classfiles in disk.</li>
<li>Always pick the most-updated class, this has the advangate of being <b>always</b> correct but has several disadvantages that make it inpracticable in my opinion: we'd have to keep track of the timestamp in which a dynamic class<br/>
is defined, and make the loadClass implementation such that if there a class is both in-memory and in-disk, it compares the timestamps and select the most updated one. This would complicate the implementation a lot and we'd<br/>
likely have to pay a substantial performance hit.</li>
<li>Prefer in-memory classes over in-disk classes, the approach proposed in the current patches. It has the advantage of being almost always correct, make repl interaction &amp; jit/aot mixing work fine and the implementation is<br/>
mostly straightforward. The downside is that in cases like gen-class where an AOT class can actually be the most updated version, the in-memory version will be used. In clojure all the forms that do bytecode emission either<br/>
only do AOT compilation or do AOT compilation on demand and <b>always</b> load the class in memory, except gen-interface that doesn't load the class in memory if it's being AOT compiled. Changing its semantics to behave like the<br/>
other jit/aot compiling forms (deftype/defrecord/reify) is the only way to make this approach work so I don't think this should go in another ticket.</li>
</ul>
<p>5) I don't like the previous testing strategy either but couldn't figure out a better way. Thanks for the pointer on the already in-place infrastructure, I'll check it out and update the patch</p>
<p>In the meantime I've uploaded a new patch addressing 3 and 6. Specifically:<br/>
3) I removed the unnecessary loadClass(String) arity, I've made loadClass(String, boolean) use findInMemoryClass(String) directly rather than relying on findClass(String) since nowhere in the documentation it guarantess that<br/>
findClass will be used by loadClass. However I've left the findClass(String) implementation in place in case there's code out there that relies on this.<br/>
6) I haven't done any serious testing but I haven't noticed any significant difference in compile times for any of my tools.* contrib libraries with the current patch. Filtering "java.*" class names before the inMemory check<br/>
didn't seem to produce any difference so it's not included in the updated patch. However I'll probably include an alternative patch with that filtering to do more performance testings and see if it can actually help a bit.</p>
<p>All this said, I'm afraid that I won't have time to personally do an in-depth benchmarking &amp; cross project testing of this patch. I've been spending almost all the free time I had in the past weeks working through a bunch of tickets (mostly this one) but now because of school and other commitments I can't promise I will be able to do anything more than maintaining the current patch &amp; answering to any questions about the bug. Any help in moving this ticket further would be appreciated, in particular to address points 2 and 6.</p><p>Thanks Nicola. I'll certainly take over sheparding the bug and appeal to the greater community for help in broad testing when I think we're ready for that.</p><p>Updated patch with better tests, addressing Alex Miller's comments.</p><p>New in-the-wild case: <a href="https://groups.google.com/forum/#!topic/clojure/WH6lJwH1vMg">https://groups.google.com/forum/#!topic/clojure/WH6lJwH1vMg</a></p><p>I believe I have a case which is now broken because of this patch. I'm not 100% clear on the details so I might be wrong, but I can't see any other explanation for what I'm seeing. The bug I'm looking at is Cursive <a href="https://github.com/cursiveclojure/cursive/issues/748">#748</a>. Cursive calls into Leiningen in-process, and before doing that I create a new DynamicClassLoader which uses an IntelliJ PluginClassLoader as its parent. I believe that what is happening is that PluginClassLoader.loadClass() delegates to various parent classloaders then throws CNFE if it can't find the class in question. Am I correct in thinking that after this patch DCL now delegates to the parent classloader before checking its URL list, whereas previously it did not?</p><p>Colin, I opened <a href="http://dev.clojure.org/jira/browse/CLJ-1663" title="DynamicClassLoader delegates to parent classloader before checking in its URL list "><del>CLJ-1663</del></a> with a patch that should fix the issue. Can try it and comment on that ticket if it does?</p>ApprovalOkGlobal RankPatchCode and Test[CLJ-944] Compiler sometimes gives constant collections types which mismatch with their runtime valueshttp://dev.clojure.org/jira/browse/CLJ-944
Clojure<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(.containsKey {:one 1} :one)
;=&gt; ClassCastException clojure.lang.PersistentArrayMap cannot be <span class="code-keyword">cast</span> to clojure.lang.PersistentHashMap</pre>
</div></div>
<p>The map is a clojure.lang.PersistentArrayMap, which obviously has a containsKey method (<a href="https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentArrayMap.java#L95">https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/PersistentArrayMap.java#L95</a>). </p>
<p>Casting it works fine though:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(.containsKey ^clojure.lang.PersistentArrayMap {:one 1} :one)
;=&gt; <span class="code-keyword">true</span></pre>
</div></div>
<p>The problem is not present in Clojure 1.2.1.</p>
<p><b>Cause:</b> In the first example, the Compiler will parse the MapExpr (which reports a Java class of IPersistentMap) into a ConstantExpr whose value is a PersistentHashMap (and reports type as same). The .containsKey() call is a HostExpr which parses to an InstanceMethodExpr. </p>
<p>When emitted to bytecode, the ConstantExpr becomes a call to RT.map() to generate a map. RT.map() produces either a PersistentArrayMap or PersistentHashMap depending on size. The InstanceMethodExpr becomes a call to PersistentHashMap.containsKey() (based on the reported type of it's expression). In the case where RT.map() will produce a PersistentArrayMap at runtime (small maps), this causes a mismatch. </p>
<p>Note also that this same scenario occurs for constant sets, vectors, etc - all of those cases produce ConstantExpr's with the type of the concrete data structure, instead of the more general IPersistentMap, IPersistentSet, etc. All of those scenarios should be addressed in the same way. But, those interfaces do not capture the full capabilities of the returned object. The compiler and the marshaller need to agree on the types to produce.</p>
<p><b>Approach:</b> One approach (A) explored previously was to have MapExpr always produce a ConstantExpr containing a value of the same type that will be produced at runtime by RT.map(). While this works, it is also fragile, and sidesteps the real issue that MapExpr creates a ConstantExpr with a type that is too specific. Rich said this is the wrong approach.</p>
<p>(B) Another approach is to create an anonymous subclass of ConstantExpr in MapExpr that overrides getJavaClass() to report a more general type. This isn't done anywhere else and it may have negative ramifications (but I have not found any).</p>
<p>(C) Another approach is to follow the path of example #2 above. That example wraps the map in a MetaExpr which reports getJavaClass() returning the hinted type. In the case above, it hints the concrete type that happens to match the return of RT.map(). Instead, we could wrap the ConstantExpr in a MetaExpr hinted on the abstract type IPersistentMap (the return type of RT.map()). </p>
<p>RH - a more general (interface) type will not give full capabilities. Perhaps APersistent*</p>
<p><b>Patch:</b> </p>
Clojure 1.3 om Mac OS 10.7, Clojure 1.5.0 alpha1 on Linux x86_64 (OpenJDK 1.7.0 b147)CLJ-944Compiler sometimes gives constant collections types which mismatch with their runtime valuesDefectMajorClosedCompletedUnassignedAlf Kristian StøylecompilerSun, 4 Mar 2012 02:45:09 -0600Thu, 6 Mar 2014 07:54:15 -0600Thu, 6 Mar 2014 07:54:15 -0600Release 1.3Release 1.4Release 1.5Release 1.624<p>The attached patch fixes the issue, by emitting IPersistentMap instead of Persistent{Hash|Array}Map as class type for maps literals</p><p>I uploaded another patch fixing the same problem in a different way.<br/>
While 0001-Fix-for-<a href="http://dev.clojure.org/jira/browse/CLJ-944" title="Compiler sometimes gives constant collections types which mismatch with their runtime values"><del>CLJ-944</del></a>.patch makes clojure.lang.Complier.ConstantExpr#getJavaClass return clojure.lang.IPersistentMap for both clojure.lang.PersistentHashMap and clojure.lang.PersistentArrayMap, 0002-Fix-for-<a href="http://dev.clojure.org/jira/browse/CLJ-944" title="Compiler sometimes gives constant collections types which mismatch with their runtime values"><del>CLJ-944</del></a>.patch makes clojure.lang.Compiler.MapExpr#parse return a PersistentArrayMap if the length is &lt;= HASHTABLE_THRESHOLD, instead of always returning a PersistentHashMap.</p>
<p>This approach is more consistent, making the type of the compiler's internal representation of a map literal equal to the one of the reader.</p>
<p>Note that this second approach while being more consistent, breaks some tests that assume some operations on maps (specifically `seq` and `print`) to be order dependent, and written with the hash-map return order implementation in mind.</p>
<p>That should not be the case and if the second patch is preferred over the first one, I'll gladly fix those tests.</p><p>Approach #2, relying on consistent choice of concrete map class by size throughout, feels quite fragile.</p>
<p>Approach #1 seems to abuse the method name getJavaClass(), now having it return "get the base type I would need for cast".</p>
<p>Maybe there needs to be a different thing entirely?</p><p>Patch #2 should get merged (IMHO) regardless of the fragility of its approach to fixing this ticket's bug, since it fixes another bug:</p>
<p>prior to the patch:</p>
<p>user=&gt; (class {:a 1})<br/>
clojure.lang.PersistentArrayMap<br/>
user=&gt; (def a {:a 1})<br/>
#'user/a<br/>
user=&gt; (class a)<br/>
clojure.lang.PersistentHashMap</p>
<p>after the patch:</p>
<p>user=&gt; (class {:a 1})<br/>
clojure.lang.PersistentArrayMap<br/>
user=&gt; (def a {:a 1})<br/>
#'user/a<br/>
user=&gt; (class a)<br/>
clojure.lang.PersistentArrayMap</p>
<p>This should also lead to some <b>minor</b> performance enhancement since prior to this moment, every map def'ed would be a HashMap instead of an ArrayMap</p>
<p>So, I think patch #2 should be applied if not for this ticket's bug, at least for the reason stated above.<br/>
If somebody has any proposal for making this patch more solid regarding this ticket's bug, any help is welcome</p><p>This should not have passed screening. There are two issues, should be separate. I have no idea what has been screened nor what will be applied should it be approved. There's contention in the discussion but no resolution.</p><p>I don't think that there are two issues here.<br/>
The issue is only one: the compiler doesn't emit maps in a way consistent with what the reader returns and with how the compiler itself uses maps.<br/>
The symptoms are two: some interop calls fail, and def'ed vars with a literal map as value never use a PersistentArrayMap.</p>
<p>The underlying cause of those two symtoms is fixed by the patch 002 that i submitted (incorporated in Stu's clj944-plus-tests patch.</p>
<p>Stuart said that this approach feels fragile but the bug is caused by the fact that everywhere else clojure returns a PersistentArrayMap when the element count is &lt;= than the PersistentHashMap threshold, and when emitting maps, it doesn't.</p>
<p>Making clojure emit maps consistently with how clojure does internally everywhere else looks to me like the only solution, and I don't really see how making clojure consistent is a fragile approach.</p>
<p>But again, if somebody can suggest a better solution to this problem, I'll gladly submit another patch.</p><p>"Compiler makes different concrete maps then the reader" is not inherently a problem. But the code posted is a problem:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(.containsKey {:one 1} :one)
;=&gt; ClassCastException clojure.lang.PersistentArrayMap cannot be <span class="code-keyword">cast</span> to clojure.lang.PersistentHashMap</pre>
</div></div>
<p>i.e. .containsKey isn't working in this scenario. Stating exactly why is the path to a good discussion and fix.</p><p>I don't know how you're getting that exception.</p>
<p>This patch removes exactly that problem, I just tried it out on a fresh clojure git repo with only that patch applied and it works:<br/>
Clojure 1.6.0-master-SNAPSHOT<br/>
user=&gt; (.containsKey {:one 1} :one)<br/>
true</p><p>Nicola, I believe Rich is saying that:</p>
<p>1) the posted original example <b>is</b> showing a real problem<br/>
2) the title of the ticket and the suggested cause (compiler makes different concrete maps than the reader) is incorrect and that it is ok for them to make different concrete maps<br/>
3) the original error message (.containsKey isn't working) is the path to a correct diagnosis. </p>
<p>Rich is NOT saying that the patch doesn't resolve the error but rather that it resolves it in an undesired way due to a misdiagnosis of cause.</p><p>I'm going to try to describe as clearly as possible what's the problem and what could be the possible solutions (and thus what made me chose this as the best possible fix)</p>
<p>Problem: .containsKeys does not work on some maps</p>
<p>Diagnosis: <br/>
Constant maps are expressed as a ConstantExprs by the compiler; the ConstantExpr#getJavaClass method is implemented as a call to .getClass() on the constant object.</p>
<p>Since MapExpr#parse in case of a constant map always delegated to ConstantExpr a PersistentHashMap, it's clear that .getJavaClass on a ConstantExpr-wrapped map will <b>always</b> return PersistentHashMap.</p>
<p>This means that interop-calls will try to resolve PersistentHashMap methods instead of IPersistentMap (since they call .getJavaClass on the object to determine what to target).</p>
<p>When and why does this bug not manifest?</p>
<ul class="alternate" type="square">
<li>if the map is not constant:<br/>
user=&gt; (.containsKey {:one (Object.)} :one)<br/>
true</li>
<li>if the map gets compiled as opposed to being .eval'ed<br/>
usser=&gt; (def a (.containsKey {:one 1} :one))<br/>
#'user/a<br/>
user=&gt; a<br/>
true</li>
</ul>
<p>This also mean that currently, maps that are hold in a Var or that are AOT compiled are always PHMs and never PAMs (removing a possible optimization)</p>
<p>What are the possible solutions?</p>
<ul class="alternate" type="square">
<li>Don't wrap constant maps in ConstantExprs so we always use MapExpr and remove the problem.</li>
<li>Special-case ConstantExpr#getJavaClass to return IPersistentMap for PHMs and PAMs (approach #1)</li>
<li>Be consistent in MapExpr#parse and create PAMs when the element number is above the PHM threshold as we do EVERYWHERE but in there. (approach #2)</li>
</ul>
<p>Of the three possible solutions, the third seems to me the best one.</p><p>Thanks for this - it was a very useful summary for me as I'm still trying to get my head around all the details of this ticket. Stepping back a bit, I'd say in your diagnosis that the point at which I see something wrong is when MapExpr#parse returns something that depends on a particular implementation type (via getJavaClass). </p>
<p>Idea: what if MapExpr#Expr instead returned a anonymous subclass of ConstantExpr that overrode getJavaClass() to return IPersistentMap in this case? </p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java"><span class="code-keyword">return</span> <span class="code-keyword">new</span> ConstantExpr(m) {
<span class="code-keyword">public</span> <span class="code-object">Class</span> getJavaClass() {
<span class="code-keyword">return</span> IPersistentMap.class;
} };</pre>
</div></div>
<p>That way, the ConstantExpr only commits to the abstraction. This seems to also fix the problem to me.</p>
<p>Separately, I agree that it may be useful as an optimization to emit a PAM instead of a PHM in this case (maybe even in all constant cases?). But I think we should not rely on two different pieces of code producing the identical map impl - that should be hidden behind the abstraction (IPersistentMap). I think I would like to break that out as an independent ticket though.</p><p>I see that my suggestion is effectively similar to the first patch on this ticket but (imho) cleaner in that it puts the change at the point where the knowledge exists and can thus be a simpler check. </p><p>Patch clj944-plus-tests.patch no longer applies cleanly to latest master, probably due to changed context lines in a test file, but I haven't checked. Not worth updating until/unless there is agreement on what should be changed.</p><p>I would very like to see this fixed and will happily update my patches, however I'd first like some feedback on the correct approach here before making yet another patch.</p>
<p>Should we go with one of my two approaches or should I implement what Alex suggested?<br/>
Each approach fixes this bug, I stated before what's causing this and I have explained my solutions, Alex's one looks reasonable too, can we please get an opinion on what's the best move to get a satisfying fix for this?</p>
<p>Thanks</p><p>Sorry it's not visible on the ticket, but I have actually spent several hours looking at this since my last comment but have not reached a final conclusion. I also would really like to see it fixed and have not given up on it yet.</p><p>The most direct cause of the exception in the example of how to reproduce it is due to the compiler emitting a type check on a type that is different than the constant emitted.</p>
<p>This is just tossing out ideas, perhaps brain-dead ones, but eliminating that type check would eliminate the exception. Also, changing the emitted type check to be for a more general type than than the constant, e.g. IPersistentMap, should also eliminate the exception.</p><p>Alex, there's no need to be apologetic, I was in no way trying to imply a lack of interest, just asking for directions <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p>
<p>Wrapping up for screeners/Rich:</p>
<p>Here is a diagnosis of the bug: <a href="http://dev.clojure.org/jira/browse/CLJ-944?focusedCommentId=31694&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-31694">http://dev.clojure.org/jira/browse/CLJ-944?focusedCommentId=31694&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-31694</a></p>
<p>To me now it looks like the best solutions right now are either:</p>
<ul class="alternate" type="square">
<li>My #2 solution: making MapExpr#parse return a PAM or a PHM consistently with RT/map</li>
<li>Alex's solution: making MapExpr#Expr return an anonymous subclass of ConstantExpr overriding getJavaClass() to return IPersistentMap</li>
</ul>
<p>Right now my patch does not apply to master, and there is no patch implementing Alex's solution, once I get a feedback on which approach is preferable i can: update my patch to apply to master or implement Alex's solution (Alex, if you want to do it yourself I will not step in)</p>
<p>I have actually built a couple of variant patches locally but still have not convinced myself that any of them are the right patch to recommend. I hope to look at it again in the next couple weeks. I think the ticket itself is in the correct state (Incomplete). Ideally I would have a few minutes to look at it with Rich for further direction.</p><p>Any update on this? this is still marked for clojure 1.6</p><p>Actually yes! I spent some time on this today and have something to attach.</p><p>A problem I see with approaches b and c is that they don't address the issue I raised about (def a {:a 1}) here: <a href="http://dev.clojure.org/jira/browse/CLJ-944?focusedCommentId=29885&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-29885">http://dev.clojure.org/jira/browse/CLJ-944?focusedCommentId=29885&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-29885</a></p>
<p>Rich stated that it's a different issue than the one being discussed in this ticked, I don't agree, to me it's simply another symptom of the same bug.</p>
<p>I want to suggest a fourth alternative approach to this that combines approach -a with whichever is preferred between -b and -c.</p><p>Nicola, I'm not getting your point here (the linked comment doesn't seem to match up to the comments about (def a {:a 1}) so that's confusing me). In the case of (def a {:a 1}), my reading of the bytecode is that RT.map() (which returns IPersistentMap) will be invoked <b>at runtime</b> to generate the map. There is more than one potential concrete class for a (depending on size), but nothing is expecting it to be a particular concrete type here. I do not expect literal maps to retain order or be ArrayMaps (you should call array-map if you want that).</p>
<p>I'm hoping to get some consensus from Rich about the newly updated problem, cause, and approach before I move further. I'd currently classify the problem as the failed .containsKey call, the cause as the compiler replacing the MapExpr with an expression that specifies its type more narrowly than it should, and B and C as some candidate solutions.</p>
<p>If we want the compiler to generate concrete maps that match RT.map(), I think that's ok too, but I don't see it as the cause right now.</p><p>Alex, I'm sorry here's the correct link: <a href="http://dev.clojure.org/jira/browse/CLJ-944?focusedCommentId=30684&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-30684">http://dev.clojure.org/jira/browse/CLJ-944?focusedCommentId=30684&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-30684</a></p>
<p>Your analysis of the emitted bytecode is correct, however when the code is loaded JIT, calls to `def` will be interpreted <a href="https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L409-L434">https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L409-L434</a> as a result, if the initexpr is a ConstantExpr, the in-memory value of that ConstantExpr will be used and if it's the case that the expression is a map, that will always be a PHM.</p>
<p>I understand that switching between PAMs and PHMs is merely an optimization but having code behave differently if it's AOT compiled rather than JIT loaded is undesiderable IMHO.</p><p>Immediatelly after posting the previous response it occurred to me that approaches b and c have an inherent problem:<br/>
with that approach it's no longer possible to access public fields/methods in instances of PAMs or PHMs that do not belong in IPersistentMap without incurring in reflection.</p>
<p>This means that for example calls to java.lang.Map instance methods would require type-hinting the map literal while now it can be resolved at compile-time without the need of type-hinting.</p>
<p>Here's an example using a first patch that implements the fix as described by approaches -b or -c</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (set! *warn-on-reflection* <span class="code-keyword">true</span>)
<span class="code-keyword">true</span>
;; pre patch
user=&gt; (.isEmpty {:a 1})
<span class="code-keyword">false</span>
;; after patch
user=&gt; (.isEmpty {:a 1})
Reflection warning, NO_SOURCE_PATH:2:2 - reference to field isEmpty on clojure.lang.IPersistentMap can't be resolved.
<span class="code-keyword">false</span></pre>
</div></div>
<p>Remove from 1.6 release. Still needs more analysis.</p><p>Reopened just to fix the ticket fields.</p>ApprovalOkGlobal RankPatchCode and Test[CLJ-918] Vars with {:macro true} meta data do not work when loaded from AOT compiled filehttp://dev.clojure.org/jira/browse/CLJ-918
Clojure<p>When defining a var with <tt>^{:macro true</tt>} the call of the binded macro does emit a warning when the definition has been AOT compiled.</p>
<p>See example outputs with demo code here: <a href="https://refheap.com/paste/389">https://refheap.com/paste/389</a></p>
<p>Bronsa on #clojure created a patch: <a href="http://sprunge.us/bWcc">http://sprunge.us/bWcc</a></p>Tested with 1.3.0 and 1.4.0CLJ-918Vars with {:macro true} meta data do not work when loaded from AOT compiled fileEnhancementTrivialClosedDuplicateUnassignedJannik SchorgCompilerenhancementmetadataMon, 23 Jan 2012 16:06:06 -0600Thu, 13 Sep 2012 14:29:01 -0500Thu, 13 Sep 2012 14:29:01 -050001<p>Duplicate of <a href="http://dev.clojure.org/jira/browse/CLJ-1021" title="(let [i 5] (defmacro m [v] v) m) interprets m as a function, not a macro">CLJ-1021</a>. Later ticket kept in preference to this one, because it has a patch and this one does not.</p>Global Rank[CLJ-887] Error when calling primitive functions with destructuring in the arg vectorhttp://dev.clojure.org/jira/browse/CLJ-887
Clojure<p>If one defines a primitive-taking function with destructuring, calling that function will result in a ClassCastException, IFF the primitive return-type hint is present.</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>Clojure 1.4.0-master-SNAPSHOT
user=&gt; (defn foo [[a b] ^long x ^long y] 0)
#'user/foo
user=&gt; (foo [1 2] 3 4)
0
user=&gt; (defn foo ^long [[a b] ^long x ^long y] 0)
#'user/foo
user=&gt; (foo [1 2] 3 4)
ClassCastException user$foo cannot be cast to clojure.lang.IFn$OLLL user/eval9 (NO_SOURCE_FILE:4)
user=&gt; (pst)
ClassCastException user$foo cannot be cast to clojure.lang.IFn$OLLL
user/eval9 (NO_SOURCE_FILE:4)
clojure.lang.Compiler.eval (Compiler.java:6493)
clojure.lang.Compiler.eval (Compiler.java:6459)
clojure.core/eval (core.clj:2796)
clojure.main/repl/read-eval-print--5967 (main.clj:244)
clojure.main/repl/fn--5972 (main.clj:265)
clojure.main/repl (main.clj:265)
clojure.main/repl-opt (main.clj:331)
clojure.main/main (main.clj:427)
clojure.lang.Var.invoke (Var.java:397)
clojure.lang.Var.applyTo (Var.java:518)
clojure.main.main (main.java:37)
nil
</pre>
</div></div>
<p><b>Cause:</b> This was happening because maybe-destructured returned the arg vector without the type hint, so the function was getting compiled to a IFn$OLLO rather than a IFn$OLLL but the :arglists vector in the var meta was still tagged, so the compiler thought that foo was a IFn$OLLL.</p>
<p><b>Approach:</b> This patch addresses this by preserving the original meta on the fn arglist.</p>
<p><b>Patch:</b> 0001-don-t-remove-meta-from-arg-vector-in-maybe-destructu.patch</p>
<p><b>Screened by:</b> Alex Miller</p>CLJ-887Error when calling primitive functions with destructuring in the arg vectorDefectMajorClosedCompletedUnassignedAlexander TaggartcompilerTue, 29 Nov 2011 15:46:47 -0600Fri, 29 Aug 2014 12:00:25 -0500Fri, 29 Aug 2014 12:00:25 -0500Release 1.6Release 1.7Release 1.701<p>This was happening because maybe-destructured returned the arg vector without the type hint, so the function was getting compiled to a IFn$OLLO rather than a IFn$OLLL but the :arglists vector in the var meta was still tagged, so the compiler thought that foo was a IFn$OLLL.</p>
<p>This patch addresses this by preserving the original meta on the fn arglist</p><p>Weirdly, I saw this happen today in my own code. <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p>ApprovalOkGlobal RankPatchCode[CLJ-874] defrecord factory inaccessibly from within type implementationhttp://dev.clojure.org/jira/browse/CLJ-874
Clojure<p>Discovered this issue working through <a href="https://github.com/relevance/labrepl">https://github.com/relevance/labrepl</a> when trying to use the new factory sytax for records:</p>
<p>(defprotocol Player<br/>
(choose <span class="error">&#91;p&#93;</span>)<br/>
(update-strategy <span class="error">&#91;p me you&#93;</span>))</p>
<p>(defrecord Mean <span class="error">&#91;last-winner&#93;</span><br/>
Player<br/>
(choose <span class="error">&#91;_&#93;</span> (if last-winner last-winner (random-choice)))<br/>
(update-strategy <span class="error">&#91;_ me you&#93;</span> (-&gt;Mean (when (iwon? me you) me))))</p>
<p>Notice that Mean returns a new instance with a different strategy. However, the factory methods are not defined until after the record has been created thus this results in a syntax error. To fix this I updated the macro to declare the factory methods before the record is emitted.</p>
CLJ-874defrecord factory inaccessibly from within type implementationDefectMinorClosedCompletedUnassignedKurt HarrigerCompilerFri, 11 Nov 2011 11:31:52 -0600Fri, 2 Dec 2011 09:03:00 -0600Fri, 2 Dec 2011 09:03:00 -0600Release 1.440<p>On github: <a href="https://github.com/kurtharriger/clojure/tree/fix-defrecord">https://github.com/kurtharriger/clojure/tree/fix-defrecord</a></p><p>looks good to me</p><p>Screened against e58a87fac72ed4b84a1d92f1e455b92d7ed3ef39</p>ApprovalOkGlobal RankPatchCode[CLJ-865] Macroexpansion discards &form metadatahttp://dev.clojure.org/jira/browse/CLJ-865
Clojure<p>This patch changes the behavior of metadata when used in conjunction with macros. The metadata &amp;form is now merged with the metadata of the macro call sexpr. This allows users to either type-hint the inner or the outer form in a macro call and have somewhat better results. In the past, the metadata from the macroexpand was used as-is. This disallowed code like the following, to work without reflection:</p>
<p>(.trim ^String (when true "hello "))</p>
<p><b>Patch:</b> 2013-10-11_<a href="http://dev.clojure.org/jira/browse/CLJ-865" title="Macroexpansion discards &amp;form metadata">CLJ-865</a>_Fix-With-Tests.diff<br/>
<b>Screened by:</b> Timothy Baldridge</p>
<p>--------- Implementation Details ----------</p>
<p>As discussed in <a href="http://groups.google.com/group/clojure/browse_thread/thread/2690cb6ca0e8beb8">http://groups.google.com/group/clojure/browse_thread/thread/2690cb6ca0e8beb8</a> there is a "surprise factor" when type-hinting an expression that represents a macro, such as with (.length ^String (doto (identity "x") prn)). Here the doto macro discards the metadata on &amp;form, causing a reflective lookup. This has the effect that while expressions representing function calls can be type-hinted, expressions representing macros in general cannot. The doto macro could be rewritten to respect its &amp;form metadata, but doing this for every macro in existence would be tedious and error-prone. Instead, I propose a change to the compiler, to cause macroexpansion to hang onto the metadata automatically.</p>
<p>The first patch attached adds a test for the behavior I propose: this test fails. After applying the second patch, the test passes.</p>
<p>There are a couple points that merit further consideration before accepting my patch:</p>
<ul class="alternate" type="square">
<li>I'm not sure I actually got the Java code formatted correctly. My editor is not well-configured to get the clojure/core style right automatically.</li>
<li>My solution is to take the &amp;form metadata, drop :line/:file keys, and then merge with the returned metadata, with &amp;form taking precedence. I'm not sure whether this is the right approach in all cases, even though it works for :tag metadata.</li>
<li>I achieved this with a change to the compiler, which makes it fairly heavy-weight. It should be possible to instead adjust defmacro if changes to the compiler are not desirable. However, I believe this would involve substantially more work and be harder to test (for example, multiple arities complicate things). It seems nicer to treat the macroexpansion as a black box and then make metadata tweaks to the result, rather than modifying their actual defmacro code.</li>
<li>If a macro expands to something that is not an IObj, such as an Integer, then my patch silently discards the caller's metadata. Would it be better to throw an exception?</li>
</ul>
CLJ-865Macroexpansion discards &form metadataEnhancementMinorOpenUnresolvedUnassignedAlan MalloyCompilerWed, 26 Oct 2011 15:19:11 -0500Fri, 5 Dec 2014 13:54:00 -06001610<p>So I went ahead and did the work of making this change in clojure.core/defmacro instead of clojure.lang.Compiler/macroexpand1. It was even worse than I expected: I didn't realize we don't yet have syntax-quote or apply at this stage in bootstrapping, so writing a non-trivial macroexpansion requires a huge amount of (list `foo (list `bar 'local-name)) and so forth.</p>
<p>I'm sure the version I wrote is not optimal, but it seemed simpler to piggyback on defn, and then use alter-var-root to shim the metadata management in, than it would have been to expand to the correct thing in the first place.</p>
<p>Anyway, attached patch #3 could be applied instead of #2 to resolve the issue in clojure.core instead of clojure.lang. The tests added in patch #1 pass either way.</p><p>I realized I can do this with a named private function instead of an anonymous function, reducing the amount of mess defmacro itself has to generate. Patch 4 is, I think, strictly better than Patch 3, if a Clojure implementation is preferred to one in Java.</p><p>I prefer patch 0002 in Java over either 0003 or 0004. Patch 0002 keeps the knowledge of how to invoke macro fns (specifically the extra &amp;form and &amp;env args) in one place, macroexpand1 rather than duplicating that knowledge in core.clj as well. Note patch 0001 is just tests.</p>
<p>The proposed default macroexpansion behavior is more useful than what we currently have, but there are two details I'd like to think about a bit more:</p>
<p>1) In exchange for a more useful default, macro writers lose the ability to consume their &amp;form metadata and have control over the resulting form metadata without the &amp;form metadata overridding it. That is, macros are no longer in complete control of their output form.</p>
<p>2) Rule (1) above has hardcoded exceptions for :line and :file, where &amp;form metadata is unable to override the results returned by the macro.</p><p>This patch incorporates all previous patches to this issue.</p>
<p>On the clj-dev mailing list, Andy Fingerhut suggested a new metadata key for allowing the macro author to specify "I've looked at their &amp;form metadata, and this form is exactly what I want to expand to, please don't change the metadata any further." I've implemented this, and I think it addresses Chouser's concern about needing a way to "break out" of the improved-default behavior.</p>
<p>One open question is, is :explicit-meta the right key to use? I spent some time tracking down a bug caused by my forgetting the keyword and using :explicit-metadata in my test; perhaps something more difficult to get confused by is available.</p><p>clj-865-updated-v2-patch.txt dated Aug 14 2013 is identical to Alan Malloy's updated.patch dated Jun 1 2012. I simply updated the patch to apply cleanly to latest master after some context lines in the test file macros.clj had gone bad due to recent commits.</p><p>Added updated patch that works against master, and also removes COLUMN_KEY from the macro's metadata</p><p>Added patch that contains all fixes plus a few more tests. </p><p>Since this could break things, we could just take metadata on the macro name to ask for this:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(defmacro ^:keep-meta simple-macro [f arg]
`(~f ~arg))</pre>
</div></div>
<p>or something</p><p>Sure, I'll put together that patch. I'm worried, though, that if it's not the default, it will just never get used, and we'll be in effectively the same situation we are now, where no macros do this right. I don't foresee anyone going through their libraries to add ^:keep-meta on every macro.</p><p>I updated the patch to behave as Rich requested, but it caused a test regression that I can't figure out, in the handling of either refer or private vars. Hopefully someone else can run the tests and figure out what is missing here; my change is supposed to be opt-in, and I can't see where I've gone wrong.</p><p>Alan, your patch clj865.patch dated Dec 3, 2013 has some HTML cruft at the beginning and end, but even after removing that it does not apply cleanly to the latest Clojure master as of today. I understand that you say it needs more work, but it would be easier for others who wish to try it out if it applied cleanly.</p><p>Sorry Andy, and thanks for noticing. I haven't been on a very developer-friendly computer recently, but I'll try to fix the patch tonight.</p><p>Here's a fix to the patch. I verified that this applies cleanly to current master.</p><p>To clarify, it's the file named clj-865.patch. I didn't realize JIRA wouldn't make it clear which file I uploaded along with the comment.</p><p>As of Eastwood version 0.2.0, it includes a new warning :unused-meta-on-macro that will warn whenever metadata is applied to a macro invocation, with the known exception of clojure.core/fn, which explicitly uses metadata applied to it. <a href="https://github.com/jonase/eastwood#unused-meta-on-macro">https://github.com/jonase/eastwood#unused-meta-on-macro</a></p>ApprovalVettedGlobal RankPatchCode and Test[CLJ-703] Improve writeClassFile performancehttp://dev.clojure.org/jira/browse/CLJ-703
Clojure<p>This <a href="http://groups.google.com/group/clojure/browse_thread/thread/be0c45b3a4f3d46/36337ac7260d7d0a?lnk=gst&amp;q=writeclassfile#36337ac7260d7d0a">Discussion</a> about timing issues when writing class files led to the the current implementation of synchronous writes to disk. This leads to bad performance/system-load when compiling Clojure code.</p>
<p>This <a href="http://groups.google.com/group/clojure-dev/browse_thread/thread/23213b787d3ec60f#">Discussion</a> questioned the current implentation.</p>
<p>Synchronous writes are not necessary and also do not ensure (crash while calling write) valid classfiles.</p>
<p>These Patches (0001 is just a code cleanup for creating the directory structure) ensures atomic creation of classfiles by using File.renameTo()</p>
CLJ-703Improve writeClassFile performanceEnhancementCriticalOpenUnresolvedUnassignedJürgen HötzelCompilerperformanceTue, 4 Jan 2011 12:58:38 -0600Wed, 8 Oct 2014 06:00:18 -0500Release 1.3199
<p>Removing sync makes clojure build much faster. I wonder why it was added in the first place? I guess only Rich knows? I assume that it is not necessary.</p>
<p>If we are removing sync though, I wouldn't bother with the atomic rename stuff. Doing that sort of thing can cause problems on some platforms, eg with search indexers and virus checkers momentarily locking files after they are created. </p>
<p>The patch seems to be assuming that sync is there for some reason, but my initial assumption would be that sync isn't necessary - perhaps it was working around some issue that no longer exists?</p>
<p>Although its unlikely: there is a possible race condition "loading a paritally written classfile"?:</p>
<p><a href="https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L393">https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L393</a></p><p>The new improve-writeclassfile-perf version of the patch combines the two previous patches into a single patch file and brings them up-to-date with master. I can split the two changes back out into separate patch files if desired, but I figured out current tooling is more geared towards a single patch being applied.</p><p>FWIW, both fixes look sane. The first one is a nice cleanup. The second one is a little more interesting in that uses a rename operation to put the final file into place. It removes the sync call, which does make things faster. In general, if we're concerned about on-disk consistency, we should really have a combination of the two: write the full contents to a tmp file, sync it, and atomically rename to the destination file name.</p>
<p>Neither the current master, nor the current patch will guarantee on-disk consistency across a machine wide crash. The current master could crash before the sync() occurs, leaving the file in an inconsistent state. With the patch, the OS may not get the file from file cache to disk before an OS level crash occurs, and leave the file in an inconsistent state as well. The benefit of the patch version is that the whole file does atomically come into view at once. It does have a nasty side effect of leaving around a temp file if the compiler crashes just before the rename though.</p>
<p>Perhaps a little more work to catch an exception and clean up is in order? In general, I like the patched version better.</p><p>File.renameTo returns false on (most?) errors, but the patch doesn't check for failure. Docs say "The return value should always be checked to make sure that the rename operation was successful." Failure might be especially likely on Windows, where files are opened by others without FILE_SHARE_DELETE.</p><p>We've been wondering why our compilation times on linux were so slow. It became the last straw when we walked away from one project and came back after 15 minutes and it was not done yet.</p>
<p>After some fruitless investigation into our linux configuration and lein java args, we stumbled upon this issue via the associated Clojure group thread. Upon commenting out the flush() and sync() lines (<a href="https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L7171-L7172">https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L7171-L7172</a>) and compiling Clojure 1.6 ourselves, our projects all started compiling in under a minute.</p>
<p>Point being, can we at least provide some flag to allow for "unsafe compilation" or something? As it is, this is bad enough that we've manually modified all our local versions of Clojure to work around the issue.</p><p>Additional motivation: This becomes <b>really</b> unpleasant on an encrypted filesystem, since write and read latency become higher.</p>
<p>As a partial workaround, I've been using this script to mount a ramdisk over top of target, which speeds up compilation 2-4x: <a href="https://gist.github.com/timmc/6c397644668bcf41041f">https://gist.github.com/timmc/6c397644668bcf41041f</a> (but removing flush() and sync() entirely would probably speed things up even more, if safe)</p><p>I'd like to explore this issue further as I also don't think the flush and sync calls add any value, but do have a severe impact on performance.</p>
<p>To resurrect the discussion, I've attached a new patch with the following approach:</p>
<ul class="alternate" type="square">
<li>create a temporary file</li>
<li>write class bytecode to temporary file, no flush or sync</li>
<li>close temporary file</li>
<li>atomic rename of temporary file to class file name</li>
</ul>
<p>It is different to previous patches in that:</p>
<ul class="alternate" type="square">
<li>it applies cleanly to master</li>
<li>it checks return value from File.renameTo</li>
<li>it omits proposed File.mkdirs change as the current implementation is actually converting from an "internal name", where forward slashes are assumed (splits on "/"), to a platform specific path using File.separator. I'm not convinced that the previous patch is safe on all versions of Windows, and I think it's separate from the main issue here.</li>
</ul>
<p>I opted for the atomic rename of a temp file to avoid leaving empty class files with a correct expected class file name in case of failure.</p>
<p>It is my understanding that this patch will guarantee that:</p>
<ul class="alternate" type="square">
<li>when writeClassFile returns successfully, a class file with the expected name will exists, and subsequent reads from that file name will return the expected bytecode (possibly from kernel buffers).</li>
<li>when writeClassFile fails, a class file with the expected name will not have been created (or updated if it previously existed).</li>
</ul>
<p>Anything preventing the operating system from flushing its buffers to disk (power failure etc) might leave a corrupt (empty, partially written) class file. It's my opinion that that is acceptable behaviour, and worth the performance improvement (I'm seeing AOT compilation reduced from 1m20s -&gt; 22s on one of our codebases, would be happy to provide more benchmarks if requested).</p>
<p>Would be grateful for feedback/testing of this patch.</p><p>We're testing this patch on various projects/platforms at my company. So far we've seen:</p>
<ul class="alternate" type="square">
<li>Significantly reduced compilation times on Linux (two typical examples: 30s to 15s, 1m30s to 30s)</li>
<li>No significant change in compilation times on Mac OSX.</li>
<li>File.renameTo consistently failing on a Windows machine.</li>
</ul>
<p>My understanding is that the performance difference between Linux and OSX is due to differences in how these platforms implement fsync. OSX by default does not actually tell the drive to flush its buffers (requires fcntl F_FULLSYNC for this, not used by JVMs) <span class="error">&#91;1&#93;</span>, whereas Linux does <span class="error">&#91;2&#93;</span>.</p>
<p>Our very limited test shows (as was previously pointed out) that File.renameTo is problematic on Windows. I've attached a new patch that doesn't use rename, and only has the the sync call removed (flush is a no-op for FileOutputStreams). We're currently testing this patch.</p>
<p>The drawback of this patch is that it may leave correctly named, but empty class files if the write fails. One option would be to try and delete the file in the catch block. Personally, I wouldn't expect a compilation that failed because of OS/IO reasons to leave my classfiles in a consistent state.</p>
<p><span class="error">&#91;1&#93;</span>: "Note that while fsync() will flush all data from the host to the drive (i.e. the "permanent storage device"), the drive itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order sequence.": <a href="https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html">https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html</a></p>
<p><span class="error">&#91;2&#93;</span>: "<span class="error">&#91;...&#93;</span> includes writing through or flushing a disk cache if present."<br/>
<a href="http://man7.org/linux/man-pages/man2/fsync.2.html">http://man7.org/linux/man-pages/man2/fsync.2.html</a></p>ApprovalTriagedGlobal RankPatchCode