Clojure JIRAhttp://dev.clojure.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+10010+AND+labels+%3D+aot
An XML representation of a search requesten-us4.464925-07-2011[CLJ-1743] Avoid compile-time static initialization of classes when using inheritancehttp://dev.clojure.org/jira/browse/CLJ-1743
Clojure<p>I'm working on a project using Clojure and RoboVM. We use AOT compilation to compile Clojure to JVM classes, and then use RoboVM to compile the JVM classes to native code. In our Clojure code, we call Java APIs provided by RoboVM, which wrap the native iOS APIs. </p>
<p>But we've found an issue with inheritance and class-level static initialization code. Many iOS APIs require inheriting from a base object and then overriding certain methods. Currently, Clojure runs a superclass's static initialization code at compile time, whether using ":gen-class" or "proxy" to create the subclass. However, RoboVM's base "ObjCObject" class <span class="error">&#91;1&#93;</span>, which most iOS-specific classes inherit from, requires the iOS runtime to initialize, and throws an error at compile time since the code isn't running on a device. </p>
<p><a href="http://dev.clojure.org/jira/browse/CLJ-1315" title="Don&#39;t initialize classes when importing them"><del>CLJ-1315</del></a> addressed a similar issue by modifying "import" to load classes without running static initialization code. I've written my own patch which extends this behavior to work in ":gen-class" and "proxy" as well. The unit tests pass, and we're using this code successfully in our iOS app.</p>
<p><b>Patch:</b> clj-1743-2.patch</p>
<p>Here's some sample code that can be used to demonstrate the current behavior (Full demo project at <a href="https://github.com/figly/clojure-static-initialization">https://github.com/figly/clojure-static-initialization</a>):</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>Demo.java</b></div><div class="codeContent panelContent">
<pre class="code-java"><span class="code-keyword">package</span> clojure_static_initialization;
<span class="code-keyword">public</span> class Demo {
<span class="code-keyword">static</span> {
<span class="code-object">System</span>.out.println(<span class="code-quote">"Running <span class="code-keyword">static</span> initializers!"</span>);
}
<span class="code-keyword">public</span> Demo () {
}
}</pre>
</div></div>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>gen_class_demo.clj</b></div><div class="codeContent panelContent">
<pre class="code-java">(ns clojure-<span class="code-keyword">static</span>-initialization.gen-class-demo
(:gen-class :<span class="code-keyword">extends</span> clojure_static_initialization.Demo))</pre>
</div></div>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>proxy_demo.clj</b></div><div class="codeContent panelContent">
<pre class="code-java">(ns clojure-<span class="code-keyword">static</span>-initialization.proxy-demo)
(defn make-proxy []
(proxy [clojure_static_initialization.Demo] []))</pre>
</div></div>
<p><span class="error">&#91;1&#93;</span> <a href="https://github.com/robovm/robovm/blob/master/objc/src/main/java/org/robovm/objc/ObjCObject.java">https://github.com/robovm/robovm/blob/master/objc/src/main/java/org/robovm/objc/ObjCObject.java</a> </p>CLJ-1743Avoid compile-time static initialization of classes when using inheritanceEnhancementMajorOpenUnresolvedUnassignedAbe FettigaotcompilerinteropTue, 2 Jun 2015 14:59:18 -0500Mon, 27 Jul 2015 18:06:30 -0500Release 1.6Release 1.735<p>No changes from previous, just updated to apply to master as of 1.7.0-RC2.</p><p>If you had a sketch to test this with proxy and gen-class, that would be helpful.</p><p>Sure, what form would you like for the sketch code? A small standalone project? Unit tests?</p><p>Just a few lines of Java (a class with static initializer that printed) and Clojure code (for gen-class and proxy extending it) here in the test description that could be used to demonstrate the problem. Should not have any dependency on iOS or other external dependencies.</p><p>Sample code added, let me know if I can add anything else!</p><p>Just out of curiosity, what are the odds this could make it into 1.8?</p><p>unknown.</p>ApprovalTriagedGlobal RankPatchCode[CLJ-1741] deftype class literals and instances loaded from different classloaders when recompiling namespacehttp://dev.clojure.org/jira/browse/CLJ-1741
Clojure<p><b>Scenario:</b> Given two files:</p>
<p>src/dispatch/core.clj:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns dispatch.core (:require [dispatch.dispatch]))</pre>
</div></div>
<p>src/dispatch/dispatch.clj:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns dispatch.dispatch)
(deftype T [])
(def t (-&gt;T))
(println <span class="code-quote">"T = (class t):"</span> (= T (class t)))</pre>
</div></div>
<p>Compile first core, then dispatch:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">java -cp src:target/classes:clojure.jar -Dclojure.compile.path=target/classes clojure.main
user=&gt; (compile 'dispatch.core)
T = (class t): <span class="code-keyword">true</span>
dispatch.core
user=&gt; (compile 'dispatch.dispatch)
T = (class t): <span class="code-keyword">false</span> ;; expected <span class="code-keyword">true</span>
dispatch.dispatch</pre>
</div></div>
<p>This scenario more commonly occurs in a leiningen project with <tt>:aot :all</tt>. Files are compiled in alphabetical order with :all. In this case, dispatch.core will be compiled first, then dispatch.dispatch.</p>
<p><b>Cause:</b></p>
<p>(compile 'dispatch.core) </p>
<ul class="alternate" type="square">
<li>transitively compiles dispatch.dispatch</li>
<li>writes .class files to <b>compile-path</b> (which is on the classpath)</li>
<li>assertion passes</li>
</ul>
<p>(compile 'dispatch.dispatch) </p>
<ul class="alternate" type="square">
<li>due to prior compile, load dispatch.dispatch__init is loaded via the appclassloader</li>
<li>-&gt;T constructor will use new bytecode to instantiate a T instance - this uses appclassloader, loaded from compiled T on disk</li>
<li>however, T class literals are resolved with RT.classForName, which checks the dynamic classloader cache, so uses old runtime version of T, instead of on-disk version</li>
</ul>
<p>In 1.6, RT.classForName() did not check dynamic classloader cache, so loaded T from disk as with instances. This was changed in <a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a> to support other redefinition and AOT mixing usages.</p>
<p><b>Approaches:</b></p>
<p>1) Compile in reverse dependency order to avoid compiling twice.</p>
<p>Either swap the order of compilation in the first example or specify the order in project.clj:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">:aot [dispatch.dispatch dispatch.core]</pre>
</div></div>
<p>This is a short-term workaround.</p>
<p>2) Move the deftype into a separate namespace from where it is used so it is not redefined on the second compile. This is another short-term workaround.</p>
<p>3) Do not put compile-path on the classpath (this violates current expectations, but avoids loading dispatch__init)</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(set! *compile-path* <span class="code-quote">"foo"</span>)
(compile 'dispatch.core)
(compile 'dispatch.dispatch)</pre>
</div></div>
<p>This is not easy to set up via Leiningen currently.</p>
<p>4) Compile each file with an independent Clojure runtime - avoids using cached classes in DCL for class literals.</p>
<p>Probably too annoying to actually do right now in Leiningen or otherwise.</p>
<p>5) Make compilation non-transitive. This is in the ballpark of <a href="http://dev.clojure.org/jira/browse/CLJ-322" title="Enhance AOT compilation process to emit classfiles only for explicitly-specified namespaces">CLJ-322</a>, which is another can of worms. Also possibly where we should be headed though.</p>
<p><b>Screening:</b> I do not believe the proposed patch is a good idea - it papers over the symptom without addressing the root cause. I think we need to re-evaluate how compilation works with regard to compile-path (#3) and transitivity (<a href="http://dev.clojure.org/jira/browse/CLJ-322" title="Enhance AOT compilation process to emit classfiles only for explicitly-specified namespaces">CLJ-322</a>) (#5), but I think we should do this after 1.7. - Alex</p>
<p>See also: <a href="http://dev.clojure.org/jira/browse/CLJ-1650" title="compile forces namespace reloading from AOT classfile"><del>CLJ-1650</del></a></p>CLJ-1741deftype class literals and instances loaded from different classloaders when recompiling namespaceDefectMajorOpenUnresolvedUnassignedStephen NelsonaotclassloadercompilerSat, 30 May 2015 20:38:20 -0500Thu, 18 Jun 2015 11:24:26 -0500Release 1.7Release 1.802<p>Pulling into 1.7 for consideration.</p><p>I've added a debug flag to my example that causes type instance hashcodes and their class-loaders to be printed.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">Compiling dispatch.core
deftype =&gt; 652433136 (clojure.lang.DynamicClassLoader@23c30a20)
defmethod =&gt; 652433136 (clojure.lang.DynamicClassLoader@23c30a20)
instance =&gt; 652433136 (clojure.lang.DynamicClassLoader@23c30a20)
dispatch: :pass
Compiling dispatch.dispatch
deftype =&gt; 652433136 (clojure.lang.DynamicClassLoader@23c30a20)
defmethod =&gt; 652433136 (clojure.lang.DynamicClassLoader@23c30a20)
instance =&gt; 760357227 (sun.misc.Launcher$AppClassLoader@42a57993)
dispatch: :fail</pre>
</div></div><p>The compiler has weird loading rules when using `compile` and both a clj file and a class file are present in the classpath.</p>
<p>This bug happens because RT.load will load the AOT class file rebinding the -&gt;Ctor to use the AOT deftype instance.</p>
<p>A fix for this would be making load "loaded libs" aware to avoid unnecessary/harmful reloadings.</p><p>The attached patch fixes this bug by keeping track of what has already been loaded and loading the AOT class only if necessary</p><p>Original description (since replaced):</p>
<p>Type-dispatching multimethods are defined using the wrong type instance</p>
<p>When using a multimethod that dispatches on types, such as print-dup/print-method, the type reference passed to <tt>addMethod</tt> in the presence of aot is incorrect on the second load of the namespace. This means that if the namespace has already been loaded as a dependency of another file, the second load when the namespace is loaded for aot compilation will produce a multimethod that fails to dispatch correctly.</p>
<p>I've created an example repository:<br/>
<a href="https://github.com/sfnelson/clj-mm-dispatch">https://github.com/sfnelson/clj-mm-dispatch</a></p>
<p>To reproduce independently, create a namespace that contains a deftype and a multimethod dispatching on the type, and a second namespace that requires the first and sorts alphabetically before the first. Aot-compile both namespaces. When the type-defining namespace is loaded via <tt>require</tt> it produces a class file for the deftype. When it is loaded the second time for aot-compilation, the type corresponding to the existing class file is given to the defmethod, instead of the new class constructed by loading the namespace. This causes the multimethod it fail to dispatch correctly.</p>
<p>To me this issue seems similar to <a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>: the type passed to the multimethod is retrieved using the wrong classloader. This suggests that it might have wider implications than AOT and multimethod dispatch.</p><p>I just realized this ticket is a duplicate of <a href="http://dev.clojure.org/jira/browse/CLJ-1650" title="compile forces namespace reloading from AOT classfile"><del>CLJ-1650</del></a></p>ApprovalIncompleteGlobal RankPatchCode[CLJ-1620] Constants are leaked in case of a reentrant evalhttp://dev.clojure.org/jira/browse/CLJ-1620
Clojure<p>Compiling a function that references a non loaded (or uninitialized) class triggers its init static. When the init static loads clojure code, some constants (source code I think) are leaked into the constants pool of the function under compilation.</p>
<p>It prevented CCW from working in some environments (Rational) because the static init of the resulting function was over 64K.</p>
<h4><a name="Stepstoreproduce%3A"></a>Steps to reproduce:</h4>
<p>Load the leak.main ns and run the code in comments: the first function has 15 extra fields despite being identical to the second one.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns leak.main)
(defn first-to-load []
leak.Klass/foo)
(defn second-to-load []
leak.Klass/foo)
(comment
=&gt; (map (comp count #(.getFields %) class) [first-to-load second-to-load])
(16 1)
)</pre>
</div></div>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java"><span class="code-keyword">package</span> leak;
<span class="code-keyword">import</span> clojure.lang.IFn;
<span class="code-keyword">import</span> clojure.lang.RT;
<span class="code-keyword">import</span> clojure.lang.Symbol;
<span class="code-keyword">public</span> class Klass {
<span class="code-keyword">static</span> {
RT.<span class="code-keyword">var</span>(<span class="code-quote">"clojure.core"</span>, <span class="code-quote">"require"</span>).invoke(Symbol.intern(<span class="code-quote">"leak.leaky"</span>));
}
<span class="code-keyword">public</span> <span class="code-keyword">static</span> IFn foo = RT.<span class="code-keyword">var</span>(<span class="code-quote">"leak.leaky"</span>, <span class="code-quote">"foo"</span>);
}</pre>
</div></div>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns leak.leaky)
(defn foo
<span class="code-quote">"Some doc"</span>
[]
<span class="code-quote">"hello"</span>)
(def unrelated 42)</pre>
</div></div>
<p><a href="https://gist.github.com/cgrand/5dcb6fe5b269aecc6a5b#file-main-clj-L10">https://gist.github.com/cgrand/5dcb6fe5b269aecc6a5b#file-main-clj-L10</a></p>
<p><b>Patch</b>: clj-1620-v5.patch</p>CLJ-1620Constants are leaked in case of a reentrant evalDefectCriticalOpenUnresolvedUnassignedChristophe GrandaotcompilerThu, 18 Dec 2014 15:50:47 -0600Tue, 19 May 2015 08:27:37 -0500Release 1.7Release 1.834<p>Patch from Nicola Mometto</p><p>Attached the same patch with a more informative better commit message</p><p>I'd like to thank Christophe and Alex for their invaluable help in understanding what was happening, formulating the right hypothesis and then finding a fix.</p>
<p>I would also mention that even if non IBM rational environments where not affected by the bug to the point were CCW would not work, they were still affected. For instance the class for a one-liner function wrapping an interop call weighs 700bytes once the patch is applied, when it weighed 90kbytes with current 1.6 or 1.7.</p><p>In CCW for the initial problematic function, the -v2 patch produces exactly the same bytecode as if the referenced class does not load any namespace in its static initializers.<br/>
That is, the patch is valid. I will test it live in the IBM Rational environment ASAP.</p><p>I confirm the patch fixes the issue detected initially in the IBM Rational environment</p><p>I have absolutely no idea why, but if I apply this patch, and the patch for <a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a> to master, and then try to build a war from this test project <a href="https://github.com/pdenhaan/extend-test">https://github.com/pdenhaan/extend-test</a> I get a scary-looking traceback:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">$ lein <span class="code-keyword">do</span> clean, war!
Exception in thread <span class="code-quote">"main"</span> java.lang.NoSuchFieldError: __thunk__0__, compiling:(route.clj:1:1)
at clojure.lang.<span class="code-object">Compiler</span>$InvokeExpr.eval(<span class="code-object">Compiler</span>.java:3606)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7299)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7289)
at clojure.lang.<span class="code-object">Compiler</span>.compile(<span class="code-object">Compiler</span>.java:7365)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
at clojure.core$load_lib.doInvoke(core.clj:5667)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5706)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$require.doInvoke(core.clj:5789)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at extend_test.core.handler$loading__5301__auto____66.invoke(handler.clj:1)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.<span class="code-object">Compiler</span>$InvokeExpr.eval(<span class="code-object">Compiler</span>.java:3601)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7299)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7289)
at clojure.lang.<span class="code-object">Compiler</span>.compile(<span class="code-object">Compiler</span>.java:7365)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
at clojure.core$load_lib.doInvoke(core.clj:5667)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5706)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$require.doInvoke(core.clj:5789)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at extend_test.core.servlet$loading__5301__auto____7.invoke(servlet.clj:1)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.<span class="code-object">Compiler</span>$InvokeExpr.eval(<span class="code-object">Compiler</span>.java:3601)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7299)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7289)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7289)
at clojure.lang.<span class="code-object">Compiler</span>.compile(<span class="code-object">Compiler</span>.java:7365)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$compile$fn__5420.invoke(core.clj:5834)
at clojure.core$compile.invoke(core.clj:5833)
at user$eval5.invoke(form-init180441230737245034.clj:1)
at clojure.lang.<span class="code-object">Compiler</span>.eval(<span class="code-object">Compiler</span>.java:6776)
at clojure.lang.<span class="code-object">Compiler</span>.eval(<span class="code-object">Compiler</span>.java:6765)
at clojure.lang.<span class="code-object">Compiler</span>.eval(<span class="code-object">Compiler</span>.java:6766)
at clojure.lang.<span class="code-object">Compiler</span>.load(<span class="code-object">Compiler</span>.java:7203)
at clojure.lang.<span class="code-object">Compiler</span>.loadFile(<span class="code-object">Compiler</span>.java:7159)
at clojure.main$load_script.invoke(main.clj:274)
at clojure.main$init_opt.invoke(main.clj:279)
at clojure.main$initialize.invoke(main.clj:307)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.doInvoke(main.clj:420)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.NoSuchFieldError: __thunk__0__
at instaparse.core__init.load(Unknown Source)
at instaparse.core__init.&lt;clinit&gt;(Unknown Source)
at java.lang.<span class="code-object">Class</span>.forName0(Native Method)
at java.lang.<span class="code-object">Class</span>.forName(<span class="code-object">Class</span>.java:344)
at clojure.lang.RT.loadClassForName(RT.java:2141)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
at clojure.core$load_lib.doInvoke(core.clj:5667)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5706)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$require.doInvoke(core.clj:5789)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at clout.core$loading__5301__auto____273.invoke(core.clj:1)
at clout.core__init.load(Unknown Source)
at clout.core__init.&lt;clinit&gt;(Unknown Source)
at java.lang.<span class="code-object">Class</span>.forName0(Native Method)
at java.lang.<span class="code-object">Class</span>.forName(<span class="code-object">Class</span>.java:344)
at clojure.lang.RT.loadClassForName(RT.java:2141)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
at clojure.core$load_lib.doInvoke(core.clj:5667)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5706)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$require.doInvoke(core.clj:5789)
at clojure.lang.RestFn.invoke(RestFn.java:482)
at compojure.core$loading__5301__auto____68.invoke(core.clj:1)
at compojure.core__init.load(Unknown Source)
at compojure.core__init.&lt;clinit&gt;(Unknown Source)
at java.lang.<span class="code-object">Class</span>.forName0(Native Method)
at java.lang.<span class="code-object">Class</span>.forName(<span class="code-object">Class</span>.java:344)
at clojure.lang.RT.loadClassForName(RT.java:2141)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5415.invoke(core.clj:5823)
at clojure.core$load.doInvoke(core.clj:5822)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5362.invoke(core.clj:5668)
at clojure.core$load_lib.doInvoke(core.clj:5667)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5706)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$require.doInvoke(core.clj:5789)
at clojure.lang.RestFn.invoke(RestFn.java:457)
at compojure.route$loading__5301__auto____1508.invoke(route.clj:1)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.<span class="code-object">Compiler</span>$InvokeExpr.eval(<span class="code-object">Compiler</span>.java:3601)
... 75 more
Subprocess failed</pre>
</div></div><p><a href="https://github.com/MichaelBlume/clojure/tree/no-field">https://github.com/MichaelBlume/clojure/tree/no-field</a><br/>
<a href="https://github.com/MichaelBlume/extend-test/tree/no-field">https://github.com/MichaelBlume/extend-test/tree/no-field</a></p>
<p>mvn clean install in the one, lein ring uberwar in the other.</p><p>Michael, thanks for the report, I've tried investigating this a bit but the big amount of moving parts involved make it really hard to figure out why the combination of the two patches causes this issue.</p>
<p>A helpful minimal case would require no lein and no external dependencies, I'd appreciate some help in debugging this issue if anybody has time.</p><p>Ok, looks like the minimal case is</p>
<p>(ns foo (:require <span class="error">&#91;instaparse.core&#93;</span>))</p>
<p>(ns bar (:require <span class="error">&#91;foo&#93;</span>))</p>
<p>and then attempt to AOT-compile both foo and bar.</p>
<p>I don't yet know what's special about instaparse.core.</p><p>Well, not a minimal case, of course, but one without lein, at least.</p><p>ok, problem is instaparse's defclone macro, I've extracted it to a test repo</p>
<p><a href="https://github.com/MichaelBlume/thunk-fail">https://github.com/MichaelBlume/thunk-fail</a></p>
<p>lein do clean, compile will get you a failure, but the repo has no dependencies so I'm sure there's a way to do that without lein.</p><p>Sorry for the barrage of questions, but these classloader bugs are subtle (and close to being solved I hope). Your report is immensely valuable, and yet it will help to be even more specific. There are a cluster of these bugs &#8211; and keeping them laser-focused is key.</p>
<p>The minimal case to which you refer is the NoSuchFieldError?<br/>
How are is this being invoked this without lein?<br/>
What are you calling to AOT? (compile 'bar) ? <br/>
What is the classpath? When you invoke originally, is ./target/classes empty?<br/>
Does the problem go away with <a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>-7 applied?</p><p>I have tried and failed to replicate without leiningen. When I just run </p>
<p>java -Dclojure.compile.path=target -cp src:../clojure/target/clojure-1.7.0-aot-SNAPSHOT.jar clojure.lang.Compile thunk-fail.first thunk-fail.second</p>
<p>everything works fine.</p><p>The NoSuchFieldError is related to the keyword lookup sites.</p>
<p>Replacing defclone's body with<br/>
`(do (:foo {})) is enough to trigger it, with the same ns structure.</p><p>I have updated the patch for <a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a>, now the combination of the new patch + the patch from this ticket should not cause any exception.</p>
<p>That said, a bug in this patch still exists since while the patch for <a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a> had a bug, it was causing a perfectly valid (albeit hardly reproducible) compilation scenario so we should keep debugging this patch with the help of the bugged patch for <a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a>.</p>
<p>I guess the first thing to do is figure out what lein compile is doing differently than clojure.Compile</p><p>Also Ghadi is right, infact replacing the whole body of thunk-fail.core with (:foo {}) is enough.</p>
<p>It would seem like the issue is with AOT (re)compiling top-level keyword lookup sites, my guess is that for some reason this patch is preventing correct generation of the __init static initializer.</p><p>I still have absolutely no idea what lein compile is doing but I figured out the issue.<br/>
The updated patch binds (in eval) the appropriate vars only when already bounded.</p><p>Would it be worth using transients on the bindings map now?</p><p>Makes sense, updated the patch to use a transient map</p><p>Is there a test we can add that'll fail in the presence of the v2 patch? preferably independent of the <a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a> patch? I can try to write one myself, but I don't have a lot of familiarity with the Clojure compiler internals.</p><p>I'll have to think about a way to reproduce that bug, it's not a simple scenario to reproduce.<br/>
It involves compiling a namespace from an evaluated context.</p><p>Hello, is there any chance left that this issue will make it to 1.7 ?</p><p>Wasn't planning on it - what's the impact for you?</p><p>The impact is that I need to use a patched version of Clojure for CCW.<br/>
While it's currently not that hard to follow clojure's main branch and regularly rebase on it or reapply the patch, it's still a waste of time.</p><p>I will check with Rich whether it can be screened for 1.7 before we get to RC.</p>
<p>same as v4 patch, but just has more diff context</p><p>the file mentioned in the patch field is not the right one IMHO</p><p>which one is?</p><p>I think you previous comment relates to clj-1620-v5.patch, but at the end of the description there's the following line:</p>
<p>Patch: 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1620" title="Constants are leaked in case of a reentrant eval">CLJ-1620</a>-avoid-constants-leak-in-static-initalizer-v4.patch</p><p>Those patches are equivalent with respect to the change they introduce; they just differ in how much diff context they have.</p><p>Rich has ok'ed screening this one for 1.7 but I do not feel that I can mark it screened without understanding it much better than I do. The description, code, and cause information here is not sufficient for me to understand what the problem actually is or why the fix is the right one. The fix seems to address the symptom but I worry that it is just a symptom and that a better understanding of the actual cause would lead to a different or better fix.</p>
<p>The evolution of the patches was driven by bugs in <a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a> (a patch which has been pulled out for being suspect for other reasons). Starting fresh, were those modifications necessary and correct? </p>
<p>Why does <b>this</b> set of vars need to push clean impls into the bindings? Why not some of the other vars (like those pushed in load())? The set chosen here seems to match that from the ReifyParser - why? Why should they only be pushed if they are bound (that is, why is "not bound" not the same as "bound but empty")? Are we affecting performance?</p>
<p>Popping all the way out, is the thing being done by CCW even a thing that should be doable? The description says "Compiling a function that references a non loaded (or uninitialized) class triggers its init static" - should this load even happen? Can we get an example that actually demonstrates what CCW was doing originally? </p><p>Alex, the question of "should what CCW is doing be doable" can be answered if you answer it on the given example, I think.</p>
<p>The question "should the initialization of the class occur when it could just be loaded" is a good one. Several reports have been made on the Clojure list about this problem, and I guess there is at least one CLJ issue about changing some more classForName into classForNameNonLoading here and there in Clojure.<br/>
For instance, it prevents referencing java classes which have code in their static initializers as soon as the code does some supposition about the runtime it is initialized in. This is a problem with Eclipse / SWT, this a problem with Cursive as I remember Colin mentioning a similar issue. And will probably is a problem that can appear each time one tries to AOT compile clojure code interoperating with java classes who happen to have, somewhere within static initializers triggered by the compilation (and this is transitive), assumptions that they are initialized in the proper target runtime environment.</p>
<p>What I don't know is if preventing the initialization to occur in the first place would be sufficient to get rid of the class of problems this bug and the proposed patch tried to solve. I do not claim to totally what is happening either (Christophe and Nicolas were of great help to analyze the issue and create the patch), but as I understand it, it's a kind of "Inception-the-movie-like" bug. Compiling a fn which triggers compiling another fn (here through the loading of clojure namespaces via a java initializer).</p>
<p>If preventing the initialization of class static methods when they are referenced (through interop calls - constructor, field, method, static field, static method-) is the last remaining bit that could cause such "compilation during compilation" scenario, then yes, protecting the compilation process like Nicolas tried to do may not be necessary, and just fixing the undesired loading may be enough.</p>
ApprovalIncompleteGlobal RankPatchCode[CLJ-1604] AOT'ed code that defs a var with clojure.core symbol name causes IllegalStateExceptionhttp://dev.clojure.org/jira/browse/CLJ-1604
Clojure<p>AOT'ed code that defs a var that is also a symbol in clojure.core results in an exception at runtime. This problem can be avoided with (:refer-clojure :exclude ...) but this requires a library author to update and release a new version. AOT'ed applications must then wait for all transitive dependencies to update before they can update to a new Clojure version. For some users, this problem prevents them from trying or adopting new releases.</p>
<p>For example, the contrib library <tt>data.int-map</tt> defines an <tt>update</tt> function. clojure.core will also have a new <tt>update</tt> function as of 1.7.0. If this library is AOT'ed, then users of the clojure.data.int-map/update function will see the exception below. This situation can commonly occur when an application uses <tt>lein uberjar</tt> to compile all of the project+libs. In this case, applications or libraries that use data.int-map (either directly or indirectly) are affected.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">java.lang.IllegalStateException: Attempting to call unbound fn: #'clojure.data.<span class="code-object">int</span>-map/update
at clojure.lang.Var$Unbound.throwArity (Var.java:43)
clojure.lang.AFn.invoke (AFn.java:40)
compiler_update_not_referenced_bug.core$foo.invoke (core.clj:5)</pre>
</div></div>
<p>Reproduce with this sample project: <a href="https://github.com/yeller/compiler_update_not_referenced_bug">https://github.com/yeller/compiler_update_not_referenced_bug</a></p>
<p><b>Cause:</b> When AOT compiling a namespace, the def forms are hoisted into the ns_&#95;init class (in the example here, clojure.data.int_map__init). The static initializer in this class creates each var in the ns via a call to RT.var(ns, name). For data.int-map the static initializer will properly create the var for clojure.data.int-map/update. But when the ns is loaded (via the clojure.data.int_map.load() method), (refer-clojure) will be called, which will remap clojure.data.int-map/update to point to clojure.core/update.</p>
<p>This problem does not affect non-AOT loading (which doesn't use the ns_&#95;init class) and does not affect collisions from any other namespace. Only collisions from clojure.core create this possibility.</p>
<p><b>Proposed:</b> The proposed patch explicitly refers the Var during ns_&#95;init.load() (after Clojure symbols are referred) rather than implicitly during ns__init static {}. </p>
<p>This change in behavior only happens during AOT in the specific case where a core symbol is being shadowed. In that case, clojure.core has already been loaded and v (the looked up var) will have ns=clojure.core. The currentNS will be (for example) data.int-map. If that's the case, and the sym has no ns, then the new logic will be emitted. </p>
<p>In the case of clojure.core itself, NO new bytecode is emitted. From tests on several projects, only shadowed vars during AOT get this additional bytecode.</p>
<p><b>Patch:</b> 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu-v2.patch</p>
<p><b>Screened by:</b> Alex Miller</p>CLJ-1604AOT'ed code that defs a var with clojure.core symbol name causes IllegalStateExceptionDefectCriticalClosedCompletedUnassignedNicola MomettoaotcompilerTue, 25 Nov 2014 15:42:04 -0600Sat, 21 Feb 2015 12:28:32 -0600Fri, 20 Feb 2015 11:42:50 -0600Release 1.7Release 1.734<p>When I try latest Clojure master plus patch <a href="http://dev.clojure.org/jira/browse/CLJ-1604" title="AOT&#39;ed code that defs a var with clojure.core symbol name causes IllegalStateException"><del>CLJ-1604</del></a>-only-core.patch with the small test project created by Tom Crayford to demonstrate this issue: <a href="https://github.com/yeller/compiler_update_not_referenced_bug">https://github.com/yeller/compiler_update_not_referenced_bug</a></p>
<p>In that project, I get the same exception thrown when attempting 'lein do clean, uberjar, test' using this patch, as without it. It is because int-map/update in namespace compiler-update-not-referenced-bug.core is an unbound var.</p><p>Andy, you're right. For some reason I attached the wrong patch to the ticket, this is the correct one</p><p>I wasn't able to write a test for this, so here's a repl session using the clojure jar demonstrating this issue:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">[˷/test]&gt; ls
classes clojure.jar test.clj
[˷/test]&gt; cat test.clj
(in-ns 'test)
(clojure.core/refer 'clojure.core)
(def foo <span class="code-quote">"bar"</span>)
(def update <span class="code-quote">"foo"</span>)
[˷/test]&gt; java -cp classes:clojure.jar:. clojure.main
Clojure 1.7.0-master-SNAPSHOT
user=&gt; (binding [*compile-files* <span class="code-keyword">true</span>] (load <span class="code-quote">"test"</span>))
WARNING: update already refers to: #'clojure.core/update in namespace: test, being replaced by: #'test/update
nil
user=&gt; test/foo
<span class="code-quote">"bar"</span>
user=&gt; test/update
<span class="code-quote">"foo"</span>
user=&gt;
[˷/test]&gt; java -cp classes:clojure.jar:. clojure.main
Clojure 1.7.0-master-SNAPSHOT
user=&gt; (load <span class="code-quote">"test"</span>)
nil
user=&gt; test/foo
<span class="code-quote">"bar"</span>
user=&gt; test/update
CompilerException java.lang.RuntimeException: No such <span class="code-keyword">var</span>: test/update, compiling: (NO_SOURCE_PATH:0:0)
user=&gt;</pre>
</div></div><p>Thanks. I have not tried to assess the details of the change, other than to say that patch 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu.patch dated 26 Nov 2014, when applied to latest Clojure master as of today, enables both 'lein do clean, test' and 'lein do clean, uberjar, test' to work as expected with Tom Crayford's test project, linked above, whereas 'lein do clean, uberjar, test' fails without this patch, due to a var being unbound that should have a value.</p><p>Copying a comment here from <a href="http://dev.clojure.org/jira/browse/CLJ-1591" title="Symbol not being bound in namespace when name clashes with clojure.core">CLJ-1591</a>, since it is more appropriate here. It is responding to Tom Crayford's posting of his example project to demonstrate the issue: <a href="https://github.com/yeller/compiler_update_not_referenced_bug">https://github.com/yeller/compiler_update_not_referenced_bug</a></p>
<p>Tom, looked at your project. Thanks for that. It appears not to have anything like (def inc inc) in it. It throws exception during test step of 'lein do clean, uberjar, test' consistently for me, too, but compiles with only warnings and passes tests with 'lein do clean, test'. I have more test results showing in which Clojure versions these results change. To summarize, the changes to Clojure that appear to make the biggest difference in the results are below (these should be added to the new ticket you create – you are welcome to do so):</p>
<p>Clojure 1.6.0, 1.7.0-alpha1, and later changes up through the commit with description "<a href="http://dev.clojure.org/jira/browse/CLJ-1378" title="Hints don&#39;t work with #() form of function"><del>CLJ-1378</del></a>: Allows FnExpr to override its reported class with a type hint": No errors or warnings for either lein command above.</p>
<p>Next commit with description "Add clojure.core/update, like update-in but takes a single key" that adds clojure.core/update: 'lein do clean, test' is fine, but 'lein do clean, uberjar' throws exception during compilation, probably due to <a href="http://dev.clojure.org/jira/browse/CLJ-1241" title="NPE when AOTing overrided clojure.core functions"><del>CLJ-1241</del></a>.</p>
<p>Next commit with description "fix <a href="http://dev.clojure.org/jira/browse/CLJ-1241" title="NPE when AOTing overrided clojure.core functions"><del>CLJ-1241</del></a>": 'lein do clean, test' and 'lein do clean, uberjar' give warnings about clojure.core/update, but no errors or exceptions. 'lein do clean, uberjar, test' throws exception during test step that is same as the one I see with Clojure 1.7.0-alpha4. Debug prints of values of clojure.core/update and int-map/update (in data.int-map and in Tom's namespace compiler-update-not-referenced-bug.core) show things look fine when printed inside data.int-map, and in Tom's namespace when not doing the uberjar, but when doing the uberjar, test, int-map/update is unbound in Tom's namespace.</p>
<p>In case it makes a difference, my testing was done with Mac OS X 10.9.5, Leiningen 2.5.0 on Java 1.7.0_45 Java HotSpot(TM) 64-Bit Server VM</p><p>The updated patch only emits the interning bytecode when necessary, avoiding the emission when a clojure.core var with the same name exists but is not mapped to the current namespace</p><p>Attached 1604-context.diff for purely informational purposes - same diff just more context in it for easier reading.</p><p>Thought I'd add a minor note in here to say I tried testing this patch out on my app (which is where I discovered this AOT bug), and the bug doesn't turn up with this patch applied to clojure (tested by applying 0001-fix-AOT-bug-preventing-overriding-of-clojure.core-fu-v2.patch to 1.7-alpha5)</p><p>I ran into this issue with Korma 0.4.0. I'm still running into it, but there is a twist.</p>
<p>My project depends on an artifact that was built with Clojure 1.7.0-alpha1. If I remove this dependency, everything is fine. However, with this dependency, I run into this issue, even if I declare a dependency on 1.7.0-master-SNAPSHOT in my project and exclude any dependency on clojure-1.7.0-alpha1.</p>
<p>I'm not sure if this is a Maven issue or a Clojure issue. Running Maven with debug on seems to show that it's using the correct version of Clojure.</p>
<p>I have created a dummy project that reproduces this issue if you are interested.</p>
<p><a href="https://github.com/deaddowney/UpdateProblem">https://github.com/deaddowney/UpdateProblem</a></p>
<p>Check it out, run "mvn install", and you will get <br/>
java.lang.RuntimeException: No such var: korma.core/update<br/>
.</p>
ApprovalOkGlobal RankPatchCode[CLJ-1544] AOT bug involving namespaces loaded before AOT compilation startedhttp://dev.clojure.org/jira/browse/CLJ-1544
Clojure<p>If namespace "a" that is being AOT compiled requires a namespace "b" that has been loaded but not AOT compiled, the classfile for that namespace will never be emitted on disk, causing errors when compiling uberjars or in other cases.</p>
<p>A minimal reproducible case is described in the following comment: <a href="http://dev.clojure.org/jira/browse/CLJ-1544?focusedCommentId=36734&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-36734">http://dev.clojure.org/jira/browse/CLJ-1544?focusedCommentId=36734&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-36734</a></p>
<p>Other examples of the bug:<br/>
<a href="https://github.com/arohner/clj-aot-repro">https://github.com/arohner/clj-aot-repro</a><br/>
<a href="https://github.com/methylene/class-not-found">https://github.com/methylene/class-not-found</a></p>
<p>A real issue triggered by this bug: <a href="https://github.com/cemerick/austin/issues/23">https://github.com/cemerick/austin/issues/23</a></p>
<p>Related ticket: <a href="http://dev.clojure.org/jira/browse/CLJ-1641" title="AOT compilation fails for a kind of circular dependency after CLJ-1544 patch"><del>CLJ-1641</del></a> contains descriptions and comments about some potentially unwanted consequences of applying proposed patch 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a>-force-reloading-of-namespaces-during-AOT-co-v3.patch</p>
<p><b>Approach:</b> The approach taken by the attached patch is to force reloading of namespaces during AOT compilation if no matching classfile is found in the compile-path or in the classpath</p>
<p><b>Patch:</b> 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1544" title="AOT bug involving namespaces loaded before AOT compilation started">CLJ-1544</a>-force-reloading-of-namespaces-during-AOT-co-v3.patch</p>
<p><b>Screened by:</b> Alex Miller</p>CLJ-1544AOT bug involving namespaces loaded before AOT compilation startedDefectCriticalReopenedUnresolvedUnassignedAllen RohneraotWed, 1 Oct 2014 13:24:36 -0500Fri, 20 Feb 2015 10:11:38 -0600Release 1.889<p>Possibly related: <a href="http://dev.clojure.org/jira/browse/CLJ-1457" title="once the compiler pops the dynamic classloader from the stack, attempts to read record reader literals will fail"><del>CLJ-1457</del></a></p><p>Has anyone been able to reproduce this bug from a bare clojure repl? I have been trying to take lein out of the equation for an hour but I don't seem to be able to reproduce it &#8211; this makes me think that it's possible that this is a lein/classlojure/nrepl issue rather than a compiler/classloader bug</p><p>I was actually able to reproduce and understand this bug thanks to a minimal example reduced from a testcase for <a href="http://dev.clojure.org/jira/browse/CLJ-1413" title="A problem involving user.clj and defrecord"><del>CLJ-1413</del></a>.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">&gt;cat error.sh
#!/bin/sh
rm -rf target &amp;&amp; mkdir target
java -cp src:clojure.jar clojure.main - &lt;&lt;EOF
(require 'myrecord)
(set! *compile-path* <span class="code-quote">"target"</span>)
(compile 'core)
EOF
java -cp target:clojure.jar clojure.main -e <span class="code-quote">"(use 'core)"</span>
&gt; cat src/core.clj
(in-ns 'core)
(clojure.core/require 'myrecord)
(clojure.core/<span class="code-keyword">import</span> myrecord.somerecord)
&gt;cat src/myrecord.clj
(in-ns 'myrecord)
(clojure.core/defrecord somerecord [])
&gt; ./error.sh
Exception in thread <span class="code-quote">"main"</span> java.lang.ExceptionInInitializerError
at java.lang.<span class="code-object">Class</span>.forName0(Native Method)
at java.lang.<span class="code-object">Class</span>.forName(<span class="code-object">Class</span>.java:344)
at clojure.lang.RT.classForName(RT.java:2113)
at clojure.lang.RT.classForName(RT.java:2122)
at clojure.lang.RT.loadClassForName(RT.java:2141)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5403.invoke(core.clj:5808)
at clojure.core$load.doInvoke(core.clj:5807)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5352.invoke(core.clj:5653)
at clojure.core$load_lib.doInvoke(core.clj:5652)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5691)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:630)
at clojure.core$use.doInvoke(core.clj:5785)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at user$eval212.invoke(NO_SOURCE_FILE:1)
at clojure.lang.<span class="code-object">Compiler</span>.eval(<span class="code-object">Compiler</span>.java:6767)
at clojure.lang.<span class="code-object">Compiler</span>.eval(<span class="code-object">Compiler</span>.java:6730)
at clojure.core$eval.invoke(core.clj:3076)
at clojure.main$eval_opt.invoke(main.clj:288)
at clojure.main$initialize.invoke(main.clj:307)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.doInvoke(main.clj:420)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.io.FileNotFoundException: Could not locate myrecord__init.class or myrecord.clj on classpath.
at clojure.lang.RT.load(RT.java:443)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5403.invoke(core.clj:5808)
at clojure.core$load.doInvoke(core.clj:5807)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5613)
at clojure.core$load_lib$fn__5352.invoke(core.clj:5653)
at clojure.core$load_lib.doInvoke(core.clj:5652)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$load_libs.doInvoke(core.clj:5691)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:628)
at clojure.core$require.doInvoke(core.clj:5774)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at core__init.load(Unknown Source)
at core__init.&lt;clinit&gt;(Unknown Source)
... 33 more</pre>
</div></div>
<p>This bug also has also affected Austin: <a href="https://github.com/cemerick/austin/issues/23">https://github.com/cemerick/austin/issues/23</a></p>
<p>Essentially this bug manifests itself when a namespace defining a protocol or a type/record has been JIT loaded and a namespace that needs the protocol/type/record class is being AOT compiled later. Since the namespace defining the class has already been loaded the class is never emitted on disk.</p><p>I've attached a tentative patch fixing the issue in the only way I found reasonable: forcing the reloading of namespaces during AOT compilation if the compiled classfile is not found in the compile-path or in the classpath</p><p>Updated patch forces reloading of the namespace even if a classfile exists in the compile-path but the source file is newer, mimicking the logic of clojure.lang.RT/load</p><p>Further testing demonstrated that this bug is not only scoped to deftypes/defprotocols but can manifest itself in the general case of a namespace "a" requiring a namespace "b" already loaded, and AOT compiling the namespace "a"</p><p>I'm also affected by this bug. Is there some workaround I can apply in the meantime, e.g., by dictating the order in which namespaces are going to be loaded/compiled in project.clj?</p><p>Tassilo, if you don't have control over whether or not a namespace that an AOT namespace depends on has already been loaded before compilation starts, requiring those namespaces with :reload-all should be enough to work around this issue</p><p>Nicola, thanks! But in the meantime I've switched to using clojure.java.api and omit AOT-compilation. That works just fine, too.</p><p>Tassilo, that's often a good solution, another is to use a shim clojure class</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(ns myproject.main-shim (:gen-class))
(defn -main [&amp; args]
(require 'myproject.main)
((resolve 'myproject.main) args))</pre>
</div></div>
<p>then your shim namespace is AOT-compiled but nothing else in your project is.</p><p>Thanks Michael, that's a very good suggestion. In fact, I've always used AOT only as a means to export some functions to Java-land. Basically, I did as you suggest but required the to-be-exported fn's namespace in the ns-form which then causes AOT-compilation of that namespace and its own deps recursively. So your approach seems to be as convenient from the Java side (no need to clojure.java.require `require` in order to require the namespace with the fn I wanna call <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/wink.gif" height="20" width="20" align="absmiddle" alt="" border="0"/>) while still omitting AOT. Awesome!</p><p>I'm marking this as incomplete to prevent further screening until the bug reported here: <a href="http://dev.clojure.org/jira/browse/CLJ-1620?focusedCommentId=37232&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-37232">http://dev.clojure.org/jira/browse/CLJ-1620?focusedCommentId=37232&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-37232</a> is figured out</p><p>Fixed the patch, I'm re marking the tickets as Vetted as it was before.</p><p>This patch is being rolled back for 1.7.0-alpha6 pending further investigation into underlying problems and possible solutions.</p><p>I'm not 100% sure, but this looks a lot like <a href="https://github.com/cursiveclojure/cursive/issues/369">Cursive issue 369</a>. It had a case that I could reproduce with JDK 7 but not JDK 8, has the same mysterious missing namespace class symptom, and involves mixed AOT/non-AOT namespaces. However it's happening at runtime, not at compile time, which doesn't seem consistent.</p><p>My error report above was incorrectly tied to this issue (see <a href="http://dev.clojure.org/jira/browse/CLJ-1636" title="SeqIterator can return incorrect results "><del>CLJ-1636</del></a>). I will delete the comment.</p><p>Since ticket <a href="http://dev.clojure.org/jira/browse/CLJ-1641" title="AOT compilation fails for a kind of circular dependency after CLJ-1544 patch"><del>CLJ-1641</del></a> has been closed, I'll repost here a comment I posted in that ticket + the patch I proposed, arguing why I think the patch I proposed for this ticket should not have been reverted:</p>
<p>Zach, I agree that having different behaviour between AOT and JIT is wrong.</p>
<p>But I also don't agree that having clojure error out on circular dependencies should be considered a bug, I would argue that the way manifold used to implement the circular dependency between manifold.stream and manifold.stream.graph was a just a hack around lack of validation in require.</p>
<p>My proposal to fix this disparity between AOT and JIT is by making require/use check for circular dependencies before checking for already-loaded namespaces.</p>
<p>This way, both under JIT and AOT code like</p>
<p>(ns foo.a (:require foo.b))<br/>
(ns foo.b)<br/>
(require 'foo.a)</p>
<p>will fail with a circular depdenency error.</p>
<p>This is what the patch I just attached (0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1641" title="AOT compilation fails for a kind of circular dependency after CLJ-1544 patch"><del>CLJ-1641</del></a><del>disallow-circular-dependencies-even-if-the</del>.patch) does.</p>ApprovalIncompleteGlobal RankPatchCode[CLJ-1392] AOT vs "var already exists warning" results in NPEhttp://dev.clojure.org/jira/browse/CLJ-1392
Clojure<p>I just saw this thread on the cascalog list: <a href="https://groups.google.com/forum/#!topic/cascalog-user/Pe5QIpmU0vA">https://groups.google.com/forum/#!topic/cascalog-user/Pe5QIpmU0vA</a></p>
<p>Apparently the "WARNING: some? already refers to: #'clojure.core/some? in namespace: jackknife.seq, being replaced by: #'jackknife.seq/some?" results in a NullPointerException when trying to AOT a namespace that results in the message being output.</p>
<p>Reproducer:</p>
<p>1) lein new aotFail<br/>
2) Edit project.clj <br/>
;add as appropriate<br/>
:aot :all<br/>
:dependencies [<span class="error">&#91;org.clojure/clojure &quot;1.6.0&quot;&#93;</span><br/>
<span class="error">&#91;cascalog &quot;2.0.0&quot;&#93;</span>]<br/>
2) Add "(:use cascalog.api)" to the ns block of src/aotFail/core.clj<br/>
3) lein compile</p>
<p>Output:</p>
<p>Compiling aotFail.core<br/>
WARNING: some? already refers to: #'clojure.core/some? in namespace: jackknife.seq, being replaced by: #'jackknife.seq/some?<br/>
Exception in thread "main" java.lang.NullPointerException, compiling:(api.clj:1:1)<br/>
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3558)<br/>
at clojure.lang.Compiler.compile1(Compiler.java:7226)<br/>
at clojure.lang.Compiler.compile1(Compiler.java:7216)<br/>
at clojure.lang.Compiler.compile(Compiler.java:7292)<br/>
at clojure.lang.RT.compile(RT.java:398)<br/>
at clojure.lang.RT.load(RT.java:438)<br/>
at clojure.lang.RT.load(RT.java:411)<br/>
at clojure.core$load$fn__5066.invoke(core.clj:5641)<br/>
at clojure.core$load.doInvoke(core.clj:5640)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:408)<br/>
at clojure.core$load_one.invoke(core.clj:5446)<br/>
at clojure.core$load_lib$fn__5015.invoke(core.clj:5486)<br/>
at clojure.core$load_lib.doInvoke(core.clj:5485)<br/>
at clojure.lang.RestFn.applyTo(RestFn.java:142)<br/>
at clojure.core$apply.invoke(core.clj:626)<br/>
at clojure.core$load_libs.doInvoke(core.clj:5528)<br/>
at clojure.lang.RestFn.applyTo(RestFn.java:137)<br/>
at clojure.core$apply.invoke(core.clj:628)<br/>
at clojure.core$use.doInvoke(core.clj:5618)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:408)<br/>
at cascalog2.core$loading_<em>4958</em><em>auto</em>_.invoke(core.clj:1)<br/>
at clojure.lang.AFn.applyToHelper(AFn.java:152)<br/>
at clojure.lang.AFn.applyTo(AFn.java:144)<br/>
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3553)<br/>
at clojure.lang.Compiler.compile1(Compiler.java:7226)<br/>
at clojure.lang.Compiler.compile1(Compiler.java:7216)<br/>
at clojure.lang.Compiler.compile(Compiler.java:7292)<br/>
at clojure.lang.RT.compile(RT.java:398)<br/>
at clojure.lang.RT.load(RT.java:438)<br/>
at clojure.lang.RT.load(RT.java:411)<br/>
at clojure.core$load$fn__5066.invoke(core.clj:5641)<br/>
at clojure.core$load.doInvoke(core.clj:5640)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:408)<br/>
at clojure.core$load_one.invoke(core.clj:5446)<br/>
at clojure.core$compile$fn__5071.invoke(core.clj:5652)<br/>
at clojure.core$compile.invoke(core.clj:5651)<br/>
at user$eval19.invoke(form-init2092370125048380878.clj:1)<br/>
at clojure.lang.Compiler.eval(Compiler.java:6703)<br/>
at clojure.lang.Compiler.eval(Compiler.java:6693)<br/>
at clojure.lang.Compiler.load(Compiler.java:7130)<br/>
at clojure.lang.Compiler.loadFile(Compiler.java:7086)<br/>
at clojure.main$load_script.invoke(main.clj:274)<br/>
at clojure.main$init_opt.invoke(main.clj:279)<br/>
at clojure.main$initialize.invoke(main.clj:307)<br/>
at clojure.main$null_opt.invoke(main.clj:342)<br/>
at clojure.main$main.doInvoke(main.clj:420)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:421)<br/>
at clojure.lang.Var.invoke(Var.java:383)<br/>
at clojure.lang.AFn.applyToHelper(AFn.java:156)<br/>
at clojure.lang.Var.applyTo(Var.java:700)<br/>
at clojure.main.main(main.java:37)<br/>
Caused by: java.lang.NullPointerException<br/>
at clojure.lang.Compiler$ObjExpr.emitVar(Compiler.java:4944)<br/>
at clojure.lang.Compiler$DefExpr.emit(Compiler.java:437)<br/>
at clojure.lang.Compiler.compile1(Compiler.java:7225)<br/>
at clojure.lang.Compiler.compile(Compiler.java:7292)<br/>
at clojure.lang.RT.compile(RT.java:398)<br/>
at clojure.lang.RT.load(RT.java:438)<br/>
at clojure.lang.RT.load(RT.java:411)<br/>
at clojure.core$load$fn__5066.invoke(core.clj:5641)<br/>
at clojure.core$load.doInvoke(core.clj:5640)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:408)<br/>
at clojure.core$load_one.invoke(core.clj:5446)<br/>
at clojure.core$load_lib$fn__5015.invoke(core.clj:5486)<br/>
at clojure.core$load_lib.doInvoke(core.clj:5485)<br/>
at clojure.lang.RestFn.applyTo(RestFn.java:142)<br/>
at clojure.core$apply.invoke(core.clj:626)<br/>
at clojure.core$load_libs.doInvoke(core.clj:5524)<br/>
at clojure.lang.RestFn.applyTo(RestFn.java:137)<br/>
at clojure.core$apply.invoke(core.clj:628)<br/>
at clojure.core$use.doInvoke(core.clj:5618)<br/>
at clojure.lang.RestFn.invoke(RestFn.java:408)<br/>
at cascalog.api$loading_<em>4958</em><em>auto</em>_.invoke(api.clj:1)<br/>
at clojure.lang.AFn.applyToHelper(AFn.java:152)<br/>
at clojure.lang.AFn.applyTo(AFn.java:144)<br/>
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3553)</p>CLJ-1392AOT vs "var already exists warning" results in NPEDefectMajorClosedDuplicateUnassignedAaron CohenaotcompilerWed, 26 Mar 2014 11:58:29 -0500Wed, 26 Mar 2014 12:42:56 -0500Wed, 26 Mar 2014 12:42:56 -0500Release 1.601<p>Duplicate of <a href="http://dev.clojure.org/jira/browse/CLJ-1241">http://dev.clojure.org/jira/browse/CLJ-1241</a></p>Global Rank[CLJ-1330] Class name clash between top-level functions and defn'ed oneshttp://dev.clojure.org/jira/browse/CLJ-1330
Clojure<p>Named anonymous <tt>fn</tt>'s are not guaranteed to have unique class names when AOT-compiled.</p>
<p>For example:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(defn g [])
(def xx (fn g []))</pre>
</div></div>
<p>When AOT-compiled both functions will emit <tt>user$g.class</tt>, the latter overwriting the former.</p>
<p><b>Impact:</b> this affects apps like Cursive, which has been using a patched version of Clojure to get around this issue for quite a while.</p>
<p><b>Demonstration script:</b> demo1.clj</p>
<p><b>Patch:</b> 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart"><del>CLJ-1093</del></a>-v3.patch (already applied, see below)</p>
<p><b>Approach:</b> Generate unique class names for named <tt>fn</tt>'s the same way as for unnamed anonymous <tt>fn</tt>'s. <br/>
The patch contains an additional enhancement to include the name of the local binding in the class name.</p>
<p>Comparison between pre and post patch naming scheme (N denotes unique number):</p>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'>code</th>
<th class='confluenceTh'>before</th>
<th class='confluenceTh'>after</th>
<th class='confluenceTh'>note</th>
</tr>
<tr>
<td class='confluenceTd'><font color="red">(defn a [])</font></td>
<td class='confluenceTd'>user$a</td>
<td class='confluenceTd'>user$a</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'><font color="red">(fn [])</font></td>
<td class='confluenceTd'>user$evalN$fn__N</td>
<td class='confluenceTd'>user$evalN$fn__N</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'><font color="red">(fn a [])</font></td>
<td class='confluenceTd'>user$evalN$a__N</td>
<td class='confluenceTd'>user$evaN$a__N</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'>(let <span class="error">&#91;a <font color="red">(fn [])</font>&#93;</span> a)</td>
<td class='confluenceTd'>user$evalN$a__N</td>
<td class='confluenceTd'>user$evalN$a__N</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'>(let <span class="error">&#91;a <font color="red">(fn x [])</font>&#93;</span> a)</td>
<td class='confluenceTd'>user$eval1N$x__N</td>
<td class='confluenceTd'>user$evalN$a_<em>x</em>_N</td>
<td class='confluenceTd'>IMPROVED - contains local binding name</td>
</tr>
<tr>
<td class='confluenceTd'>(def a <font color="red">(fn [])</font>)</td>
<td class='confluenceTd'>user$a</td>
<td class='confluenceTd'>user$a</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'>(def a <font color="red">(fn x [])</font>)</td>
<td class='confluenceTd'>user$x</td>
<td class='confluenceTd'>user$a_<em>x</em>_N</td>
<td class='confluenceTd'>FIXED conflict with <tt>(defn x [])</tt></td>
</tr>
<tr>
<td class='confluenceTd'>(def ^{:foo <font color="red">(fn [])</font>} a)</td>
<td class='confluenceTd'>user$fn__N</td>
<td class='confluenceTd'>user$fn__N</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'>(def ^{:foo <font color="red">(fn a [])</font>} a)</td>
<td class='confluenceTd'>user$a</td>
<td class='confluenceTd'>user$a__N</td>
<td class='confluenceTd'>FIXED conflict with <tt>(defn a [])</tt></td>
</tr>
<tr>
<td class='confluenceTd'>(def a (fn [] <font color="red">(fn [])</font>))</td>
<td class='confluenceTd'>user$a$fn__N</td>
<td class='confluenceTd'>user$a$fn__N</td>
<td class='confluenceTd'>same</td>
</tr>
<tr>
<td class='confluenceTd'>(def a (fn [] <font color="red">(fn x [])</font>))</td>
<td class='confluenceTd'>user$a$x__N</td>
<td class='confluenceTd'>user$a$x__N</td>
<td class='confluenceTd'>same</td>
</tr>
</tbody></table>
<p><b>See also:</b> This patch also fixes the issue reported in <a href="http://dev.clojure.org/jira/browse/CLJ-1227" title="Definline functions do not work as higher-order functions when AOT compiled"><del>CLJ-1227</del></a>.</p>
<p><b>Screened by:</b> Alex Miller - I am not sure whether the local binding name enhancement is worth doing. It improves debugging of which anonymous class you're talking about but has the downsides of increasing class name (and file name) length.</p>
<p><b>REOPENED Patch:</b> 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart"><del>CLJ-1093</del></a>-v3-no-locals-improv.patch<br/>
<b>REOPENED UPDATE:</b> The local improvement version of this patch increases class names (and thus .class names) and we've had several reports of exceeding file system limits (~143 chars) - see comments for details. Thus, we should rollback the prior patch (0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart"><del>CLJ-1093</del></a>&#45;v3.patch) and apply the version without this enhancement. The replacement patch (0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart"><del>CLJ-1093</del></a>-v3-no-locals-improv.patch) does not have the same effect on class name length.</p>CLJ-1330Class name clash between top-level functions and defn'ed onesDefectCriticalClosedCompletedUnassignedNicola MomettoaotcompilerWed, 22 Jan 2014 08:08:16 -0600Fri, 14 Nov 2014 10:08:16 -0600Fri, 14 Nov 2014 10:08:16 -0600Release 1.7811<p>This seems like the reason why jvm.tools.analyzer cannot analyze clojure.core. On analyzing a definline, there is an "attempted duplicate class definition" error.</p>
<p>This doesn't really matter, but I thought it may or may not be useful information to someone.</p><p>Attached a fix.</p>
<p>This also fixes AOT compiling of code like:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(def x (fn foo []))
(fn foo [])</pre>
</div></div><p>Cleaned up patch</p><p>It looks like the patch changes indentation of some of the code - can you fix that?</p><p>Updated patch without whitespace changes</p><p>Thanks, that's helpful.</p><p>There is consensus that this is a problem, however this is an area of the code with broad impacts as it deals with how classes are named. To that end, there is some work that needs to be done in understanding the impacts before we can consider it.</p>
<p>Some questions we would like to answer:</p>
<p>1) According to Rich, naming of (fn x []) function classes used to work in the manner of this patch - with generated names. Some code archaeology needs to be done on why that was changed and whether the change to the current behavior was addressing problems that we are likely to run into.</p>
<p>2) Are there issues with recursive functions? Are there impacts either in AOT or non-AOT use cases? Need some tests.</p>
<p>3) Are there issues with dynamic redefinition of functions? With the static naming scheme, redefinition causes a new class of the same name which can be picked up by reload of classes compiled to the old definition. With the dynamic naming scheme, redefinition will create a differently named class so old classes can never pick up a redefinition. Is this a problem? What are the impacts with and without AOT? Need some tests.</p><p>Looks like the current behaviour has been such since <a href="https://github.com/clojure/clojure/commit/4651e60808bb459355a3a5d0d649c4697c672e28">https://github.com/clojure/clojure/commit/4651e60808bb459355a3a5d0d649c4697c672e28</a></p>
<p>My guess is that Rich simply forgot to consider the (def y (fn x [] ..)) case.</p>
<p>Regarding 2 and 3, the dynamic naming scheme is no different than what happens for anonymous functions so I don't see how this could cause any issue.</p>
<p>Recursion on the fn arg is simply a call to .invoke on "this", it's classname unaware.</p>
<p>I can add some tests to test that <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(def y (fn x [] 1))</pre>
</div></div> and <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(fn x [] 2)</pre>
</div></div> compile to different classnames but other than that I don't see what should be tested.</p><p>incomplete pending the answers to Alex Miller's questions in the comments</p><p>I believe I already answered his questions, I'll try to be a bit more explicit:<br/>
I tracked the relevant commit from Rich which added the dynamic naming behaviour <a href="https://github.com/clojure/clojure/commit/4651e60808bb459355a3a5d0d649c4697c672e28#diff-f17f860d14163523f1e1308ece478ddbL3081">https://github.com/clojure/clojure/commit/4651e60808bb459355a3a5d0d649c4697c672e28#diff-f17f860d14163523f1e1308ece478ddbL3081</a> which clearly shows that this bug was present since then so.</p>
<p>Regarding redefinitions or recursive functions, both of those operations never take in account the generated fn name so they are unaffected.</p><p>Summarizing some cases here from before/after the patch:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>1) top-level fn (always has name)
1.6 - namespace$name
patch - namespace$name
2) non-top-level fn with name
1.6 - namespace$name (collides with prior case)
patch - namespace$topname__x__name &lt;-- CHANGED
3) anonymous fn (no name)
1.6 - namespace$name$fn__x
patch - namespace$name$fn__x
4) top-level anonymous fn (no name, not at all useful :)
1.6 - namespace$fn__x
patch - namespace$fn__x
</pre>
</div></div>
<p>The key problem is that the first 2 cases produce the identical class name on 1.6. The patch alters the non-top-level named fn so there is no conflict. </p>
<p>Prior to the referenced old commit, I believe cases 1 and 2 would both produce namespace$name__x (where x is unique) so they would not collide. The change was made to prevent the top-level name from changing ("don't append numbers on top-level fn class names"). While the similar change was made on non-top-level fn names, I do not think it needed to be.</p>
<p>I've thought through (and tried) a bunch of the implications of this with the help of Nicola's comments above and I do not see an issue with any of the things I've considered. From a binary compatibility point of view with existing AOT code, old code compiled together should be self-consistent and continue to work. New compiled code will also be consistent. I can't think of a way that new code would expect to know the old name of a non-top-level function such that there could be an issue. </p>
<p>One question - why change the code such that the new class name is namespace$name$topname_&#95;x_&#95;name instead of namespace$name$topname_name_&#95;x (or something else?). And relatedly, while the diff is small, could we refactor a couple more lines to make the intent and cases clearer?</p>
<p>I am 90% ok with this patch but want a little thought into that question before I mark screened. </p><p>Alex, the attached patch munges into ns$topname_&#95;name_&#95;x, not into ns$topname_&#95;x_&#95;name.</p><p>The attached patch 0001-Fix-<a href="http://dev.clojure.org/jira/browse/CLJ-1330" title="Class name clash between top-level functions and defn&#39;ed ones"><del>CLJ-1330</del></a><del>refactored.patch contains the same fix from 0001-Fix</del><a href="http://dev.clojure.org/jira/browse/CLJ-1330" title="Class name clash between top-level functions and defn&#39;ed ones"><del>CLJ-1330</del></a>-make-top-level-named-functions-classnam.patch but also refactors the code that deals with fn name munging</p><p>Hmmm.. I will double-check. That's not why I recall seeing when I did AOT.</p><p>New patch 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart"><del>CLJ-1093</del></a>-v2.patch improves the fn naming scheme a lot.<br/>
I've threw together a number of test cases that show the improvement + bug fixes:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (fn [])
;; pre:
#&lt;user$eval1$fn__2 user$eval1$fn__2@4e13aa4e&gt;
;; post: (no change)
#&lt;user$eval1$fn__3 user$eval1$fn__3@3c92218c&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (fn a [])
;; pre:
#&lt;user$eval5$a__6 user$eval5$a__6@6946a317&gt;
;; post: (no change)
#&lt;user$eval6$a__8 user$eval6$a__8@6f85c59c&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (let [a (fn [])] a)
;; pre:
#&lt;user$eval9$a__10 user$eval9$a__10@15fdf894&gt;
;; post: (no change)
#&lt;user$eval11$a__13 user$eval11$a__13@4d051922&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (let [a (fn x [])] a)
;; pre: (only contains the name of the fn)
#&lt;user$eval17$x__18 user$eval17$x__18@7f0cd67f&gt;
;; post: (contains the name of the local aswell as the name of the fn
#&lt;user$eval21$a__x__23 user$eval21$a__x__23@528ef256&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def a (fn [])) a
#'user/a
;; pre:
#&lt;user$a user$a@33e1ccbc&gt;
;; post: (no change)
#&lt;user$a user$a@6bef63f9&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def a (fn x [])) a
#'user/a
;; pre: (BUG!)
#&lt;user$x user$x@59a04a1b&gt;
;; post: (bug fixed)
#&lt;user$a__x__28 user$a__x__28@5f0bebef&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def ^{:foo (fn [])} a) (-&gt; (meta #'a) :foo)
#'user/a
;; pre:
#&lt;user$fn__23 user$fn__23@d9c21c6&gt;
;; post: (no change)
#&lt;user$fn__30 user$fn__30@4cf0f2eb&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def ^{:foo (fn a [])} a) (-&gt; (meta #'a) :foo)
#'user/a
;; pre: (BUG!)
#&lt;user$a user$a@420dd874&gt;
;; post: (bug fixed)
#&lt;user$a__35 user$a__35@37ff95a9&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def a (fn [] (fn []))) (a)
#'user/a
;; pre:
#&lt;user$a$fn__30 user$a$fn__30@6f57be76&gt;
;; post: (no change)
#&lt;user$a$fn__41 user$a$fn__41@fd34eac&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (def a (fn [] (fn x []))) (a)
#'user/a
;; pre:
#&lt;user$a$x__35 user$a$x__35@79930089&gt;
;; post: (no change)
#&lt;user$a$x__48 user$a$x__48@6fc334de&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (let [x (fn [])] (def a (fn [] x))) a (a)
#'user/a
;; pre:
#&lt;user$eval40$a__43 user$eval40$a__43@6db1694e&gt;
#&lt;user$eval40$x__41 user$eval40$x__41@20bd16bb&gt;
;; post (no change)
#&lt;user$eval54$a__58 user$eval54$a__58@7c721de&gt;
#&lt;user$eval54$x__56 user$eval54$x__56@43f7b41b&gt;</pre>
</div></div><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (let [x (fn a [])] (def a (fn [] x))) (a)
#'user/a
;; pre: (the local binding name doesn't appear in the class name)
#&lt;user$eval48$a__49 user$eval48$a__49@75d6d1d4&gt;
;; post: (the local binding name is included in the class name)
#&lt;user$eval64$x__a__66 user$eval64$x__a__66@460d4&gt;</pre>
</div></div>
<p>As you can see, this last patch not only fixes the two bugs, but also improves fn naming in let contexts by preserving the name of the local binding in the class name, this I believe will be a great improvement in the understandability of stacktraces.</p><p>The patch should be changed to not create suffix if it's not going to be used. Please update the patch to inline that into each branch <tt>name = nm.name + "__" + RT.nextID();</tt>.</p>
<p>I am unsure whether the "enhancement" part of this patch goes too far. I think it does provide some improvements in debugging but those seem small to me. I am somewhat concerned about greatly increasing the name of the class for nested locals thus making it <b>harder</b> to read stack traces. There is a large limit to class name size of 16 bits (what you can put in the constant table) but class names also map to file names and there have historically been issues on some older Windows architectures with file size limits - we are increasing the risk of running into problems with this. Small risks. I am ok with passing this on to Rich though and he can decide whether to kick that part back or not.</p><p>0001&#45;CLJ&#45;1093&#45;v3.patch is identical to 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart"><del>CLJ-1093</del></a>-v2.patch except it doesn't call RT.nextID() when not necessary, as per Alex's request</p>
<p>Alex, if this is ok please change the "Patch:" field in the description, I won't do that myself since this ticket is now screened</p><p>Addressing the screening comment by Alex Miller, I've attached an alternative patch "0001-<a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart"><del>CLJ-1093</del></a><del>v3-no-locals-improv.patch" which is identical to "0001</del><a href="http://dev.clojure.org/jira/browse/CLJ-1093" title="Empty PersistentCollections get incorrectly evaluated as their generic clojure counterpart"><del>CLJ-1093</del></a>-v3.patch" except it doesn't include the local binding name enhancement, so that it can be picked in case Rich decides that that improvement is out of scope for this ticket.</p><p>I've reopened this issue based on early reports of breakage due to long file names. </p>
<p>Two reports:<br/>
<a href="https://groups.google.com/d/msg/clojure-dev/hnkJb9_il_M/4e5smM6mVlIJ">https://groups.google.com/d/msg/clojure-dev/hnkJb9_il_M/4e5smM6mVlIJ</a><br/>
<a href="https://groups.google.com/d/msg/clojure/hnkJb9_il_M/QOaTdCo5wmkJ">https://groups.google.com/d/msg/clojure/hnkJb9_il_M/QOaTdCo5wmkJ</a></p>
<p>Here's an example of a class name that is too long on Ubuntu 12.04 LTS 64bit / Java8 - reported max file size is 143 chars: </p>
<p><a href="https://github.com/ska2342/nested-fn-breaks-clojure-17/blob/master/src/nested_fn_breaks_clojure_17/core.clj">https://github.com/ska2342/nested-fn-breaks-clojure-17/blob/master/src/nested_fn_breaks_clojure_17/core.clj</a></p>
<p>With 1.6.0: (95 chars)<br/>
core$this_function_breaks_with_clojure_1_7$my_anonymous_function_<em>18$iter</em><em>19</em><em>23$fn</em><em>24$fn</em>_25</p>
<p>With 1.7.0-alpha3: (144 chars)<br/>
core$this_function_breaks_with_clojure_1_7$my_anonymous_function_<em>my_anonymous_function</em><em>19$iter</em><em>4951</em><em>auto</em>__<em>iter</em><em>20</em><em>24$fn</em><em>25$fn</em>_26.class</p>
<p>With the alternate patch here, the name would be: (95 chars)<br/>
core$this_function_breaks_with_clojure_1_7$my_anonymous_function_<em>19$iter</em><em>20</em><em>24$fn</em><em>25$fn</em>_26</p>
<p>patch "0001&#45;CLJ&#45;1330&#45;remove&#45;local&#45;binding&#45;name&#45;enhancement.patch" has the same effect of reverting f149260c14a75367dc9eba91cbe9b78110113566 and applying "0001&#45;CLJ&#45;1093&#45;v3&#45;no&#45;locals&#45;improv.patch" in case this is preferable</p><p>The tiny and unusual max file size of 143 is standard in the Ubuntu 12.04 crypto container for the home directory. You can get it for any directory with 'getconf NAME_MAX /path/to/dir'.</p>
<p>My initial problem (other than the file to reproduce on github) was triggered by the fns in a for-expression. Don't know if that makes any difference for you. </p>ApprovalOkGlobal RankPatchCode[CLJ-1315] Don't initialize classes when importing themhttp://dev.clojure.org/jira/browse/CLJ-1315
Clojure<p><b>Problem:</b> When classes are imported in Clojure, the class is loaded using Class.forName(), which causes its static initialisers to be executed. This differs from Java where compilation does not cause classes to be loaded. </p>
<p><b>Motivation:</b> In many cases when those classes are normally loaded by Java code during execution of a framework of some kind (IntelliJ in my case, and RoboVM is another culprit mentioned in that thread) the initialiser expects some infrastructure to be in place and will fail when it's not. This means that it's impossible to AOT compile namespaces importing these classes, which is a fairly serious limitation.</p>
<p><b>Approach:</b> Modify ImportExpr to call RT.classForNameNonLoading() instead of Class.forName(), which will load the class but not initialise it. This change causes the Clojure test suite to fail, since clojure.test-clojure.genclass imports a gen-class'ed class which no longer loads its namespace on initialisation. I'm not sure if this is considered an incorrect use of such a class (IIRC with records it's required to import the class and require its namespace), but given that it's in the Clojure test case it's reasonable to assume that this fix would be a breaking change for more code out there. This test failure is also corrected in the attached patch.</p>
<p><b>Patch:</b> 0001-Don-t-initialize-classes-during-import.patch</p>
<p><b>Screened by:</b> Alex Miller - I have tested many open source Clojure projects with this change (particularly seeking out large, complicated, or known users of genclass/deftype/etc) and have found no projects adversely impacted. I know that Cursive has been running with this modification for a long time with no known issues. I am ok with unconditionally enabling this change (re the comment below). The impact is described in more detail in the suggested changelog diff in the comments below.</p>
<p><b>Alternative:</b> This patch enables the change unconditionally, but depending on the extent of breakage it causes, it might need to be enabled with a configuration flag. I propose we make it unconditional in an early 1.7 beta and monitor the fall-out.</p>
<p><b>Background:</b> This issue has been discussed in the following threads<br/>
<a href="https://groups.google.com/d/topic/clojure/tWSEsOk_pM4/discussion">https://groups.google.com/d/topic/clojure/tWSEsOk_pM4/discussion</a><br/>
<a href="https://groups.google.com/forum/#!topic/clojure-dev/qSSI9Z-Thc0">https://groups.google.com/forum/#!topic/clojure-dev/qSSI9Z-Thc0</a></p>CLJ-1315Don't initialize classes when importing themEnhancementCriticalClosedCompletedUnassignedAaron CohenaotcompilerinteropSat, 28 Dec 2013 21:12:49 -0600Fri, 22 May 2015 14:58:37 -0500Tue, 7 Oct 2014 12:32:40 -0500Release 1.1Release 1.2Release 1.3Release 1.4Release 1.5Release 1.797<p>From original post:</p>
<p>This issue was originally reported by Zach Oakes and Colin Fleming and this patch was also tested by Colin.</p>
<p>I'm duplicating here my suggested release notes for this issue, which includes my current thoughts on potential breakage (it's also in the commit message of the patch):</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre> "import" no longer causes the imported class to be initialized. This
change better matches Java's import behavior and allows the importing of
classes that do significant work at initialization time which may fail.
This semantics change is not expected to effect most code, but certain
code may have depended on behavior that is no longer true.
1) importing a Class defined via gen-class no longer causes its defining
namespace to be loaded, loading is now deferred until first reference. If
immediate loading of the namespace is needed, "require" it directly.
2) Some code may have depended on import to initialize the class before it
was used. It may now be necessary to manually call (Class/forName
"org.example.Class") when initialization is needed. In most cases, this
should not be necessary because the Class will be initialized
automatically before first use.
</pre>
</div></div><p>I'm not sure if this should also be fixed, but it would be nice if you could emit the code for a proxy of one of these non-initialized classes without forcing initialization. For example, the following raises an exception (I'm using Java 8):</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">Clojure 1.6.0
user=&gt; (def cname <span class="code-quote">"javafx.scene.control.ListCell"</span>)
#'user/cname
user=&gt; (let [cls (<span class="code-object">Class</span>/forName cname <span class="code-keyword">false</span> (clojure.lang.RT/baseLoader))] (.importClass *ns* cls))
javafx.scene.control.ListCell
user=&gt; (defn fails [] (proxy [ListCell] [] (updateItem [item empty] (proxy-<span class="code-keyword">super</span> item empty))))
CompilerException java.lang.ExceptionInInitializerError, compiling:(NO_SOURCE_PATH:3:16)</pre>
</div></div>
<p>The exception was ultimately caused by "IllegalStateException Toolkit not initialized", which javafx throws if you attempt to initialize a Control class outside of <tt>Application.launch</tt>.</p>
<p>Not sure if this should properly be considered a bug in Cloverage, but since this patch landed, I've been unable to get coverage in gen-class methods.</p>
<p><a href="https://github.com/lshift/cloverage/issues/74">https://github.com/lshift/cloverage/issues/74</a></p>ApprovalOkGlobal RankPatchCode[CLJ-1244] :prefix is ignored when trying to gen-class with custom methodshttp://dev.clojure.org/jira/browse/CLJ-1244
Clojure<p>I'm trying to generate several classes defined in one namespace:</p>
<p> (ns aot.core)</p>
<p> (gen-class<br/>
:name aot.core.ClassA<br/>
:preﬁx "classA-")</p>
<p> (gen-class<br/>
:name aot.core.ClassB<br/>
:preﬁx "classB-")</p>
<p> (defn classA-toString<br/>
<span class="error">&#91;this&#93;</span><br/>
"I'm an A.")</p>
<p> (defn classB-toString<br/>
<span class="error">&#91;this&#93;</span><br/>
"I'm a B.")</p>
<p>After AOT I can see that</p>
<p>(.toString (ClassA.)) </p>
<p>doesn't produce "I'm an A." string but rather uses method from the superclass &gt;&gt; "aot.core.ClassA@33ba4e15" </p>
<p>If on other hand I do:</p>
<p> (ns aot.core)</p>
<p> (gen-class<br/>
:name aot.core.ClassA<br/>
:preﬁx "classA-")</p>
<p> (gen-class<br/>
:name aot.core.ClassB<br/>
:preﬁx "classB-")</p>
<p> (defn -toString<br/>
<span class="error">&#91;this&#93;</span><br/>
"I'm an A.")</p>
<p> (defn -toString<br/>
<span class="error">&#91;this&#93;</span><br/>
"I'm a B.")</p>
<p>then both</p>
<p> (.toString (ClassA.)) </p>
<p>and</p>
<p> (.toString (ClassB.)) </p>
<p>obviously give "I'm a B." as there is only one -toString really defined.</p>
<p>Is it a bug? Am I doing something wrong? How can I make clojure respect :prefix option.</p>CLJ-1244:prefix is ignored when trying to gen-class with custom methodsDefectMajorClosedDeclinedUnassignedDaniel KwiecinskiAOTgen-classWed, 14 Aug 2013 16:14:50 -0500Fri, 16 Aug 2013 08:59:47 -0500Fri, 16 Aug 2013 08:59:47 -0500Release 1.501<p>Please close the ticket. I have copied an example from a blog post which had wrong keyword.<br/>
Instead of :prefix it had :preﬁx. Notice that ﬁ in :preﬁx is single character (see: ﬁ fi ) </p><p>Closed per request</p>Global Rank[CLJ-1241] NPE when AOTing overrided clojure.core functionshttp://dev.clojure.org/jira/browse/CLJ-1241
Clojure<p>When performing AOT compilation on a namespace that overrides a clojure.core function without excluding the original clojure.core function from the ns, you get a NullPointerException. </p>
<p>To reproduce aot compile a namespace like "(ns x) (defn get [])"</p>
<p>For example:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">$ lein <span class="code-keyword">new</span> aot-get
$ cd aot-get
$ sed -i s/foo/get/
$ lein compile :all
WARNING: get already refers to: #'clojure.core/get in namespace: aot-get.core, being replaced by: #'aot-get.core/get
Exception in thread <span class="code-quote">"main"</span> java.lang.NullPointerException
at clojure.lang.<span class="code-object">Compiler</span>$ObjExpr.emitVar(<span class="code-object">Compiler</span>.java:4858)
at clojure.lang.<span class="code-object">Compiler</span>$DefExpr.emit(<span class="code-object">Compiler</span>.java:428)
at clojure.lang.<span class="code-object">Compiler</span>.compile1(<span class="code-object">Compiler</span>.java:7152)
at clojure.lang.<span class="code-object">Compiler</span>.compile(<span class="code-object">Compiler</span>.java:7219)
at clojure.lang.RT.compile(RT.java:398)
at clojure.lang.RT.load(RT.java:438)
at clojure.lang.RT.load(RT.java:411)</pre>
</div></div>
<p><b>Cause:</b> DefExpr.parse does not call registerVar for vars overridding clojure.core ones, thus when AOT compiling the var is not registered in the constant table.</p>
<p><b>Proposed:</b> The attached patch makes DefExpr.parse call registerVar for vars overridding clojure.core ones.</p>
<p><b>Patch:</b> 0001-fix-<a href="http://dev.clojure.org/jira/browse/CLJ-1241" title="NPE when AOTing overrided clojure.core functions"><del>CLJ-1241</del></a>.patch</p>
<p><b>Screened by:</b> Alex Miller</p>CLJ-1241NPE when AOTing overrided clojure.core functionsDefectMajorClosedCompletedUnassignedPhil HagelbergaotcompilerTue, 30 Jul 2013 13:56:10 -0500Fri, 29 Aug 2014 12:00:25 -0500Fri, 29 Aug 2014 12:00:25 -0500Release 1.5Release 1.6Release 1.726<p>DefExpr.parse was not calling registerVar for vars overridding clojure.core ones.</p><p>Verified on Clojure 1.5.1.</p><p>Reproduced with `key` function without `(:refer-clojure :exclude <span class="error">&#91;key&#93;</span>)`</p><p>This doesn't meet triage guidelines - i.e. there is this problem, therefore we will fix it by _____ so it then does _____</p><p>This is still present in the 1.6 release. I think it's mis-classified as low priority.</p><p>See for instance the cascalog mailing list: <a href="https://groups.google.com/forum/#!topic/cascalog-user/Pe5QIpmU0vA">https://groups.google.com/forum/#!topic/cascalog-user/Pe5QIpmU0vA</a></p><p>It may help if someone could clarify Rich's comment.</p>
<p>Does it mean that the ticket <b>should</b> include a plan of the form "therefore we will fix it by _____ so it then does _____", but this ticket doesn't have that?</p>
<p>Or perhaps it means that the ticket <b>should not</b> include a plan of that form, but this ticket does? If so, I don't see it, except perhaps the very last sentence of the description. If that is a problem for vetting a ticket, perhaps we could just delete that sentence and proceed from there?</p>
<p>Something else?</p><p>Andy, I added the two last lines in the description after reading Rich's comment to explain why this bug happens and how the patch I attached works around this.</p>
<p>I don't know if this is what he was asking for though.</p><p>I think Rich meant that a ticket <b>should</b> have a plan of that form but does not. My own take on "triaged" is that it should state actual and expected results demonstrating a problem - I don't think it needs to actually describe the solution (as that can happen later in development). It is entirely possible that Rich and I differ in our interpretation of that. <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/> I will see if I can rework the description a bit to match what I've been doing elsewhere. </p><p>Alex, I have looked through the existing wiki pages on the ticket tracking process, and do not recall seeing anything about this desired aspect of a triaged ticket. Is it already documented somewhere and I missed it? Not that it has to be documented, necessarily, but Rich saying "triage guidelines" makes it sound like a filter he applies that ticket creators and screeners maybe should know about.</p><p>To me, Triage (and Vetting) is all about having good problem statements. For a defect, it is most important to demonstrate the problem (what happens now) and what you expect to happen instead. I do not usually expect there to necessarily be "by ____" in the ticket - to me that is part of working through the solution (although it is typical to have this in an enhancement). This ticket, as it stands now, seems to have both a good problem statement and a good cause/solution statement so seems to exceed Triaging standards afaik.</p>
<p>Two places where I have tried to write about these things in the past are <a href="http://dev.clojure.org/display/community/Creating+Tickets">http://dev.clojure.org/display/community/Creating+Tickets</a> and in the Triage process on the workflow page <a href="http://dev.clojure.org/display/community/JIRA+workflow">http://dev.clojure.org/display/community/JIRA+workflow</a>. </p>ApprovalOkGlobal RankPatchCode[CLJ-1227] Definline functions do not work as higher-order functions when AOT compiledhttp://dev.clojure.org/jira/browse/CLJ-1227
Clojure<p>See discussion on Clojure group: <a href="https://groups.google.com/d/topic/clojure/v0ipoiP8X1o/discussion">https://groups.google.com/d/topic/clojure/v0ipoiP8X1o/discussion</a></p>
<p>Functions defined with definline appear to misbehave when AOT compiled and used with higher-order functions - it seems like the macro function is stored instead of the expansion. I've attached a small test project here reproducing the issue. It can be run with:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">lein compile
lein uberjar
java -jar target/aot-test-0.1.0-SNAPSHOT-standalone.jar</pre>
</div></div>
<p>The relevant part of the test namespace is:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">(definline java-list? [element]
`(instance? List ~element))
(defn -main [&amp; args]
(println (java-list? 2))
(println ((<span class="code-keyword">var</span>-get #'java-list?) 2))
(println (filter java-list? [1 2 3]))
(println (map java-list? [1 2 3])))</pre>
</div></div>
<p>The output produced is:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java"><span class="code-keyword">false</span>
(clojure.core/instance? java.util.List 2)
(1 2 3)
((clojure.core/instance? java.util.List 1) (clojure.core/instance? java.util.List 2) (clojure.core/instance? java.util.List 3))</pre>
</div></div>OSX 10.8, LinuxCLJ-1227Definline functions do not work as higher-order functions when AOT compiledDefectBlockerClosedDeclinedUnassignedColin FlemingaotThu, 27 Jun 2013 01:44:18 -0500Wed, 22 Jan 2014 15:35:32 -0600Tue, 21 Jan 2014 09:36:08 -0600Release 1.513<p>I've just fallen into the very same trap. This is definitively a blocker as it means that AOT-compiled code will probably not work correctly if there's a definline in any dependency (transitively).</p><p>BTW, the problem also applies to the definlines defined in clojure.core such as ints, longs, doubles, etc.</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>;; This NS is AOT-compiled
(ns aot-test.core
(:gen-class))
(defn -main [&amp; args]
(let [ary (make-array Integer/TYPE 5)]
(println (apply ints [ary]))))
</pre>
</div></div>
<p>The output is:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>% lein uberjar &amp;&amp; java -jar target/aot-test-0.1.0-SNAPSHOT-standalone.jar
Compiling aot-test.core
Compiling aot-test.core
Created /home/horn/tmp/aot-test/target/aot-test-0.1.0-SNAPSHOT.jar
Created /home/horn/tmp/aot-test/target/aot-test-0.1.0-SNAPSHOT-standalone.jar
(. clojure.lang.Numbers clojure.core/ints #&lt;int[] [I@39b65439&gt;)
</pre>
</div></div><p>I debugged a bit further. What made me wonder is why standard clojure functions with :inline metadata work correctly in AOT-scenarios whereas definlines which expand to normal functions with :inline metadata do not.</p>
<p>The bug can be fixed by adding the :inline metadata immediately in the defn form in the expansion instead of providing it after the defn form using alter-meta!. Here's a demo example:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(ns aot-test.core
(:gen-class))
;; (definline n? [x]
;; `(clojure.lang.Util/identical ~x nil))
;;
;; It expands into the following which doesn't work in AOT-scenarios, e.g.,
;; (apply n? [nil]) returns (clojure.lang.Util/identical nil nil) rather than
;; true.
(do
(clojure.core/defn n? [x] (clojure.lang.Util/identical x nil))
(clojure.core/alter-meta!
#'n?
clojure.core/assoc
:inline
(clojure.core/fn n? [x]
(clojure.core/seq
(clojure.core/concat
(clojure.core/list 'clojure.lang.Util/identical)
(clojure.core/list x)
(clojure.core/list 'nil)))))
#'n?)
;; If the expansion would look like this, i.e., the metadata is added directly
;; instead of with alter-meta!, then it works fine in AOT-scenarios.
(do
(clojure.core/defn nl?
{:inline (clojure.core/fn fn? [x]
(clojure.core/seq
(clojure.core/concat
(clojure.core/list 'clojure.lang.Util/identical)
(clojure.core/list x)
(clojure.core/list 'nil))))}
[x] (clojure.lang.Util/identical x nil)))
(defn -main [&amp; args]
(println "n?")
(println (meta #'n?)) ;=&gt; {:inline #&lt;core$n_QMARK_ aot_test.core$n_QMARK_@78ffb648&gt;, :ns #&lt;Namespace aot-test.core&gt;, :name n?, :arglists ([x]), :column 3, :line 8, :file aot_test/core.clj}
(println (apply n? [nil])) ;=&gt; (clojure.lang.Util/identical nil nil) BROKEN!!!
(println (apply n? [1])) ;=&gt; (clojure.lang.Util/identical 1 nil) BROKEN!!!
(println (n? nil)) ;=&gt; true
(println (n? 1)) ;=&gt; false
(println "nl?")
(println (meta #'nl?)) ;=&gt; {:ns #&lt;Namespace aot-test.core&gt;, :name nl?, :file aot_test/core.clj, :column 3, :line 22, :arglists ([x]), :inline #&lt;core$fn_QMARK_ aot_test.core$fn_QMARK_@3b7541a&gt;}
(println (apply nl? [nil])) ;=&gt; true
(println (apply nl? [1])) ;=&gt; false
(println (nl? nil)) ;=&gt; true
(println (nl? 1))) ;=&gt; false
</pre>
</div></div><p>Here's a patch fixing the issue by making definline expand only to a defn form where the :inline metadata is added immediately instead of providing it afterwards with alter-meta!.</p>
<p>I also added a deftest in test_clojure/macros.clj which checks for the correctness of the expansions.</p>
<p>Although this patch fixes the problem, it should be somehow/somewhere documented why the former approach didn't work in AOT-scenarios.</p><p>definline should be considered to be like defmacro in that it is not a function and cannot be used as a HOF. Additionally, definline is considered to be an experimental feature and Rich would like to discourage its use as the hope is to remove it in the future. The desired replacement is something like common lisp compiler macros that could allow the compiler to detect special situations and optimize the result but leave behind a function invocation for the case where no special behavior is available. </p>
<p>I am declining the ticket based on the info above and per Rich's request.</p><p>This is a little disappointing since there's really no alternative right now, and I'm assuming that this sort of advanced optimisation is a ways off. If this is the plan, I'd definitely recommend marking definline as deprecated and making clear in the docstring that it shouldn't be relied on to return a function. Currently it states: "like defmacro, except defines a named function...", and based on this ticket that is clearly people's expectation.</p><p>Sorry about that. I will now split some definitional hairs by saying that definline is marked as "experimental" to indicate it may or may not even become a feature whereas "deprecation" indicates that it was a feature which you should no longer use. I think in this case they serve the same functional intent which is to warn you away from relying on it as part of the language. </p><p>Fair enough, although it's been experimental for five major releases now. If we're convinced it's a failed experiment (and if it can't be relied on to create a function, it pretty much is since you might as well use a macro) I'd argue that deprecation is justified and sends a stronger signal that it doesn't actually work as intended. But either way, I'm certainly convinced now <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/smile.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p><p>Alex, <tt>definline</tt>'s intention is to be considered like defmacro in case of <tt>(my-definline x)</tt> where it saves one function call, but to be callable as a normal function in higher-order scenarios like (map my-definline <span class="error">&#91;x y z&#93;</span>). Therefore, it expands to a normal <tt>defn</tt> form with <tt>:inline</tt> metadata which is a macro that's used by the compiler.</p>
<p>Wether it should be removed in the future is a separate issue. It is there right now, and people use it. The consequence of this bug is that you cannot trust AOT-compiled code, because there might be some dependency that uses definline. Additionally, all clojure.core definlines (booleans, bytes, chars, shorts, floats, ints, doubles, longs) are broken when applied as HOF, because clojure itself is always distributed in AOT-compiled form.</p>
<p>Really, it would be grossly negligent to release Clojure 1.6 knowing of this bug.</p>
<p>I've written a more detailed mail to the clojure-dev list:</p>
<p> <a href="https://groups.google.com/d/msg/clojure-dev/UeLNJzp7UiI/UCyVvYLfHWMJ">https://groups.google.com/d/msg/clojure-dev/UeLNJzp7UiI/UCyVvYLfHWMJ</a></p><p>The root cause of this issue is <a href="http://dev.clojure.org/jira/browse/CLJ-1330" title="Class name clash between top-level functions and defn&#39;ed ones"><del>CLJ-1330</del></a> as investigated by Nicola Mometto in the thread cited above: <a href="http://dev.clojure.org/jira/browse/CLJ-1330">http://dev.clojure.org/jira/browse/CLJ-1330</a><br/>
So fixing the latter will fix this one, too.</p>Global RankPatchCode and Test[CLJ-1129] Invalid ns macro can yield a difficult to trace exceptionhttp://dev.clojure.org/jira/browse/CLJ-1129
Clojure<p>I inadvertently stripped off the namespace part of my ns macro, so that it was <tt>(ns (:use ...</tt>. Clearly a user error, but an easy one. However, the result (from the REPL or the AOT compiler) was not ideal:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>Exception in thread "main" java.lang.ClassCastException: clojure.lang.PersistentList cannot be cast to clojure.lang.Named
at clojure.core$name.invoke(core.clj:1489)
at clojure.core$root_resource.invoke(core.clj:5210)
at clojure.core$load_one.invoke(core.clj:5227)
at clojure.core$compile$fn__4895.invoke(core.clj:5426)
at clojure.core$compile.invoke(core.clj:5425)
at clojuresque.tasks.compile$main$fn__64.invoke(compile.clj:23)
at clojuresque.cli$with_command_line_STAR_.invoke(cli.clj:92)
at clojuresque.tasks.compile$main.doInvoke(compile.clj:6)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:601)
at clojure.lang.Var.invoke(Var.java:419)
at clojuresque.Driver.main(Driver.java:39)
</pre>
</div></div>
<p>The problem here is that there is no indication of what file was being loaded and compiled at the time of the error. Since I was in the middle of refactoring a big swath of code, I had some work to do to track down which file I had mangled.</p>
<p>I would like to see a little more logging to the System/err to identify what resource file was being read and compiled by the compiler at the time of the exception.</p>CLJ-1129Invalid ns macro can yield a difficult to trace exceptionDefectMinorClosedDuplicateUnassignedHoward Lewis ShipaotfeedbackMon, 17 Dec 2012 16:56:53 -0600Wed, 19 Dec 2012 01:09:27 -0600Wed, 19 Dec 2012 01:09:27 -0600Release 1.400<p>Howard, is this perhaps a duplicate of <a href="http://dev.clojure.org/jira/browse/CLJ-939" title="Exceptions thrown in the top level ns form are reported without file or line number"><del>CLJ-939</del></a>? Let me know, and I can close this ticket as a duplicate if so.</p><p>Yes, looks like a dupe to me. Sorry about that, I did do a search for existing issue before adding mine, but they can be hard to find.</p><p>Closed as duplicate of <a href="http://dev.clojure.org/jira/browse/CLJ-939" title="Exceptions thrown in the top level ns form are reported without file or line number"><del>CLJ-939</del></a>.</p>Global Rank[CLJ-979] Clojure resolves to wrong deftype classes when AOT compiling or reloadinghttp://dev.clojure.org/jira/browse/CLJ-979
Clojure<p>Compiling a class via `deftype` during AOT compilation gives different results for the different constructors. These hashes should be identical.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user=&gt; (binding [*compile-files* <span class="code-keyword">true</span>] (eval '(deftype Abc [])))
user.Abc
user=&gt; (hash Abc)
16446700
user=&gt; (hash (class (-&gt;Abc)))
31966239 ;; should be 16446700</pre>
</div></div>
<p>This also means that whenever there's a stale AOT compiled deftype class in the classpath, that class will be used rather then the JIT compiled one, breaking repl interaction.</p>
<p>Another demonstration of this classloader issue (from <a href="http://dev.clojure.org/jira/browse/CLJ-1495" title="Defining a record with defrecord twice breaks record equality"><del>CLJ-1495</del></a>) when reloading deftypes (no AOT) :</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">user&gt; (defrecord Foo [bar])
user.Foo
user&gt; (= (-&gt;Foo 42) #user.Foo{:bar 42}) ;;expect <span class="code-keyword">this</span> to evaluate to <span class="code-keyword">true</span>
<span class="code-keyword">true</span>
user&gt; (defrecord Foo [bar])
user.Foo
user&gt; (= (-&gt;Foo 42) #user.Foo{:bar 42}) ;;expect <span class="code-keyword">this</span> to evaluate to <span class="code-keyword">true</span> also -- but it doesn't!
<span class="code-keyword">false</span>
user&gt;</pre>
</div></div>
<p>This bug also affects AOT compilation of multimethods that dispatch on a class, this affected core.match for years see <a href="http://dev.clojure.org/jira/browse/MATCH-86">http://dev.clojure.org/jira/browse/MATCH-86</a>, <a href="http://dev.clojure.org/jira/browse/MATCH-98">http://dev.clojure.org/jira/browse/MATCH-98</a>. David had to work-around this issue by using a bunch of protocols instead of multimethods.</p>
<p><b>Cause of the bug:</b> currently clojure uses Class.forName to resolve a class from a class name, which ignores the class cache from DynamicClassLoader thus reloading deftypes or mixing AOT compilation at the repl with deftypes breaks, resolving to the wrong class.</p>
<p><b>Approach:</b> the current patch (<a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>-v7.patch) addresses this issue in multiple ways:</p>
<ul class="alternate" type="square">
<li>it makes RT.classForName/classForNameNonLoading look in the class cache before delegating to Class/forName if the current classloader is not a DynamicClassLoader (this incidentally addresses also <a href="http://dev.clojure.org/jira/browse/CLJ-1457" title="once the compiler pops the dynamic classloader from the stack, attempts to read record reader literals will fail"><del>CLJ-1457</del></a>)</li>
<li>it makes clojure use RT.classForName/classForNameNonLoading instead of Class/forName</li>
<li>it overrides Classloader/loadClass so that it's class cache aware &#8211; this method is used by the jvm to load classes</li>
<li>it changes gen-interface to always emit an in-memory interface along with the <span class="error">&#91;optional&#93;</span> on-disk interface so that the in-memory class is always updated.</li>
</ul>
<p><b>Patch:</b> <a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>-v7.patch</p>
<p><b>Screened by:</b> Alex Miller</p>CLJ-979Clojure resolves to wrong deftype classes when AOT compiling or reloadingDefectCriticalClosedCompletedUnassignedEdmund JacksonaotclassloadercompilerThu, 3 May 2012 03:01:37 -0500Wed, 18 Feb 2015 04:21:23 -0600Sat, 10 Jan 2015 09:24:25 -0600Release 1.3Release 1.4Release 1.5Release 1.6Release 1.7Release 1.71410<p>I can't reproduce this under Clojure 1.3 or 1.4, and Leiningen 1.7.1 on either Java 1.7.0-jdk7u4-b21 OpenJDK 64-Bit or Java 1.6.0_31 Java HotSpot 64-Bit. OS is Mac OS X 10.7.</p>
<p>Edmund, how are you running this AOT code? I wrapped your code in a main function and built an uberjar from it.</p><p>Hi Scott,</p>
<p>Interesting. </p>
<p>I have two use cases<br/>
1. AOT compile and call from repl.<br/>
My steps: git clone, lein compile, lein repl, (use 'aots.death), (in-ns 'aots.death), (= (class (Dontwork. nil)) (class (map-&gt;Dontwork {:a 1}))) =&gt; false</p>
<p>2. My original use case, which I've minimised here, is an AOT ns, producing a genclass that is called instantiated from other Java (no main). This produces the same error. I will produce an example of this and post it too.</p><p>Hi Scott,</p>
<p>Here is an example of it failing in the interop case: <a href="https://github.com/ejackson/aotquestion2">https://github.com/ejackson/aotquestion2</a><br/>
The steps I'm following to compile this all up are</p>
<p>git clone git@github.com:ejackson/aotquestion2.git<br/>
cd aotquestion2/cljside/<br/>
lein uberjar<br/>
lein install<br/>
cd ../javaside/<br/>
mvn package<br/>
java -jar ./target/aotquestion-1.0-SNAPSHOT.jar</p>
<p>and it dies with this:</p>
<p>Exception in thread "main" java.lang.ClassCastException: cljside.core.Dontwork cannot be cast to cljside.core.Dontwork<br/>
at cljside.MyClass.makeDontwork(Unknown Source)<br/>
at aotquestion.App.main(App.java:8)</p>
<p>The error message is really confusing (to me, anyway), but I think its the same root problem as for the REPL case.</p>
<p>What do you see when you run the above ?</p><p>Ah, yes, looks like my initial attempt to reproduce was too simplistic. I used your second git repo, and can now confirm that it's failing for me with the same error.</p><p>I looked into this a little further and the AOT generated code looks correct, in the sense that both code paths appear to be returning the same type.</p>
<p>However, I wonder if this is really a ClassLoader issue, whereby two definitions of the same class are being loaded at different times, because that would cause the x.y.Class cannot be cast to x.y.Class exception that we're seeing here.</p><p>This could be related to <a href="http://dev.clojure.org/jira/browse/CLJ-1157" title="Classes generated by gen-class aren&#39;t loadable from remote codebase for mis-implementation of static-initializer"><del>CLJ-1157</del></a> which deals with a ClassLoader issue with AOT compiled code.</p><p>I've tried <a href="http://dev.clojure.org/jira/secure/attachment/12711/20140121_fix_classloader.diff">this</a> patch attached to <a href="http://dev.clojure.org/jira/browse/CLJ-1157" title="Classes generated by gen-class aren&#39;t loadable from remote codebase for mis-implementation of static-initializer"><del>CLJ-1157</del></a> and it did not solve this issue.</p><p>This bug seems to be rooted in different behaviour for do/let under compilation. Attached a patch showing these symptoms in the hope it helps people find the cause.</p><p>Just a quick note to confirm that this still seems to be around as of Clojure 1.7.0-alpha2. Don't have any useful input on possible solutions, sorry.</p><p>Duplicates - <a href="http://dev.clojure.org/jira/browse/CLJ-1495" title="Defining a record with defrecord twice breaks record equality"><del>CLJ-1495</del></a>, <a href="http://dev.clojure.org/jira/browse/CLJ-1132" title="For record type Rec, (instance? Rec (map-&gt;Rec {...})) need not return true, though (instance? Rec (Rec. ...)) does."><del>CLJ-1132</del></a></p><p>The attached patch fixes the classloader issues by routing RT.classForName &amp; variants through the DynamicClassLoader class cache before invoking Class.forName</p><p>Re-adding triaged status added by Alex Miller that got accidentaly nuked by a race-condition between my edits to the ticket description and Alex's ones</p><p>0001-<a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>&#45;make-clojure-resolve-to-the-correct-Class-in-v2.patch is the same as 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>-make-clojure-resolve-to-the-correct-Class-in.patch except it unconditionally looks for classes in the class cache of DynamicClassLoader, even if baseLoader() is not a DynamicClassLoader.<br/>
This fixes the bug of <a href="http://dev.clojure.org/jira/browse/CLJ-1457" title="once the compiler pops the dynamic classloader from the stack, attempts to read record reader literals will fail"><del>CLJ-1457</del></a> but might just be a workaround</p><p>Current patch blows up my Clojure build</p>
<p><a href="https://gist.github.com/MichaelBlume/aa26fc715cbbdf711290">https://gist.github.com/MichaelBlume/aa26fc715cbbdf711290</a></p><p>Michael: the current patch builds clojure fine for me, I'll try to reproduce. Which jvm version are you using?</p><p><span class="error">&#91;14:24&#93;</span><span class="error">&#91;michael.blume@tcc-michael-4:~/workspace/clojure((0fc43db...))&#93;</span>$ java -version<br/>
java version "1.8.0_25"<br/>
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)<br/>
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)<br/>
<span class="error">&#91;14:24&#93;</span><span class="error">&#91;michael.blume@tcc-michael-4:~/workspace/clojure((0fc43db...))&#93;</span>$ mvn -version<br/>
Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T13:58:10-07:00)<br/>
Maven home: /usr/local/Cellar/maven/3.2.3/libexec<br/>
Java version: 1.8.0_25, vendor: Oracle Corporation<br/>
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/jre<br/>
Default locale: en_US, platform encoding: UTF-8<br/>
OS name: "mac os x", version: "10.10.1", arch: "x86_64", family: "mac"</p>
<p>build was after I applied the patch to the current master branch of the clojure github repo</p><p>I am seeing a similar compilation error as Michael Blume, with both JDK 1.7 and 1.8 on Mac OS X 10.9.5.</p>
<p>By accident I found that if I take latest Clojure master and do 'mvn package', then apply the patch <a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>.patch dated Dec 11 2014, then do 'mvn package' again without 'mvn clean', it compiles with no errors. If I do 'mvn clean' then 'mvn package' in a patched tree, I get the error every time I've tried.</p><p>The updated patch fixes the LinkageError Andy and Michael were getting. </p>
<p>Andy, Michael, can you confirm?</p><p>Added more testcases to new patch</p><p>Cleaned up the patch from whitespace changes</p><p>I tried latest Clojure master plus patch <a href="http://dev.clojure.org/jira/browse/CLJ-979" title="Clojure resolves to wrong deftype classes when AOT compiling or reloading"><del>CLJ-979</del></a>-v4.patch, dated 12 Dec 2014, with Mac OS X 10.9.5 + JDK7, and Ubuntu Linux 14.04 with JDKs 6 through 9, and 'mvn clean' followed by 'mvn package' built and passed tests successfully with all of them.</p>
<p>I did notice that some files were created in the test directory that were not cleaned up by the end of the test, which you can use 'git status .' to see. Not sure if that is considered a bad thing for Clojure tests.</p><p>Thanks Andy, I've updated the patch and it now should remove all temporary classes created by the test.<br/>
It's probably not the best way to do it but I couldn't figure out how to do it another way.</p><p>Yep, looks good to me =)</p><p>Thanks first to Nicola for all his work so far on this!</p>
<p>Some feedback:<br/>
1) While the ticket itself isn't bad, I would really like to focus the title and description on a crisp statement of the real problem now that we understand it more. I'd like help on making sure we capture that correctly - how is this for a title: "Uses of Class.forName() miss classes in DynamicClassLoader cache?" ? </p>
<p>Similarly, the description should focus on the problem and show some examples. The defrecord one is good. The first example works for me before the patch and fails after? </p>
<p>2) The crux of this whole thing is the change in loading order in DCL.loadClass() - changing this is a big deal. We really need broader testing with things likely to be affected - off the top of my head: Immutant, Pomegranate, Leiningen, or anything else that monkeys with classloader stuff. Maybe something with Eclipse/OSGi if there is something testable out there.</p>
<p>3) DynamicClassLoader comments:<br/>
a) loadClass(String name) - I believe this is identical to the super impl, so can be removed.<br/>
b) findClass(String name) - now that we are hijacking loadClass(), I'm not sure it's even necessary to implement this or to call super.findClass() - if you get to the super.findClass(), I would expect that to always throw CNFE. Potentially, this method could even be removed (but this might do bad things if there are subclasses of DCL out there in the wild). <br/>
c) loadClass(String name, ...) - instead of calling findClass() and using the CNFE for control flow, you could just directly call findInMemoryClass(), then use a returned null as the decision factor. Again, this is possibly bad if there are DCL subclasses, so I'm on the fence about it. </p>
<p>4) Is the change in gen-interface something that should be a separate ticket? Seems like it could be separable.</p>
<p>5) I don't like the test changes with respect to set up and cleanup. The build already supports compiling a subset of test ns'es (like clojure.test-clojure.genclass.examples). I'd prefer to use that existing mechanism if at all possible. Check the build.xml for the hard-coded ns list. </p>
<p>6) What are the performance implications? I'm not expecting they are significant but we just made a bunch of changes for compilation performance and I'd hate to undo them all. Could findInMemoryClass be smarter about avoiding checks that can't succeed (avoiding "java.*" for example?).</p><p>1) It's not really about Class.forName() specifically, it's about DynamicClassLoader not being class cache aware in the loadClass method. The JVM uses the classloader loadClass method for resolving all kind of class usages,<br/>
including but not limited to Class.forName() (i.e. when loading some bytecode containing a "new" instruction, that class reference will be resolved via a call to loadClass)<br/>
I'll try to make the documentation a bit more clear, the first example is an exhibit of the bugged behaviour, the two calls should output the same hash.</p>
<p>2,4) So, there are 3 approaches to how DynamicClassLoader could go at it:</p>
<ul class="alternate" type="square">
<li>Prefer in-disk classes over in-memory classes, roughly the current approach (sometimes it will pick the in-memory class over the in-disk one causing weird bugs like Foo. and -&gt;Foo constructing different classes), has the<br/>
negative effect of breaking interaction between AOT compilation and JIT loading, which has created all sorts of troubles with redefinig deftypes/defprotocols in repls while having stale classfiles in disk.</li>
<li>Always pick the most-updated class, this has the advangate of being <b>always</b> correct but has several disadvantages that make it inpracticable in my opinion: we'd have to keep track of the timestamp in which a dynamic class<br/>
is defined, and make the loadClass implementation such that if there a class is both in-memory and in-disk, it compares the timestamps and select the most updated one. This would complicate the implementation a lot and we'd<br/>
likely have to pay a substantial performance hit.</li>
<li>Prefer in-memory classes over in-disk classes, the approach proposed in the current patches. It has the advantage of being almost always correct, make repl interaction &amp; jit/aot mixing work fine and the implementation is<br/>
mostly straightforward. The downside is that in cases like gen-class where an AOT class can actually be the most updated version, the in-memory version will be used. In clojure all the forms that do bytecode emission either<br/>
only do AOT compilation or do AOT compilation on demand and <b>always</b> load the class in memory, except gen-interface that doesn't load the class in memory if it's being AOT compiled. Changing its semantics to behave like the<br/>
other jit/aot compiling forms (deftype/defrecord/reify) is the only way to make this approach work so I don't think this should go in another ticket.</li>
</ul>
<p>5) I don't like the previous testing strategy either but couldn't figure out a better way. Thanks for the pointer on the already in-place infrastructure, I'll check it out and update the patch</p>
<p>In the meantime I've uploaded a new patch addressing 3 and 6. Specifically:<br/>
3) I removed the unnecessary loadClass(String) arity, I've made loadClass(String, boolean) use findInMemoryClass(String) directly rather than relying on findClass(String) since nowhere in the documentation it guarantess that<br/>
findClass will be used by loadClass. However I've left the findClass(String) implementation in place in case there's code out there that relies on this.<br/>
6) I haven't done any serious testing but I haven't noticed any significant difference in compile times for any of my tools.* contrib libraries with the current patch. Filtering "java.*" class names before the inMemory check<br/>
didn't seem to produce any difference so it's not included in the updated patch. However I'll probably include an alternative patch with that filtering to do more performance testings and see if it can actually help a bit.</p>
<p>All this said, I'm afraid that I won't have time to personally do an in-depth benchmarking &amp; cross project testing of this patch. I've been spending almost all the free time I had in the past weeks working through a bunch of tickets (mostly this one) but now because of school and other commitments I can't promise I will be able to do anything more than maintaining the current patch &amp; answering to any questions about the bug. Any help in moving this ticket further would be appreciated, in particular to address points 2 and 6.</p><p>Thanks Nicola. I'll certainly take over sheparding the bug and appeal to the greater community for help in broad testing when I think we're ready for that.</p><p>Updated patch with better tests, addressing Alex Miller's comments.</p><p>New in-the-wild case: <a href="https://groups.google.com/forum/#!topic/clojure/WH6lJwH1vMg">https://groups.google.com/forum/#!topic/clojure/WH6lJwH1vMg</a></p><p>I believe I have a case which is now broken because of this patch. I'm not 100% clear on the details so I might be wrong, but I can't see any other explanation for what I'm seeing. The bug I'm looking at is Cursive <a href="https://github.com/cursiveclojure/cursive/issues/748">#748</a>. Cursive calls into Leiningen in-process, and before doing that I create a new DynamicClassLoader which uses an IntelliJ PluginClassLoader as its parent. I believe that what is happening is that PluginClassLoader.loadClass() delegates to various parent classloaders then throws CNFE if it can't find the class in question. Am I correct in thinking that after this patch DCL now delegates to the parent classloader before checking its URL list, whereas previously it did not?</p><p>Colin, I opened <a href="http://dev.clojure.org/jira/browse/CLJ-1663" title="DynamicClassLoader delegates to parent classloader before checking in its URL list "><del>CLJ-1663</del></a> with a patch that should fix the issue. Can try it and comment on that ticket if it does?</p>ApprovalOkGlobal RankPatchCode and Test[CLJ-956] [java.lang.ClassFormatError: Duplicate method name&signature] when using gen-classhttp://dev.clojure.org/jira/browse/CLJ-956
Clojure<p>Hi,</p>
<p>When extending a class that defines a method with the signature "public void load();", I get the following load time error:</p>
<p>java.lang.ClassFormatError: Duplicate method name&amp;signature</p>
<p>Looking into javap, I see that in fact gen-class is generating two entries for the load method. Prefix does not help in this case, as the defined load method is generated anyway.</p>
<p>Cheers,</p>
<ul class="alternate" type="square">
<li>Daniel</li>
</ul>
$ java -version
<br/>
java version &quot;1.6.0_29&quot;
<br/>
Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
<br/>
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
<br/>
<br/>
CLJ-956[java.lang.ClassFormatError: Duplicate method name&signature] when using gen-classDefectMajorClosedDeclinedUnassignedDaniel RibeiroAOTTue, 20 Mar 2012 04:06:40 -0500Fri, 1 Mar 2013 12:47:05 -0600Fri, 9 Nov 2012 08:40:05 -0600Release 1.301<p>Hi,</p>
<p>I've create this small sample project to exemplify the bug ocurring: <a href="https://github.com/danielribeiro/ClojureBugReport">https://github.com/danielribeiro/ClojureBugReport</a></p>
<p>Cheers,</p>
<ul class="alternate" type="square">
<li>Daniel</li>
</ul>
<p>Note that the name load is not special: it works with any abstract method, no matter the name.</p><p>Not a bug: I was declaring the method load, which was already defined on the abstract class. The docs do mention this:</p>
<p><a href="http://clojuredocs.org/clojure_core/clojure.core/gen-class">http://clojuredocs.org/clojure_core/clojure.core/gen-class</a></p><p>Closed as not-a-bug.</p>Global Rank[CLJ-760] ClassNotFound when AOT compiling a self-referring deftype extended to a protocolhttp://dev.clojure.org/jira/browse/CLJ-760
Clojure<p>If I create a deftype that refers to itself in a protocol extension like below:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(ns type-test)
(defprotocol Foo
(isa-foo [x]))
(deftype TypeTest []
Foo
(isa-foo [x]
(instance? TypeTest x)))
</pre>
</div></div>
<p>And use that code via another namespace:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(ns test-type-user
(:use [type-test :only (isa-foo)])
(:import [type-test TypeTest]))
(isa-foo (TypeTest.))
</pre>
</div></div>
<p>When I try to AOT compile the test-type-user namespace with Clojure 1.2.0, I get <em>java.lang.NoClassDefFoundError: compile</em><em>stub/type-test/TypeTest (test_type_user.clj:5)</em>. Full stack trace attached. Running the same code on 1.2.1 and 1.3.0-alpha6 yielded the same exception with a slightly different error message (stacktrace for 1.2.1 is also in the attached file).</p>
<p>This came up in a test at Revelytix. We worked around this issue by not using <em>instance?</em> and instead comparing based on class name. Another workaround is to define the deftype and the extension separately (using extend-type or something similar). This problem also doesn't occur if the usage of the deftype and the definition of it are in the same namespace (i.e. if <em>type-test</em> and <em>test-type-user</em> were in the same file).</p>
Clojure 1.2.0, 1.2.1, 1.3.0-alpha6, JDK 1.6.0_24, Ubuntu 10.10CLJ-760ClassNotFound when AOT compiling a self-referring deftype extended to a protocolDefectMajorClosedNot ReproducibleUnassignedRyan SenioraotFri, 18 Mar 2011 09:06:37 -0500Fri, 18 Apr 2014 07:03:05 -0500Fri, 18 Apr 2014 07:03:05 -0500Release 1.201<p>The first case where we saw this was actually in having a deftype implement a Java interface (not a protocol) and in that case you cannot extend the interface outside the deftype (although comparing based on class name of course works). </p><p>I wonder if this could be a problem with the way the compiler does intrinsic magic for "instance?", there have been other bugs and significant changes to that part of the compiler in 2 years.</p>
<p>I am unable to reproduce this issue on clojure 1.6 or 1.5.1</p><p>may be a dup of <a href="http://dev.clojure.org/jira/browse/CLJ-698">http://dev.clojure.org/jira/browse/CLJ-698</a></p><p>NR on Clojure 1.6.0.</p>Global Rank[CLJ-322] Enhance AOT compilation process to emit classfiles only for explicitly-specified namespaceshttp://dev.clojure.org/jira/browse/CLJ-322
Clojure<p>Summary: still needs decision on implementation approach.</p>
<p>This was <a href="https://www.assembla.com/spaces/clojure-contrib/tickets/23">originally/erroneously reported</a> by Howard Lewis Ship in the clojure-contrib assembla:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">My build file specifies the namespaces to AOT compile but <span class="code-keyword">if</span> I include another namespace
(even from a JAR dependency) that is not AOT compiled, the other namespace will be compiled as well.
In my <span class="code-keyword">case</span>, I was using clojure-contrib's clojure.contrib.str-utils2 namespace, and I got a bunch of
clojure/contrib/str_utils2 classes in my output directory.
I think that the AOT compiler should NOT precompile any namespaces that are transitively reached,
only namespaces in the set specified by the command line are appropriate.
As currently coded, you will frequently find unwanted third-party dependencies in your output JARs;
further, <span class="code-keyword">if</span> multiple parties depend on the same JARs, <span class="code-keyword">this</span> could cause bloating and duplication in the
eventual runtime classpath.</pre>
</div></div>
<p>Having the option of shipping either all AOT-compiled classfiles or mixed source/AOT depending upon one's distribution requirements would make that phase of work with a clojure codebase significantly easier and less error-prone. The only question in my mind is what the default should be. We're all used to the current behaviour, but I'd guess that any nontrivial project where the form of the distributable matters (i.e. the source/AOT mix), providing as much control as possible by default makes the most sense. Given the tooling that most people are using, it's trivial (and common practice, IIUC) to provide a comprehensive list of namespaces one wishes to compile, so making that the default shouldn't be a hurdle to anyone. If an escape hatch is desired, a --transitive switch to clojure.lang.Compile could be added.</p>CLJ-322Enhance AOT compilation process to emit classfiles only for explicitly-specified namespacesEnhancementMajorIn ProgressUnresolvedUnassignedChas EmerickaotThu, 29 Apr 2010 00:20:00 -0500Thu, 8 Jan 2015 16:30:11 -06002519<p>Converted from <a href="http://www.assembla.com/spaces/clojure/tickets/322">http://www.assembla.com/spaces/clojure/tickets/322</a><br/>
Attachments:<br/>
aot-transitivity-option-compat-322.diff - <a href="https://www.assembla.com/spaces/clojure/documents/aI7Eu-HeGr35ImeJe5cbLA/download/aI7Eu-HeGr35ImeJe5cbLA">https://www.assembla.com/spaces/clojure/documents/aI7Eu-HeGr35ImeJe5cbLA/download/aI7Eu-HeGr35ImeJe5cbLA</a><br/>
aot-transitivity-option-322.diff - <a href="https://www.assembla.com/spaces/clojure/documents/aIWFiWHeGr35ImeJe5cbLA/download/aIWFiWHeGr35ImeJe5cbLA">https://www.assembla.com/spaces/clojure/documents/aIWFiWHeGr35ImeJe5cbLA/download/aIWFiWHeGr35ImeJe5cbLA</a></p><p>hlship said: I'd like to reinforce this. I've been doing research on Clojure build tools for an upcoming talk and all of them (Maven, Leiningen, Gradle) have the same problem: the AOT compile extends from the desired namespaces (such as one containing a :gen-class) to every reached namespace. This is going to cause a real ugliness when application A uses libraries B and C that both depend on library D (such as clojure-contrib) and B and C are thus both bloated with duplicate, unwanted AOT compiled classes from the library D.</p><p>cemerick said: This behaviour is an implementation detail of Clojure's AOT compilation process, and is orthogonal to any particular build tooling.</p>
<p>I am working on a patch that would provide a mechanism for such tooling to disable this default behaviour.</p><p>cemerick said: A first cut of a change to address this issue is here (<b>caution, work in progress!</b>):</p>
<p><a href="http://github.com/cemerick/clojure/commit/6f14e0790c0d283a7e44056adf1bb3f36bb16e0e">http://github.com/cemerick/clojure/commit/6f14e0790c0d283a7e44056adf1bb3f36bb16e0e</a></p>
<p>This makes available a new recognized system property, <tt>clojure.compiler.transitive</tt>, which defaults to <tt>true</tt>. When set/bound to false (i.e. <tt>-Dclojure.compiler.transitive=false</tt> when using <tt>clojure.lang.Compile</tt>), only the first loaded file (either the ns named in the call to <tt>compile</tt> or each of the namespaces named as arguments to <tt>clojure.lang.Compile</tt>) will have classfiles written to disk.</p>
<p>This means that this compilation invocation:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">java -cp &lt;your classpath&gt; -Dclojure.compiler.transitive=<span class="code-keyword">false</span> clojure.lang.Compile com.bar com.baz</pre>
</div></div>
<p>will generate classfiles only for <tt>com.bar</tt> and <tt>com.baz</tt>, but not for any of the namespaces or other files they load, require, or use.</p>
<hr />
<p>The only shortcoming of this WIP patch is that classfiles <b>are</b> still generated for proxy and gen-class classes defined outside of the explicitly-named namespaces. What I thought was a solution for this ended up breaking the loading of generated interfaces (as produced by defprotocol, etc).</p>
<p>I'll take a second look at this before the end of the week, but wanted to get this out there so as to get any comments people might have.</p><p>technomancy said: Looks good, but I'm having trouble getting it to work. I tried compiling from master of Chas's fork on github, but I still got the all the .class files generated with -Dclojure.compiler.transitive=false. It could be a quirk of the way I'm using ant to fork off processes though. Is it possible to set it using System/setProperty, or must it be given as a property on the command-line?</p><p>cemerick said: Bah, that's just bad documentation. :-/</p>
<p>The system property is only provided by <tt>clojure.lang.Compile</tt>; the value of it drives the binding of <tt>clojure.core/<b>transitive-compile</b></tt>, which has a root binding of true.</p>
<p>You should be able to configure the transitivity the same way you configure <tt><b>compile-path</b></tt> (system prop to <tt>clojure.lang.Compile</tt> or a direct binding when at the REPL, etc).</p>
<p>If not, ping me in irc or elsewhere.</p><p>meikelbrandmeyer said: I think, excluding parts 'load'ed is a little strong. I have some namespaces which load several parts from different files, but which belong to the same namespace. The most prominent example of such a case is clojure.core itself. I'm find with stopping require and use, but load is a bit too much, I'd say.</p><p>technomancy said: Chas: Thanks; will give that a go.</p>
<p>Meikel: Do people actually use load outside of clojure.core? I thought it was only used there because clojure.core is a "special" namespace where you want more vars to be available than can reasonably fit in a single file. Splitting up a namespace into several files is quite unadvisable otherwise.</p><p>technomancy said: I can confirm that this works for me modulo the proxy/gen-class issue that Chas mentioned. I would love to see this in Clojure 1.2; it would really clean up a lot of build-related issues.</p><p>meikelbrandmeyer said: I used it several times and this is the first time, I hear that it is unadvisable to do so. Even with a lower number of Vars in the namespace (c.c is here certainly exceptional) and might be of use to split several "sections" of code which belong to the same namespace but have different functionality. Whether to use a big comment in the source to indicate the section or split things into subfiles is a matter of taste. But it's a perfectly reasonable thing todo.</p>
<p>Another use case, where I use this (and c.c.lazy-xml, IIRC) is to conditionally load code depending on whether a dependency is available or not. Eg. vimclojure uses c.c.pprint and c.c.stacktrace/clj-stacktrace transparently depending on their availability.</p>
<p>There are perfectly legal uses of load. I don't see any "unadvisable" here.</p><p>cemerick said: Thanks, Meikel; I had forgotten about that use case, as I don't use load directly myself at all. I probably wouldn't say it's inadvisable, just mostly unnecessary. In any case, that's a good catch. It complicates things a bit, but we'll see what happens. I'm going to take another whack at resolving the proxy/gen-class case and narrowing the impact of nontransitivity to use and require later tonight.</p>
<p>I agree wholeheartedly that this should be in 1.2, assuming the technical bits work out. This has been an irritant for quite a long time. I actually believe that nontransitivity should be the <b>default</b> &#8211; no one wants or expects to have classfiles show up for dependencies show up when a project is AOT-compiled. I think the only negative impact would be whoever still fiddles with compilation at the REPL, and doesn't use maven or lein &#8211; and even then, it's just a matter of binding another var.</p><p>meikelbrandmeyer said: Then the var should be added to the default bindings in the clojure.main repl. Then it's set!-able like the other vars ��� <b>warn-on-reflection</b> and friends.</p><p>cemerick said: This is looking pretty good (<b>still WIP</b>):</p>
<p><a href="http://github.com/cemerick/clojure/commit/fedfb022ecef420a932b3d69c182ec7a8e5960a6">http://github.com/cemerick/clojure/commit/fedfb022ecef420a932b3d69c182ec7a8e5960a6</a></p>
<p>Thank you again for mentioning load, Meikel: it was very helpful in resolving the proxy/gen-class issue as well.</p>
<p>Just a single data point: the jar produced by the medium-sized project I've been using for testing the changes has shrunk from 1.8MB to less than 1MB. That's not the only reason this is a good change, but it's certainly a nice side-effect.</p><p>cemerick said: [<a href="file:aIWFiWHeGr35ImeJe5cbLA">file:aIWFiWHeGr35ImeJe5cbLA</a>]</p><p>cemerick said: [<a href="file:aI7Eu-HeGr35ImeJe5cbLA">file:aI7Eu-HeGr35ImeJe5cbLA</a>]</p><p>cemerick said: Patched attached. The <del>compat</del> one retains the current default behaviour <span class="error">&#91;*transitive-compile* true&#93;</span>, the other changes the default so that transitivity is a non-default option. At least of those I've spoken to about this, the latter is preferred.</p>
<p>The user impact of changing the default would be:</p>
<ol>
<li>The result of compiling from the REPL will change. Getting back current behaviour would require adding a <span class="error">&#91;*transitive-compile* true&#93;</span> binding to the existing bindings one must set when compiling from the REPL.</li>
<li>The same as #1 goes for those scripting AOT compilation via <tt>clojure.lang.Compile</tt> as well (whether by shell scripts, ant, etc).</li>
<li>Those using lein, clojure-maven-plugin, gradle, and others will likely have a new option provided by those tools, and perhaps a different default than the language's. I suspect those using such tools would much prefer a change from the default behaviour in any case.</li>
</ol>
<p>hlship said: Just had a brain-storm:</p>
<p>How about an option to support transitive compilation, but only if the Clojure source file being compiled as a file: URL (i.e., its a local file on the file system, not a file stored in a JAR). That would make it easier to use compilation on the local project without transitively compiling imported libraries, such as clojure-contrib.</p>
<p>So <b>transitive-compile</b> should be a keyword, not a boolean, with values :all (for 1.1 behavior), :none (to compile only the exact specified namespaces) or :local (to compile transitively, but only for local files, not source files from JARs).</p><p>cemerick said: (Crossposted to the clojure-dev list)</p>
<p>I thought about this some, and I don't think that's a good idea, at least for now. I'm uncomfortable with semantics changing depending upon where code is being loaded from &#8211; which, depending upon a tool's implementation, might be undefined. E.g. if the com.foo.bar ns is available in source form in one directory, but as classes from a jar, and classpaths aren't being constructed in a stable fashion, then the results of compilation will change.</p>
<p>If we decide that special treatment depending upon the source of code is warranted in the future, that's a fairly straightforward thing to do w.r.t. the API &#8211; we could have :all and :local as you suggest, with nil representing :none.</p><p>stu said: Rich is not comfortable enough with the implementation complexity of this patch (e.g. the guard clause for proxies and gen-class) to slide this in as a minor fix under the wire for 1.2. </p>
<p>Better to live with the pain we know a little longer than ship something we don't have enough experience with to be confident.</p><p>Updated patch to cleanly apply to HEAD and address issues raised by screening done by <a href="http://groups.google.com/group/clojure-dev/msg/0771729b72e04c9e">Cosmin Stejerean</a>. Also includes proper tests.</p>
<p><b>Note: this patch's tests require the fix for</b> <a href="http://dev.clojure.org/jira/browse/CLJ-432" title="deftype does not work if containing ns contains dashes"><del>CLJ-432</del></a>!</p><p>the "-resolved" patch resolves a conflict in main.clj</p><p>Several questions:</p>
<ol>
<li>I am getting an ant build error: "/Users/stuart/repos/clojure/build.xml:137: java.io.FileNotFoundException: Could not locate clojure/test_clojure/aot/nontransitive__init.class or clojure/test_clojure/aot/nontransitive.clj on classpath:"</li>
<li>It feels icky to have a method named writeClassFile that, under some circumstances, does <b>not</b> write a class file, but instead loads it via a dynamic loader. Maybe this is just a naming issue.</li>
<li>Are there any other ways to accomplish the goals of <tt><b>load-level</b></tt>? Or, taking the other side, if we are going to have a <b>load-level</b>, are there other possible consumers who might have different needs?</li>
<li>(Minor) Why the <tt>use :only</tt> idiom instead of just <tt>require</tt>?</li>
</ol>
<p>An alternative approach: patch write-classes-1.diff.gz</p>
<p>From <a href="https://github.com/stuartsierra/clojure/tree/write-classes">my forked branch</a></p>
<p>What this patch does:</p>
<ul>
<li>Keeps 'compile' and '<b>compile-files</b>' exactly the same</li>
<li>Adds 'compile-write-classes' to write .class files for specifically named classes</li>
<li>Minor compiler changes to support this</li>
</ul>
<p>This approach was prompted by the following observations:</p>
<ul>
<li>Java interop is the dominant reason for needing .class files</li>
<li>Things other than namespaces can generate classes for Java interop:
<ul>
<li>deftype/defrecord</li>
<li>defprotocol</li>
<li>gen-class/gen-interface</li>
</ul>
</li>
<li>For library releases, we want to control which .class files are emitted on a per-class basis, not per-namespace</li>
<li>Some legitimate uses of AOT compilation will want transitive compilation
<ul>
<li>Pre-compiling an entire application before release</li>
</ul>
</li>
</ul>
<p>S. Halloway: My apologies, I didn't know you had commented. I thought that, having assigned this issue to myself, I'd be notified of updates. <img class="emoticon" src="http://dev.clojure.org/jira/images/icons/emoticons/sad.gif" height="20" width="20" align="absmiddle" alt="" border="0"/></p>
<p>FWIW, I aim to review your comments and SS' approach over the weekend.</p><p>S. Halloway:</p>
<p>1. Certainly shouldn't happen. AFAIK, others have screened the patch, presumably with a successful build.<br/>
2. Agreed; given the approach, I think it's just a bad name.<br/>
3. Yes, I think S. Sierra's is one. See my next comment.<br/>
4. Because the <tt>:use</tt> form was already there. I've actually been using that form of <tt>:use</tt> more and more; I've found that easier than occasionally having to shuffle around specs between <tt>:use</tt> and <tt>:require</tt>. I think I'm aping Chris Houser in that regard.</p><p>I think S. Sierra's approach is fundamentally superior what I offered. I have two suggestions: one slight perspective change (which implies no change in the actual implementation), and an idea for an even simpler approach (at least from a user perspective), in that order.</p>
<p>While interop is the driving requirement behind AOT, I absolutely do not want to have to keep an updated enumeration of all of the classes resulting from each and every <tt>defrecord</tt> et al. usages in my <tt>pom.xml</tt>/<tt>project.clj</tt> (and I wouldn't wish the task of ferreting those usages and their resulting classnames on any build tool author).</p>
<p>Right now, <tt>&#42;compile-write-classes&#42;</tt> is documented to be a set of classname strings, but could just as easily be any other function. <tt>&#42;compile-write-classes&#42;</tt> should be documented to accept any predicate function (renamed to e.g. <tt>&#42;compile-write-class?&#42;</tt>?). There's no reason why it shouldn't be bound to, e.g. <tt>#(re-matches #"foo\.bar\.[\w&#95;]+$" %)</tt> if I know that all my records are defined in the <tt>foo.bar</tt> namespace.</p>
<p>To go along with that, I think some package/classname-globbing utilities along with corresponding options to <tt>clojure.lang.Compile</tt> would be most welcome. Classname munging rules are not exactly obvious, and it'd be good to make things a little easier for users in this regard.</p>
<hr />
<h5><a name="Anotheralternative"></a>Another alternative</h5>
<p>If there's a closed set of forms that generate classes that one might reasonably be interested in having in a build result (outside of use cases for pervasive AOT), then why not have a simple option that only those forms utilize? <tt>gen-class</tt> and <tt>gen-interface</tt> already do this, but reusing the all-or-nothing <tt>&#42;compile-files&#42;</tt> binding; if they keyed off of a binding that implied a diminished scope (e.g. <tt>&#42;compile-interop-forms&#42;</tt> – which would be <tt>true</tt> if <tt>&#42;compile-files&#42;</tt> were <tt>true</tt>), then they'd do exactly what we wanted. Extending this approach to <tt>deftype</tt> (and therefore <tt>defrecord</tt>) <em>should</em> be straightforward.</p>
<p>An implementation of this would probably be somewhat more complicated than S. Sierra's patch, though not as complex as my original stab at the problem (i.e. no <tt>&#42;load-level&#42;</tt>). On the plus side:</p>
<p>1. No additional configuration for users or implementation work for build tool authors, aside from the addition of the boolean diminished-scope AOT option<br/>
2. Class file generation would remain opaque from a build process standpoint<br/>
3. Future/other class-generating forms (there are a few people futzing with ASM independently, etc) can make local decisions about whether or not to participate in interop-centric classfile generation. This might be particularly helpful if a given form emits multiple classes, making the determination of a classname-based filter fn less straightforward.</p>
<p>I can see wanting to further restrict AOT to specific classnames in certain circumstances, in which case the above and S. Sierra's patch might be complimentary.</p><p>I like the idea of <tt>&#42;compile-interop-forms&#42;</tt>. But is it always possible to determine what an "interop form" is? I <em>think</em> it is, I'm just not sure.</p><p>I'm also in favor of compile-interop-forms. As far as determining, how about sticking metadata on the var?</p>
<p>(defmacro ^{:interop-form true} deftype ...)</p><p>Summary and design discussion on wiki at <a href="http://dev.clojure.org/display/design/Transitive+AOT+Compilation">http://dev.clojure.org/display/design/Transitive+AOT+Compilation</a></p><p>New attachment <tt>compile-interop-1.patch</tt> has new approach: Add a third possible value for <tt>&#42;compile-files&#42;</tt>. True and false keep their original meanings, but <tt>:interop</tt> causes <b>only</b> interop-related forms to be written out as .class files. "Interop forms" are gen-class, gen-interface, deftype, defrecord, defprotocol, and definterface.</p>
<p>Pros:</p>
<ul>
<li>doesn't change existing behavior</li>
<li>handles common case for non-transitive AOT (interop)</li>
<li>minimal changes to the compiler</li>
</ul>
<p>Cons:</p>
<ul>
<li>not flexible</li>
</ul>
<p>Just realized my patch doesn't solve the transitive compilation problem. If library A loads library B, then compiling interop forms in A will also emit interop .class files in B.</p><p>It's disappointing to see an important issue like this still unresolved after 2.5 years. This is a real pain for us. We have a large closed source project where shipping source is not an option. This forces us to manage the AOT'ing of dependencies due to the hard dependency on protocol interfaces introduced by transitive AOT compilation (see <a href="https://groups.google.com/forum/?fromgroups=#!topic/clojure-dev/r3A1JOIiwVU">https://groups.google.com/forum/?fromgroups=#!topic/clojure-dev/r3A1JOIiwVU</a>).</p><p>Paul, do you have a suggestion for which of the approaches described in comments here, or on the wiki page <a href="http://dev.clojure.org/display/design/Transitive+AOT+Compilation">http://dev.clojure.org/display/design/Transitive+AOT+Compilation</a> would be preferable solution for you? Or perhaps even a patch that implements your preferred approach?</p><p>Andy,</p>
<p>I'm now consulting with Paudi's organization, so I think I can speak for him (I'm now the default buildmeister).</p>
<p>I like Stuart's :interop idea, but that is somewhat orthogonal to our needs.</p>
<p>I return to what I would like; compilation would compile specific namespaces; dependencies of those namespaces would not be compiled.</p>
<p>To be honest, I'm still a little hung up on the interop forms: especially defprotocol and friends; from a cursory glance, it appears that todays AOT compilation will compile the protocol into a Java class, then compile the namespace that references the protocol with the assumption that the protocol's Java class is available. When we use build rules to only package our namespace's class files into the output JAR, the code fails with a NoClassDefFoundError because the protocol really needs to be recompiled, at runtime compilation, into an in-memory Java class.</p>
<p>Obviously, supporting this correctly will be a challenge; the compiled bytecode for our namespace would ideally:<br/>
1) check to see if the Java class already exists and use it if so<br/>
2) load the necessary namespaces so as to force the creation of the Java class</p>
<p>I can imagine any number of ways to juggle things to make this work, so I won't suggest a specific implementation.</p>
<p>In the meantime, our workaround is to create a "stub" module as part of our build; it simply requires in the necessary namespaces (for example, org.clojure:core.cache); this forces an AOT compile of the dependencies and we have a special rule to package such dependencies in the stub module's output JAR. This may not be a scalable problem, and it is expensive to identify what 3rd party dependencies require this treatment.</p><p>I am marking this incomplete because there does not yet seem to be plurality, much less consensus or unanimity, on approach.</p>
<p>Personally I am in favor of a solution based on a predicate that gets handed the class name and compiled bits, and then can choose whether to write the class. Pretty close to Stuart Sierra's compile-write-classes. Might be possible to flow more information than classname to the predicate.</p>
<p>Removed the 1.6 release from this and added to Release.Next list to make this a priority for the next release.</p><p>Howard,</p>
<p>I don't exactly understand your write up, I am reading the compiler, the emit-protocol macro, and emit-method-builder to try and understand it.</p>
<p>You might check to see if you have a situation similar to the following:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(ns a.b)
(defprotocol P1
(pm [a]))
</pre>
</div></div>
<p>then either</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(ns a.c
(:import (a.b P1))
(defrecord R []
P1
(pm [x] x))
</pre>
</div></div>
<p>or</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(ns a.c)
(defrecord R []
a.b.P1
(pm [x] x))
</pre>
</div></div>
<p>in both examples defrecord is actually getting the class behind the protocol instead of the protocol, the correct thing to do is</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre>(ns a.c
(:require [a.b :refer [P1]]))
(defrecord R []
P1
(pm [x] x))
</pre>
</div></div>
<p>This is an <b>extremely</b> common mistake people make when using protocols, unfortunately the flexibility of using interfaces directly in defrecord forms, and protocols being backed by interfaces means it is very easy to unwittingly make such a mistake. Both of the mistake examples could result in missing classes/namespace problems.</p><p>The write-classes patch still applies cleanly, I just tried it out, made a one-line change to lein-ring and got a war-file for my app with two generated classes instead of hundreds. This seems like a good fix to me. What do we need to do to get it into release?</p><p>As Stu said above, we do not have consensus on whether this is the right way to go with this. I am planning to look at all outstanding AOT tickets (including this one) early in 1.8.</p>ApprovalVettedGlobal RankPatchCode and TestWaiting Oncemerick[CLJ-130] Namespace metadata lost in AOT compilehttp://dev.clojure.org/jira/browse/CLJ-130
Clojure<p>AOT-compilation drops namespace metadata.</p>
<p>This also affects all of the namespaces packaged with Clojure, except <tt>clojure.core</tt>, for which metadata is <a href="https://github.com/clojure/clojure/blob/229bf8fe9a751e4f48bb2b7ea57e27ebc43d26ae/src/clj/clojure/core.clj#L6158">explicitly added</a> in core.clj.</p>
<p><b>Cause of the bug:</b></p>
<ul class="alternate" type="square">
<li>a namespace inherits the metadata of the symbol used to create that namespace the first time</li>
<li>the namespace is created in the load() method, that is invoked <b>after</b> the __init() method</li>
<li>the __init0() method creates all the Vars of the namespace</li>
<li>interning a Var in a namespace that doesn't exist forces that namespace to be created</li>
</ul>
<p>This means that the namespace will have been already created (with nil metadata) by the time the load() method gets invoked and thus the call to in-ns will be a no-op and the metadata will be lost.</p>
<p><b>Approach:</b> The attached patch fixes this issue by explicitely attaching the metadata to the namespace after its creation (via ns) using a .resetMeta call<br/>
<b>Patch:</b> 0001-<a href="http://dev.clojure.org/jira/browse/CLJ-130" title="Namespace metadata lost in AOT compile"><del>CLJ-130</del></a>-preserve-metadata-for-AOT-compiled-namespace.patch<br/>
<b>Screened by:</b></p>CLJ-130Namespace metadata lost in AOT compileDefectMajorClosedCompletedUnassignedStuart SierraaotmetadataFri, 19 Jun 2009 00:47:00 -0500Fri, 31 Jul 2015 15:39:55 -0500Fri, 31 Jul 2015 15:39:55 -0500Release 1.855<p>Converted from <a href="http://www.assembla.com/spaces/clojure/tickets/130">http://www.assembla.com/spaces/clojure/tickets/130</a></p><p>richhickey said: Updating tickets (#127, #128, #129, #130)</p><p>juergenhoetzel said: This is still a issue on </p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">Clojure 1.2.0-master-SNAPSHOT</pre>
</div></div>
<p>Any progress, hints? I prefer interactive documentiation via slime/repl</p><p>This is of great concern to me, as the Rook web services framework we're building depends on availability of namespace metadata at runtime.</p><p>BTW, I verified that this still exists in 1.6.0.</p><p>For me personally, I would raise the priority of this issue. And I think in general, anything that works differently with AOT vs. non-AOT should be major, if not blocker, priority.</p><p>Alex Miller:</p>
<blockquote><p>@hlship I think the question is where it would go. note no one has suggested a solution in last 5 yrs.</p></blockquote>
<p>Alas, I have not delved into the AOT compilation code (since, you know, I value my sanity). But it seems to me like the __init class for the namespace could construct the map and update the Namespace object.</p><p>Just playing with javap, I can see that the meta data is being assembled in some way, so it's a question of why it is not accessible ...</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre> public static void __init0();
Code:
0: ldc #108 // String clojure.core
2: ldc #110 // String in-ns
4: invokestatic #116 // Method clojure/lang/RT.var:(Ljava/lang/String;Ljava/lang/String;)Lclojure/lang/Var;
7: checkcast #12 // class clojure/lang/Var
10: putstatic #10 // Field const__0:Lclojure/lang/Var;
13: aconst_null
14: ldc #118 // String fan.auth
16: invokestatic #122 // Method clojure/lang/Symbol.intern:(Ljava/lang/String;Ljava/lang/String;)Lclojure/lang/Symbol;
19: checkcast #124 // class clojure/lang/IObj
22: iconst_4
23: anewarray #4 // class java/lang/Object
26: dup
27: iconst_0
28: aconst_null
29: ldc #126 // String meta-foo
31: invokestatic #130 // Method clojure/lang/RT.keyword:(Ljava/lang/String;Ljava/lang/String;)Lclojure/lang/Keyword;
34: aastore
35: dup
36: iconst_1
37: aconst_null
38: ldc #132 // String meta-bar
40: invokestatic #130 // Method clojure/lang/RT.keyword:(Ljava/lang/String;Ljava/lang/String;)Lclojure/lang/Keyword;
43: aastore
44: dup
45: iconst_2
46: aconst_null
47: ldc #134 // String doc
49: invokestatic #130 // Method clojure/lang/RT.keyword:(Ljava/lang/String;Ljava/lang/String;)Lclojure/lang/Keyword;
52: aastore
53: dup
54: iconst_3
55: ldc #136 // String Defines the resources for the authentication service.
57: aastore
58: invokestatic #140 // Method clojure/lang/RT.map:([Ljava/lang/Object;)Lclojure/lang/IPersistentMap;
61: checkcast #64 // class clojure/lang/IPersistentMap
64: invokeinterface #144, 2 // InterfaceMethod clojure/lang/IObj.withMeta:(Lclojure/lang/IPersistentMap;)Lclojure/lang/IObj;
</pre>
</div></div>
<p>If I'm reading the code correctly, a Symbol named after the namespace is interned, and the meta-data for the namespace is applied to the symbol, so it's just a question of commuting that meta data to the Namespace object. I must be missing something.</p><p>Attached patch fixes this issue by explicitely attaching the metadata to the namespace after its creation using a .resetMeta call.</p>
<p>Here's an explaination of why this bug happens:</p>
<ul class="alternate" type="square">
<li>a namespace inherits the metadata of the symbol used to create that namespace the first time</li>
<li>the namespace is created in the load() method, that is invoked <b>after</b> the __init() method</li>
<li>the __init0() method creates all the Vars of the namespace</li>
<li>interning a Var in a namespace that doesn't exist forces that namespace to be created</li>
</ul>
<p>This means that the namespace will have been already created (with nil metadata) by the time the load() method gets invoked and thus the call to in-ns will be a no-op and the metadata will be lost.</p>
<p>Added new test script that works with 1.8.0-master. Also, added a script that will enable direct linkage before testing.</p><p>Tested with the latest 1.8.0-master source and ran modified versions of the test script to ensure that the patch worked with and without direct linking.</p>ApprovalOkGlobal RankPatchCode