trptcolin said: This patch adds support and tests for both docstrings and metadata, and also fixes the regex in the existing test to correctly identify errors. The current regex has unescaped characters: | ( ) that were almost certainly typos, as matching (for instance) only the last line of output would pass the test. The docstring/metadata support is a simplified version of what's in the source of defn.

Currently using clojure.contrib.condition is awkward if you need to handle both Conditions and Exceptions. This patch by Steve Gilardi and myself allows you to include catch clauses in your handle-case forms for regular Exceptions.

Root cause is in clojure.contrib.core/defnilsafe, so similar functions like .?. will have the same problem. I futzed around for a while to try and figure it out, but defnilsafe is a little too meta for me. The problem is, predictably enough once you've glanced at the code, with ~'~nil-safe-name.

The handler-case macro detects forms of the form "(handle ...)" by checking for items in the body for which list? returns true that begin with the symbol handle. code can be built from seqs that are not lists (e.g., clojure.lang.Cons's or lazy-seqs). The proper predicate is "seq?" not "list?".

(require 'clojure.contrib.io)
(clojure.contrib.io/append-writer "/xyz.txt")
</code></pre>
which will throw an Exception ���Cannot change an open stream to append mode���.
This is caused because of a flaw in the call chain.
*clojure.contrib.io/writer* for Strings calls again *writer* for whatever *clojure.contrib.io/output-stream*
returns. We call *append-writer* which then calls *writer* on the String which in turn then calls:
<pre><code>
output-stream for ^String which calls
output-stream for ^URL which calls
output-stream for ^File which binds *append* to false and calls
output-stream for ^OutputStream which
returns a BufferedOutputStream instance which then jumps back to the
^File handler, which removes its binding for *append*!! and returns the BOS to the
^URL handler which returns the BOS to the
^String handler

At this point the outer binding (for append) is set to true again, and assert-not-appending fails.

I'm finding the control flow in c.c.io rather difficult to follow, what with all those extensions of the Streams protocol delegating to one-another. I'm reasonably certain that the solution is to special case the creation of a reader in append mode.

I'll attach a patch that does this by providing a default implementation for :writer, which seems to solve the problem, but I think I still need to think about it some more before I can call it 'done'.

bpsm said: Correction: This issue is notpresent in clojure 1.2.0. my bad.
I was mistaken about this bug being present in clojure.java.io. append-writer doesn't even exist there and the append magic is gone there too as per clojure#311.
I think there was just something wonky with my swank-clojure:

Since 1.2.0 is at RC3 already, you must mean 1.3.0 when you say "next release", yes/no? Predicated on that:

I think this should be fixed on the 1.2.x branch, where c.c.io is available, though deprecated. ("deprecated, but working" is better than "deprecated by slightly buggy.") The attached patch is against 1.2.x, not master, despite what my initial comment on this issue says.

on clojure-contrib master:
gen-html-docs :uses c.c.def without actually using anything from it.
This causes the maven build to fail because clojure.contrib:def is not listed as a dependency in gen-html-docs' pom.xml.

In clojure.contrib.command-line, you could specify a 'catch-all' argument at the end of your spec and all arguments after the ones that were parsed would be collected in. In tools.cli, nothing like this seems to exist.

For example, with tools.cli, there is no way to handle stuff like this: "myapp --foo bar --baz foo a b c". The last three arguments, a, b, and c will all be discarded with no way to retrieve them.

Finally, if this wasn't the right place to put this issue, I apologize. tools.cli doesn't seem to be listed as a project and you guys don't accept issues on Github, so I wasn't sure what else to do.

The problem is that c.c.sql.internal calls `.println` on a "stream" to print update counts and errors. That method only exists on a `PrintWriter`. This isn't true within Slime, where it's a `StringWriter`.

The attached patch binds `out` and uses `println` instead, which does the right thing. I've augmented the docstrings to request a `Writer` as the value of `stream`, which aligns with the docstring for `print`.

scgilardi said: Following the discussion in the developer group and as suggested by Stuart Halloway, I'm planning to make these changes:

[1] adopt the substance of Richard Newman's patch, changes to sql.internal's print-sql-exception, print-sql-exception-chain, print-update-counts functions[2] remove the automatic printing to err on exceptions. It will be up to the caller to handle the exceptions as he desires: eating, logging, or printing, etc.[3] move the print* functions to c.c.sql proper as they will now be "public".

Was this discussed on the group? If so, please provide a link to the discussion[13:51] <na_ka_na> Hey guys I think there's a bug in clojure.contrib.json at line 116[13:51] <na_ka_na> in clojure 1.2[13:52] <na_ka_na> it has a (char i) and then checks for (= i -1) ... but (char -1) fails[13:52] <na_ka_na> where can I report it ?[13:53] <LauJensen> na_ka_na: assembla/support

The new version breaks clojure.contrib.shell/sh's :out :bytes mode (mapping byte over values returned by InputStream.read in shell.clj, line 125).
Perhaps have sh use #(.byteValue %) instead of byte? (untested)

I ran into the situation where reflection in clojure.contrib.condition was taking up much more time than I would have liked it too. I added the needed type hints to avoid reflection and the changes have solved the problem for me.

I've attached the patch. It is also available on github if you prefer:

Current version of JSON output functions perform escaping of non-ASCII characters into \uXXXX sequences, although json allows to use any Unicode character in strings. When working with non-Latin-1 based texts, this lead to additional traffic.

Given patch fixes this problem, leaving old behavior by default, but adding new option to output functions, that now accept :escape-chars option, that control how non-ASCII characters in string will output. When this option has value 'true' (default), then hex sequences are used for these characters, and when 'false', then characters are output as-is

For other configuration options (keywordizing object fields) I used local arguments, assuming it would be faster than Var lookups. With recent enhancements to Var lookup, that may no longer be true. Need to investigate.

Clojurescript is wildly outdated. To use it, it would be best to use contrib and clojure from approximately Clojure 1.0. Thanks for patch, but I think forcing Clojurescript to use a recent version of Clojure would actually complicate the situation for anyone desperate enough to try to use it.

cemerick said: Based on discussion on the clojure-dev list, I've eliminated the buffer-streams var and the associated usage. We can revisit buffering control strategies / APIs at some future point without any API incompatibilities.

When read-json encounters an end-of-file, an IllegalArgumentException is thrown regardless of the user's eof preferences. This can easily be reproduced by running:

(read-json "")
</code></pre>
which throws the exception
<pre><code>java.lang.IllegalArgumentException: Value out of range forchar: -1
</code></pre>
This is caused by the .read method of the Reader returning -1 when the end is reached. (.read returns an int.) This value is then passed to 'char', which tries to make the corresponding character from the code point number. However, -1 is not a valid code point, so it throws an exception.
A solution is to check for equality to -1 before trying to convert to a character. I also have an idea for a helper function that could be used:
<pre><code>(defn- read-char"Reads a character from a Reader. Returns nil if the end of the stream is reached"
[stream]
(let [x (.read stream)]
(if (not= x -1)
(char x))))

This function will always return either a Character or nil. Using it would make it possible to have the eof case along the other cases in the existing cond forms.

I know commons-codec contains a quality implementation of both functions, but adding yet another jar in your build just for base64 decode purposes is an unnecessary complication. For symmetry's sake, decode should be added.

Too many people are doing crazy stuff with conditional loading just to support both 1.1 and 1.2 simultaneously. This way lies madness.

To avoid this, lets forward-port all the 1.1 namespaces that were removed or renamed in master, e.g. duck-streams, seq-utils, str-utils, etc. Just copy the old files from the 1.1 branch, so that no behavior changes.

Add a printed deprecation warning at the top-level of each of the old namespaces, like
(println "The foo namespace is deprecated, please use bar instead.")

cemerick said: FWIW, the various signature and fn name changes between 1.1 and 1.2 still need to be unraveled by any codebase that wants to target both, and choosing which impl to use is still easiest by eagerly loading whereever the new impls are in a try/catch. I'm not arguing against restoring the old namespaces (at the moment! , just pointing out that doing so doesn't impact 1.1/1.2 compat.

bpsm said: I'll submit further patches as I have time. Not being "in the loop", I can't be certain that we really want all the namespaces marked as deprecated gone before 1.3, but that's my operating assumption. (I'm attaching one patch per namespace so that the comitters can be selective.)

bpsm said: Proposal: when a namespace is partially promoted to Clojure proper, it might be good to provide a reduced version of the old namespace, providing just the functionality that was not promoted as an alternative to complete removal.

clojure-contrib 1.2 deprecates, but does not remove clojure.contrib.io

clojure-contrib 1.3 provides a reduced version of clojure.contrib.io containing only non-deprecatd functionality not available elsewhere (i.e. in Clojure ��� 1.2).

This new clojure.contrib.io is not a strict subset of the old clojure.contrib.io, as some changes seemed sensible to harmonize with clojure.java.io's conventions. This means, it's not a drop-in replacement, but at least provides equivalents for anyone trying to get off the deprecated clojure.contrib.io.

One problem that's apparent with this proposal is that, since the 'new' io only becomes available in 1.3, there would be something of a "flag day" when switching from 1.2 to 1.3, but then there would be if 1.3 completely eliminated c.c.io as well – only it would be worse.

Those moving their code off of c.c.io to c.j.io will just have to write their own versions of a subset of {file-str, write-lines, read-lines, pwd, with-out, with-in, to-byte-array, relative-path-string and delete-file-recursively}, or get these from a library outside of clojure-contrib. Am I understanding correctly?

Patch 0010 contains (in the comments) some tips for porting from deprecated c.c.io to c.j.io. It would make sense to gather that kind of information in some kind of a guide/faq about how to replace uses of deprecated c.c. namespaces promotion to core was not 1-to-1.

stuart.sierra said: I don't think this was caused by changes in the build. Something has changed about how arguments get passed to init functions. The ColumnWriter init function is getting the same arguments as the PrettyWriter init function (a Writer, 72, and 40).

puzzler said: I realized shortly after uploading the file that although I made the equality of priority maps behave like regular maps and sorted maps, I forgot to adjust the hashCode method accordingly.

If you have a query that results in a very large dataset you will run out of memory because by default, with-query-results pulls the entire result set into memory (even though it creates a lazy-seq using resultset-seq).

The fix is simple, just call (.setFetchSize stmt 1). But, with-query-results doesn't give you a way to do that currently. I propose adding an optional parameter after sql-params that is a hash of attributes to set on the PreparedStatement before executing it.

Any progress on this? Another option instead of passing a map of attributes and using bean would be to allow the user to pass an optional function that would be called on the statement object. I've seen code like this in compojure, I think:

stu said: The problem isn't lazy sequences, but the absence of any data to report on. Commit 6dd033d9e12337f6630faa3d3f5c2e901a28c4f4 prevents the exception, and causes the printed report to be empty in the case there is nothing to report.

On the calling side, the examples shown above aren't profiling anything, because the lazy calls are never evaluated. You should use some kind of "do" form instead.

cemerick said: I mentioned this elsewhere, but perhaps there's a reason why this isn't workable in this case: I believe if you require clojure.contrib.pprint.ColumnWriter in PrettyWriter's ns declaration (perhaps before the :gen-class form?), then the compiler will visit and gen CW before attempting to gen-class PW.

This would eliminate the need to explicitly specify which namespaces to compile.

Currently: When duck-streams is given a string, it attempts to open it for reading as a URL and if that fails it tries the string as a path specifying a File. When the string doesn't specify a readable file either, we fail.

I'd like to see duck-streams try to resolve the string as a resource name before finally giving up.

ataggart said: The only valid levels are the ones specified in the documentation, those levels in common use across multiple logging implementations. If one uses an invalid value one should not expect it to work.

The "correct" solution would be to provide a better exception in this error case.

rnewman said: "If one uses an invalid value one should not expect it to work"

No, but one should expect something more useful than a NPE. The patch I attached throws an exception when an invalid value is used (indeed, at compile time if it's a constant keyword).

c.c.l already translates from one set of log levels to those in use by the underlying implementation (e.g., if you're using Java Logging it'll convert :debug to FINE). This patch ensures that you can use the other set of names, too, which seems perfectly reasonable, but I can understand if you disagree. In that case, simply change translate-log-level to something like

In clojure.contrib.duck-streams append-spit writes out encoding
markers (for UnicodeLittle for example this is a FEFF in hex)
each time it appends to a file. This should happen only when
the file is initially created.

stu said: I am not sure there is a good answer here. The code above chooses an encoding with an explicit marker, and gets what it asks for.

One proposed solution (http://github.com/sergey-miryanov/clojure-contrib/commits/bug-30) tries to detect this scenario, and recover via a hard-coded mapping between encodings-with-markers and similar-encodings-without. But I don't think this can work in general, because the set of possible encodings is open and the Charset API doesn't provide a mapping between the with-markers and without-markers versions.

Sorry, and please feel free to reopen this if I am missing an obvious approach.

I've cleaned up the documentation and got rid of the old assert-args code from c.c.mock. The attached patches contain the two commits I made on my repository. I'm new to patches and git so let me know if I missed anything obvious!

(load "clojure/contrib/load_all") has not worked since the penultimate commit to clojure-contrib, as libraries are being moved to core. all-contrib-libs needs to be adjusted to match the new set of libraries.

scgilardi said: In case somebody likes this idea gets to this before I do: I think it would be cool for load-all to figure out what all the clojure-contrib libraries are on its own: perhaps by searching in the resources in the clojure-contrib Jar file for ns declarations that are exactly "clojure.contrib.[^.]+$" (only one component after clojure.contrib).

We could then maintain a list in the load-all vicinity of only those libraries that should not be loaded by load-all. That seems much easier to maintain as we'll automatically be prompted (by a failure to load) if we forget an exclusion.

I recall now why I didn't have this as an option already. with-connection retrieves an open connection based on the info in db-spec, allows body to use it, then closes it. If we have the connection object in the db-spec object itself, it will only be good for one use.

How about this as an alternative to your patch (in addition to the other methods that will remain): Allow the user to provide a function (:factory) whose argument is a map of the remaining keys and values in the db-spec object. To retrieve a new, open connection, with-connection would call the factory object supplying a map of its params.

I think this will provide enough of an escape hatch for any source of open database connections to be used while still preserving db-spec as an immutable object that specifies how to retrieve them.

gmfawcett said: Just to clarify my last comment: I understand your comment about not wanting a possibly-closed connection object as a value in the map. In my test code, the :external worked just fine – in spite of the "closed" object in the map, I could re-use it without any trouble. Proxool must be doing some magic to make this work. But this is bad form.

+1 for your :factory patch, and I'm still in favour of :factory-close if you would accept a patch for (with-connection*) as well.

scgilardi said: I like the uniformity of using with-open on all the connections. I'd like to save :factory-close until we have a usecase where it's needed. Could you please test the :factory patch in your setup? If you find that it works well for you, I'd like to commit it. Thanks!

Unfortunately there are still some issues with rounding. I've noticed that when the
mantissa is rounded up from 9 to 10 the exponent is not changed
accordingly. As a result, we get for example
clojure.contrib.pprint> (cl-format nil "~,0F" 9.4) => "9."
clojure.contrib.pprint> (cl-format nil "~,0F" 9.5) => "1.0"

I've compared the output from Clojure and Common Lisp for a handful of
cases. Below we see how "0.10" is returned instead of "1.0" when
printing 0.99 with a single digit after the decimal point. There is
also a minor problem with "01." being printed instead of "1.", but
returning a different representation for a number is of course not as
bad as returning a wrong representation.

The branch itself breaks the change down into a sequence of (hopefully) logical steps, so I encourage those interested to have a look at it.

Features

input-stream is like reader, except for bytes.

output-stream is like writer, except for bytes.

reader and writer now build on input-stream and output-stream where appropriate.

Compatibility

I've renamed append-to-writer because it applies not just to writers, but also to output-streams. This might be a compatibility issue for clients that did not heed the advice given in documentation to use append-writer or append-spit instead of directly binding this var.

I'd like an easy way to consume a text file written in Clojure syntax as pure data: as a sequence of Clojure data structures without evaluation. This allows Clojure to be used where property files, XML or JSON might otherwise be chosen.

bpsm said: Well, more-or-less. The intent of my patch was to allow a file containing multiple forms to be read lazily (i.e. one form at a time). The (read-string (slurp* x)) solution is eager.

That said, I'd like to withdraw the patch because I'm not really happy with the fact that the implementation fails to close the underlying file if the sequence is not consumed fully. (This is a more general problem with lazy sequences based on resources not managed by the GC.)

We shouldn't AOT logging.clj since this causes the log implementation to be determined at compile-time. This is almost always the wrong thing; it should be determined at load-time since it's determined by what classes are on the classpath at runtime.

I've encountered a couple of issues with the cl-format function
included in contrib.pprint

(cl-format nil "~1,1$" -12.0) ;; => "12.0" the sign is lost

I think the problem is the following assignment in the dollar-float
function
add-sign (and (:at params) (not (neg? arg))) ;; wrong
(the sign is only printed when the colon modifier is present and only
for positive numbers)
that should read, if I understand correctly the logic,
add-sign (or (:at params) (neg? arg))

The second issue is not so straightforward to solve:

(cl-format true "1,1$%" 0.001) ;; => String index out of range: -1

I've tracked down the bug into the function round-str (the variable
round-pos will be negative and this case is not handled properly), but
I don't understand the code well enough to propose a fix.

clojure.contrib.str-utils2 contains several functions with the same names as clojure.core functions: take, drop, butlast. However, their argument order is reversed from the order used by the clojure.core functions.

One of the stated design goals of str-utils2 is to have all functions take the string they are operating on as their first argument, to enable easy composition with ->

Either these functions should be renamed, as "str-take", or a note should be added to their documentation explaining the argument order.

Patch addresses remaining items to be converted to the new string lib:
��� "str-join" to "join" in c.c.command-line
��� "re-partition" to "partition" in c.c.repl-utils
��� correction to the example function call in c.c.string/partition's docs (reversed order of arguments)

As discussed in this thread on the main Clojure list, I'm submitting the let-kw keyword argument processing macro. Please see the attached patch (against the master branch). This is the version from this message, which uses symbols for binding, like the normal let form (rather than keywords). The attached patch has a few tests which also provide decent usage examples.

tomfaulhaber said: Sorry, Brendan I didn't see your patch before I did almost exactly the same thing. I went ahead and pushed my stuff instead since I don't think that yours would have passed test_contrib (the pprint test you modified would have failed in your version, I just took it out in my version). Let me know if I missed anything.

By adding print statements to the various 'discovery' macros I could confirm that a logging framework was being found. However, when my app (servlet in this case) actually executed a log statement, I got an error that clojure.contrib.logging/agent-log was undefined.

This confused me, because the presence of any send-log form should cause it to be defined in the macroexpansion. My code simply called (log :info "some string"), which eventually expands into (amongst other things) a send-log form.

Simply lifting the function definition out of the macro made the problem go away

chouser@n01se.net said: Did you try test-delete-file-recursively on Windows? Looks like it might have issues there. I'll apply the 0001 and 0002 patches, but please take a look at 0003 to see if there's a way to be sure it won't cause false negatives.

hlship said: The patch addresses a number of compilation problems, and fixes things so that a number of tests that were present (but not run) are now run.

I think this is just a starting point, and we need seperate folders to identify main source code, gen classes, and tests so that we don't need so many exclude rules when identifying the namespaces to AOT compile.

hlship said: The patch automatically AOT compiles the code; it automatically locates namespaces to compile. I also located a number of "dead" tests (tests that were not being run), some of which needed to be corrected for the move of c.c.test-is to c.test.

hlship said: Just added two new patch files that were created against the latest pull from GitHub master, and do not have the erroneous lines (an XML formatting issue). Should be much more straight forward to apply. Sorry for the trouble.

My build file specifies the namespaces to AOT compile but if I include another namespace (even from a JAR dependency) that is not AOT compiled, the other namespace will be compiled as well.

In my case, I was using clojure-contrib's clojure.contrib.str-utils2 namespace, and I got a bunch of clojure/contrib/str_utils2 classes in my output directory.

I think that the AOT compiler should NOT precompile any namespaces that are transitively reached, only namespaces in the set specified by the command line are appropriate.

As currently coded, you will frequently find unwanted third-party dependencies in your output JARs; further, if multiple parties depend on the same JARs, this could cause bloating and duplication in the eventual runtime classpath.

Note: I'm tracking against master for both clojure and clojure-contrib.

hlship said: I believe this is the last problem with the CI server build (and thus the Maven nightlies). Because the version number is wrong, clojure-contrib can't build on the CI server, since clojure-contrib master needs clojure-lang master but is getting clojure-lang 1.0.

Failing to do this causes a lot of confusion (I've already gotten bit) when you have stable releases out in the world and the docs reflect an alpha version. This has been painful, for example, with the shift of clojure.contrib.test-is to clojure.test.

It would be nice if there was a compatibility layer for older code compiled against 1.0's clojure.contrib.test-is to work with 1.1's clojure.test. In other words, have a clojure.contrib .test-is namespace that re-exports the corresponding code from the new clojure.test. Handling the change in the (are) macro's semantics will be a challenge, however.

This was discussed briefly on the clojure dev list, but no consensus was reached. Instead, I have attached the changes that have fixed the original library and if the clojure-contrib members wish to include it (again), then it's readily available.

The original code by Sierra required 2 changes to get working. First, @ was macro-expanding to a lazy-seq, so the filtering based on list? would fail. Second, I added a check for an explicit call to (deref). Also, I re-added auto-agent namespace for compilation to build.xml.

Maintainers: I know the Clojure patch doc says "create a patch". Would a link to the commit suffice? It seems that a `git pull` and a link to a permanent record of the contents would serve better than using the issue tracker and patch files as a primitive form of version control, at least for real work-in-progress stuff���

ataggart said: Implementation-specific functions no longer need reflection. Now the only reflection warnings remaining are a result of the proxy methods for the log-stream (flush and reset), which I can't figure out how to solve.

Using ant to build clojure-contrib.jar causes clojure.contrib.logging to be AOT-compiled.

This means that the classpath at contrib build time affects which logging implementation is used, not the classpath in use when the code using c.c.l is compiled.

Using a standard pre-built clojure-contrib.jar one can never use anything but java.util.logging! That's bad for anyone who uses released snapshots, or who uses different log implementations on different projects, but doesn't want to maintain different builds of clojure-contrib.jar.

Furthermore, someone building clojure-contrib.jar with, say, log4j in their classpath will produce a jar in which c.c.l will fail when log4j is unavailable:

cemerick said: That's not really a solution, especially for those of us who can't (or don't want to) ship .clj files.

Would folks be amenable to having each logging system setup fn return a map of implementation fns, which can sit in a top-level delay? This would push selection of a logging system out of compile-time, regardless of AOT. Adds the cost of a deref to each logging call, but I can certainly live with that.

ataggart said: I must say, I'm a bit put off by the notion that we should have to discard perfectly useful aspects of Clojure to suit some peoples reaction to whether a JAR contains some .clj files vs. exclusively .class files. AOT should be an option for code, not a mandate that infects all code in contrib.

scgilardi said: Checked in to the master branch: change to build.xml so it doesn't compile logging.clj. (suggested by Richard Newman, Phil Hagelberg).

Do we need this interim change on the 1.1.x branch? Re: 1.1.x, Stuart mentioned "assume all library code is frozen on these branches". I don't plan to push this change to the 1.1.x branch without more discussion.

rnewman said: The problem is actually worse for people who use a pre-packaged
release: c.c.logging will use the logging implementation available on
the machine on which the release was built. I'm in favor of pushing
the fix wherever it can go.

Might it be possible to have the best of both worlds by having a dynamic binding of the macros? By this I mean the log macros can be rebound when the lib is used, so macro expansions after loading the lib will use the current environment. This would preclude AOT libs from using logging though, as the macro would be different in the lib compile environment and the user compile environment.

It seems the most robust solution would be to allow the choice of either fully dynamic log binding (might be useful for libs), or macro expanded logging... which would require two namespaces: clojure.contrib.logging and clojure.contrib.logging-dynamic or something like that.