The core.typed checkers runs forever on my files. I've added markers to pinpoint the problem (unchecked functions are mentioned on the standard-output). Thus I was able to prune my failing file to a fairly minimal test-case that runs forever.

I would be nice if (check-ns) had an option to output logging/information for each function that is checked. As this would save significant time in pin-pointing where core.typed stagnates.

I'm using clojure.core.typed 0.2.19 with the slim classifier but I've observed the same without slim.

My suspicion is that the latent filters associated with functions grow in size exponentially with each extra optional key to a HMap (based on the output when you have a type error). I think it's generating all combinations of present and absent keys for the HMap when calculating latent filters for a function.

I've attached a tarball with a lein project with ten namespaces that all contain the same ten simple functions in the form

I've spend quite some time to learn core.typed on a real use case, and I have to say that I'm amazed by the power of the analysis and the potential to prevent errors that otherwise would be detected to late! Apart from that the type-annotations are valuable and precise information about the interface when doing maintenance at code you did not see for some time.
Two big reasons to use core.typed, however, as I turned more and more code into typed code the number of optional keys increased in my core data-record increase, and core.typed (check-ns) slowed down to the extreme (using version 0.2.25 (current stable)). I refactored the code to get a single line of code that triggers the issue on my core data structure a (Seqable DbMsg). Attached you find the project.clj and code (core.clj vs 8:09pm).

When I reduce the number of optional keys to 4-5 the (check-ns) runs in a few seconds. But having 9-11 optional keys kills the (check-ns) process.

I hope this one-liner helps making the issues easy to reproduce.
Currently I don't know a work-around for this issue,
so I can not use core.typed to check my project.

I guess that many people/projects should run into the same issues,
as the pattern is returning quite often in clojure:

1. start with data-set with small Hashmaps
2. the hashmaps evolve in a number steps (meaning more keys are added and some of the existing keys are removed)
I guess the workaround would be to use limited or no optional keys, and prepare a custom HMap definition for each stage of analysis. Although this would work, the amount of bookkeeping already significant for a all records are running through the same stages (linear process), and will be a showstopper if it is a non-linear process (not all records follow the same path (sequence of analysis stages)).
I will investigate whether splitting in multiple types (for different stages) will rescue my day.

This is theoretically trivial to implement, and just involves shuffling around where optional keys are handled. However, this is spread out all over the code base, so I may not complete the patch for a week.

The code in hmap/big-options.clj (from your test-suite) still contains a workaround to make core.typed proceed without errors.
The (if (string? notifs) ... on line 107 is introduced to prevent a type-error (core.typed infers it that notifs might als be a (sequable String). However, as this is the else-branch of line 104 (if (or (seq? notifs) (vector notifs)) ... the case that notifs is a (Seqable string) can be excluded already).
Also using on line 104 the more general (if (sequential? notifs) ... does not resolve this issues.

Is this inference error already on the list, or would you recommend posting a seperate JIRA-case for this issue?

ann-record's docstring says "Annotate record Class name dname with expected fields", but it does not specify the syntax that it expects for the fields. The Quick Guide and Types pages on the Wiki don't mention it either.

I believe it needs to be updated to account for returning a nil value if the channel is closed, or if the operation is a put, given the doc for alts!/alts!!:

clojure.core.async/alts!
([ports & {:as opts}])
Completes at most one of several channel operations. Must be called
inside a (go ...) block. ports is a vector of channel endpoints, which
can be either a channel to take from or a vector of[channel-to-put-to val-to-put], in any combination. Takes will be
made as if by <!, and puts will be made as if by >!. Unless
the :priority option is true, if more than one port operation is
ready a non-deterministic choice will be made. If no operation is
ready and a :default value is supplied, [default-val :default] will
be returned, otherwise alts! will park until the first operation to
become ready completes. Returns [val port] of the completed
operation, where val is the value taken for takes, and nil for puts.

opts are passed as :key val ... Supported options:

:default val - the value to use if none of the operations are immediately ready
:priority true - (default nil) when true, the operations will be tried in order.

Note: there is no guarantee that the port exps or val exprs will be
used, nor in what order should they be, so they should not be
depended upon for side effects.

[10:29:10] hugod: I'm getting "AssertionError Assert failed: (symbol? y) clojure.core.typed.check/abo/lookup--34688 (check.clj:2906)" - trying to work out what abo does...[10:31:00] ambrosebs: hugod: IIRC it's "abstract object"[10:31:40] ambrosebs: hugod: it's how we abstract an "object" in the occurrence typing sense from a local name (symbol) to a position arg[10:32:13] ambrosebs: ie. how we convert (fn [a] a) into (All [x][x -> x :object {:id 0}])[10:32:32] ambrosebs: hugod: unsure of where the error is coming from tho[10:33:04] hugod: this is what it is checking https://www.refheap.com/19405[10:34:50] hugod: at least that is what is at the last line number reported by :trace true

To reproduce, clone pprng (note the WIP-typed-clojure branch). The project uses cljx, so you probably should run lein cljx auto while you're tinkering with the source so that the cljx transformation is run automatically in between your running (check-ns) (cljx doesn't generally require preprocessing, but TC seems to always reload definitions from disk and doesn't support REPL interactions based on in-file annotations.) Then:

Unit tests involving intersections often randomly fail. I think we need to carefully reconsider the constraint resolution algorithm cases for intersections, in particular we need to be more consistent about which combinations of intersection members we use in the final cset.

Clunk. You're absolutely right. Assuming there's no (cf) there, is there any way you could log an error for an annotation with no associated actual definition? Obviously, to be useful, it'd have to appear in the appropriate location in the file.

Turns out one of my dependencies, Quil, transitively depended on an old version of ClojureScript [org.clojure/clojurescript "0.0-2080"]. Sorry about the invalid report, I'll make sure to come up with a clean reproduction the next time.

[abonnair@catbert datomic]$ git am --keep-cr -s --ignore-whitespace < 0001-More-fixes-for-nses-with-no-source.patch
Applying: More fixes for nses with no source
error: patch failed: module-check/src/main/clojure/clojure/core/typed/check_ns_common.clj:89
error: module-check/src/main/clojure/clojure/core/typed/check_ns_common.clj: patch does not apply
Patch failed at 0001 More fixes for nses with no source
When you have resolved this problem run "git am --resolved".
If you would prefer to skip this patch, instead run "git am --skip".
To restore the original branch and stop patching run "git am --abort".

It looks like the first didn't apply because the code has drifted. Uploaded a second attempt, ctyp-174-2.diff. It should apply cleanly to master, on its own. The change is very simple, it just returns an empty set for the ns deps for closed source nses. The check-ns run looks like:

WARNING:File for datomic.api not found on classpath: datomic/api.clj
Not checking datomic.api (ns form missing)

>> according to @ambrosebs:
19:56 <freakhill> 19:05 <freakhill> do you have an idea of what
causes that bug?
19:56 <freakhill> so i can isolate a failing example?
20:00 *** arohner QUIT Ping timeout: 265 seconds
20:11 <ambrosebs> freakhill: this one I think (t/cf (t/for [x :-
t/AnyInteger, ['(1 2) '(3 4)]] :- (t/Seq
t/AnyInteger) x))
20:21 <freakhill> thanks
20:29 <ambrosebs> nil strikes againhttps://twitter.com/ambrosebs/status/552401747084595200
20:30 <freakhill> (t/fn [] :- '{} (into {} '())) crashes too
20:31 <ambrosebs> that makes sense
20:32 <ambrosebs> the issue is anonymous functions that don't have
an explicit :filters (which is impossible to
expression atm)
20:32 <ambrosebs> *express
20:32 <ambrosebs> I use a NoFilter to say "infer this filter and
add to the resulting function type"
20:33 <ambrosebs> but I forgot to handle NoFilter deeper in the
proposition simplification machinery
20:33 <ambrosebs> that assertion catches the sloppiness, a NoFilter
is being returned
20:34 <ambrosebs> I don't know how I don't have a unit test that
fails
20:34 <freakhill> thanks for the deeper insight!
20:35 <ambrosebs> ah probably because the proposition environment
only needs to simplify if there's a complex
expression inside of the fn
20:35 <ambrosebs> I bet I just have values and branches, instead of
`into` which is a polymorphic invocation
20:36 <ambrosebs> eh maybe
>

The attached code fails when checking the namespace, as core.typed incorrectly assumes that the return-type is incorrect. Part of error-output is included as string ErrInfo and compared for difference with the return-type from the proto-type of the failing function.

When you load the file the analysis of the error-message is shown (run check-ns to see the message)

The objective here is to be able to apply typed-clojure to libraries, without having their downstream dependents require the typed-clojure dependency as well. The libraries in question would then be able to fold in a separate TC dependency (which would bring in e.g. analyzer, ClojureScript, etc) for doing the actual type checking at test-time, in addition to their "regular" tests which would be run without the TC machinery. (Leiningen profiles would be ideal for setting up these different sorts of testing contexts.)

Tasks include:

1. Defining a "top-level" zero-dependency namespace(s) that provides all of the user-facing type-annotation macros (which must be no-ops when evaluated outside of a type-checking process).
2. Modifying the TC pom.xml and other project configuration so that two dependencies are produced; one without dependencies containing only the above-noted namespace(s), and another that depends upon the first and which also carries the necessary dependencies for actually performing type checking.

Attached an additional patch ({{CTYP-145-1.diff}}) that adds a minimal test to ensure that core.typed.rt can load. This will catch cases when that subset of typed-clojure expands, but e.g. the file list in module-rt/pom.xml hasn't been updated to match.

This error is related to the example longOptions.clj of CTYPE-102 and the derived hmap/big-options mentioned as CTYPE-99. I've prepared a new example to clearly show the issue.

The first function passes type-checking. However, it contains a redundant check on line 83 and a redundant base-case on line 85.

The second function does not pass the (check-ns) as it assumes that notifs can be a (seqable String). However, as this is the else-branch of line 112 (if (or (sequential? notifs) (seq? notifs) (vector notifs)) ... so the notifs can not be a (Seqable String) in this branch.

NOTE: core.typed does not know clojure.core/sequential? yet. Therefore the seq? and vector? tests are added too.

Seqable doesn't extend Sequential, ISeq or IPersistentVector, so it's expected that this fails.

I suspect that the type for :st_notif is always an immutable clojure collection? If so, annotate :st_notif as (U (t/Coll String) String nil), and use (coll? notifs) as the predicate. Otherwise, try (not (string? notifs)) as the predicate.

clojure.core/re-find returns different things depending on the regular expression used and the string it matches against. Currently core.typed annotates this as returning (Option String). However, in the presence of capture groups, it may return a vector of strings, so the current annotation is too specific.

Additionally, clojure.core/re-groups is not annotated, and uses the same return type as clojure.core/re-find.

Affects version 0.1.8 (but version isn't listed as a valid version in JIRA)

clojure.core.typed refers to clojure.main/repl (line 725 in master)

However clojure.main is not required by the clojure.core.typed namespace, so you get an error if clojure.main has not already been loaded. This can happen in various circumstances, e.g. running "mvn test"

When I first tried to use check-ns, I assumed incorrectly that it could be called from within a source file.
I was led to believe this from the example here: https://github.com/clojure/core.typed/wiki/User-Guide#var-warnings
Where (check-ns), although being commented out, is included in the code of a snippet which looks like a file.

Tweaking the docstring for check-ns to state that the function is intended for use from the REPL would probably reduce noobie-flailing.

Also a friendly reminder for you to update the docstring for cf to indicate that it should not be called recursively.

Although the user guide mentions defprotocol>, it's very easy to run into this error by accident, and the error message does not give the user any hints as to what might be wrong. In the long term it would be best if core.typed could deal with clojure.core/defprotocol. As a stopgap measure, maybe core.typed could detect this error and instruct the user to use clojure.core.typed/defprotocol> instead?

Should type check the current namespace with this code and match up the line/col/source positions in the actual Light Table window, and display the type error message above/below it (whatever is more common in LT).

Also "check-form" should be a command, that type checks the current form with clojure.core.typed/check-form-info.

2 refactorings should be included:

ann-form - wrap current form in an clojure.core.typed/ann-form (which could be qualified by an alias) and leave the cursor on the second argument. If it's possible to add a default to the second argument, it should be `Any`.

eg. my-form
-> (t/ann-form my-form |Any)
; where | is the current cursor

ann-var - bring out the var on the current cursor into an `ann` into the top-level form above the current form.

if the var is from another namespace than the current, it should be fully qualified and prefixed with ^:no-check
eg. n/another-var => (t/ann ^:no-check long-another-ns/another-var Any)

previously, user-defined function couldn't be annotate as dotted polymorphic function, this patch make this possible. Also, while writing the test case, I found that the code assume function's requried-params is more than or equal to the dom of function's type, but not less like `(fn [& y] (if (empty? y) nil (first y))) (All [x y ...][x y ... y -> (U x nil)])`, so we may skip the function type `x`, I fixed this by changing the interface of check-fn-method1-rest-type from `(fn [rest drest kws]` to `(fn [remain-dom rest drest kws]`.

Instead of using a HVec (which in itself isn't accurate, the rest argument is a nilable non-empty seq), just use the same approach as previous, except take the union of the remaining dom and the dotted pretype.

I think your approach throws away the rest and drest information which is an issue.

HVec supports rest/drest, but HSeq does not. You can add support by copying how HeterogeneousVector does it.

Also we should introduce a HSequential that just extends clojure.lang.Sequential. Then we can use it to abstract over HVec, HSeq and HList, and use it in the first argument of `first`, rather than hard-coding HVec.

Which reminds me, clojure.lang.ISeq isn't sequential, so we need to ensure HSeq is sequential.

I'm pretty sure the code you commented out is incorrect. It doesn't use the rest/drest type properly. better just to remove it and just leave a comment referencing this ticket. We can always look back at the ticket.

add (every? r/Type? remain-dom) to the :pre of check-fn-method1-rest-type

Thanks for the change, but there is no need to apologies to Eastwood It still has rough edges all over the place where code that the Clojure compiler handles causes Eastwood to throw exceptions for various reasons, and not all of them are because the Clojure compiler is lenient.

I could understand core.typed thinking it could be `nil` if it looked at both members of the union, but it is very strange to me that it thinks the second item in the vector could be :params or :no-params, as those should only occur in the first position...

Perhaps this isn't something that's possible right now? As a human, I know that (:op stmt) can only take one of two values, each of which uniquely picks between the union values. I think this would involve changing variable stmt based on information from key, which I have yet to find in the source code.

The change I am suggesting here may be a bug, and I simply do not realize it yet.

clojure.core/defn checks whether the first arg is called

&form

, and if so, it removes the first two args from the arg vector to create the value for the :arglists key of the var's metadata. I believe this is because almost every time the first arg is called

&form

, it is a function created by a macro definition, and the first two args are

&form &env

, and Clojure programmers would typically prefer not to see those hidden args when they do (doc name-of-macro).

Function clojure.core.typed/add-to-alias-env has a first arg

&form

, and thus its :arglists are set to

[t]

rather than

[&form qsym t]

. I don't think there is any bug here, but if it is correct to change the name of the first arg to something else, e.g.

form

, the :arglists would be closer to what one would expect.

The only reason I noticed this is due to a bunch of :wrong-arity warnings from Eastwood when linting core.typed, since the situation described above means that it appears that all calls to add-to-alias-env with the correct number (3) of args appear to be the wrong number of args.

tools.analyzer.jvm returns a :host-interop node when, like in this case, it encounters an interop form of the form (.foo bar) and can't determine whether it's a no-arg method call or a field-access.

I don't know enough about core.typed internals but it looks like there should be an add-check-method for :host-interop that behaves like check methods for :instance-field/:instance-call that resolve to runtime reflection (not :validated)

DEPRECATED SYNTAX (typed_test/core.clj): Any syntax is deprecated, use clojure.core.typed/Any

This comes from parse-unparse/parse-type-symbol 'Any but I haven't been able to figure out what triggers it yet. It's clearly not triggered by anything in user code tho' since it happens even when user code does not mention Any.

Currently it is fairly difficult to use core.typed to achieve/ensure production quality code. Often it happens that part of your own code does not pass the core.typed checker. Of course you can add the :no-check to your function, but in that case all type-checking is bypassed for this function and it can inject arbitrary data into the rest of your program (the type-signature is not enforced anymore.

It would be possible to prevent this issue if core.typed offered the opportunity to derive pre- and post-conditions based on the type-signature. This way unchecked code would use run-time checking as a fall-back scenario to prevent unchecked code could inject arbitrary data into the rest of your program (like the schema-library by prismatic, but using the core.typed annotations instead of schemas).
One step beyond would be to:
1. Use dynalint for the unchecked code (if this is possible)
2. Generate wrappers with pre- and post-conditions for external libaries.

This is highly desirable. Typed Racket does an excellent job, so it's simple enough to follow their lead.

The biggest issue is actually the time it takes to load the type checker. I don't want to impose this on anyone who isn't generating contracts from types.

I've pondered writing a simpler intermediate AST representation of types that requires no further loading of the type system. It would look very much like the output of tools.analyzer or CLJS analysis. When a global annotation is added, it would be converted to an AST (very cheaply). Converting this AST to a contract would be less robust than using the type manipulating utilities of the full type checker, but the idea is that a type contract might generate incorrectly, but a "check-ns" always tells you after the fact.

It sucks, but some people rely on the zero load-time of core.typed (it's a pretty sweet property too!).

In short, I've thought a lot about this, and it's a glaring missing component in a gradual type system.

I can imagine the dilemma. Core-typed has zero slow-down as all checking is taken off-line. I can image that the launch of the type-system at runtime is a significant overhead. To prevent this overhead the preparations of the type-checking should be brought to compile-time. Your suggestion of generating AST representations does this.

I was considering generation of source-code at compile time to generate ordinary pre- and post-conditions, or to wrap a function symbol with some checks. However, it seemed to be less than trivial to locate where the ann-macro store the type-annotations. I suppose the limited documention on this type of internal.

I guess the manual insertion of clojure pre- and post-conditions on unchecked functionscurrently is the best way out for the time being.

Thanks, this is a great tool to implement a postcondition for a function that has a ^:no-check directive. This way you can prevent that unchecked functions insert incorrect data into your process-flow.
Cool!

Well, I think we're talking about two different problem here, the problem you're talking about is how to compare HSequential with other type, the one I'm talking about is how to use CountRange to extends HSequential type, because

(HSequential [Any *])

can never be the subtype of

(HSequential [Any Any *])

, only when we put constraint that the count of first one is at least 1. Right?

So, I'll first extends `In` as I described here, and leave your solution to future. For the efficience consideration, I think it's not a big problem, because most CountRange just specified low bound as `empty?`, so we don't need to generate a large set very often.

but after changing back to r/-hsequential, there'll be a type error. I don't know what I'm missing, I've debugged it for a whole day and still feel lost. I've greped the code that support HVec in fold_default.clj, frees.clj, promote_demote.clj and subst.clj. And copy the code to implement HSequential, but still got type error. I don't know what's going on here.

ps. the bug you mentioned in the first comment is not a bug, I was going to test if t is HSequential or not, and upcast s to HSequential if so, just like the {HVec,Hlist,HSeq}->HSequential you menntioned.

Well, turns out we don't need CountRange to pass the test case, but we also can't pass it with `first`, because both function and arguments have free variable, so change the test case to concrete type.

That particular addition fixed (cf (-> {:a 1} first second) Number), because AMapEntry has a HVec ancestor and we used to rely on `second` having a HVec arity. Now `second` has a HSequential arity, we want to trigger this case if there's a (cs-gen RClass HSequential), instead of just (cs-gen RClass HVec).

This was implemented sometime in 2014, but is fairly undocumented and not really supported. Once support for negation types is better supported by the constraint generation algorithm (cs_gen.clj), we should reconsider fully supporting this feature.

Problem

We want to rewrite code in the body of functions if they are
only checked once (ie., the functions are not of an intersection
type that overloads the same arities). Currently, function bodies
are not rewritten directly to the respective AST node they originated
from, in case they are checked more than once.

Solution

If an arity is checked only once, then associate the result of type
checking over the old body. Otherwise, keep the current behaviour.