From 55bc27f2ac979acf28e8352a42f8e0802d2d96f9 Mon Sep 17 00:00:00 2001
From: Francois-Rene Rideau
Date: Sat, 22 Oct 2016 17:57:26 -0400
Subject: [PATCH] Update TODO wrt support for proper phase separation
---
TODO | 258 ++++++++++++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 210 insertions(+), 48 deletions(-)
diff --git a/TODO b/TODO
index 1899bc60..bb0303d6 100644
--- a/TODO
+++ b/TODO
@@ -1,6 +1,6 @@
* Decree that versions older than 2 years are not supported;
- yet, keep supporting an upgrade from whatever vendors provide
- — so push them to provide recent stuff.
+ yet, keep supporting an upgrade from whatever vendors provide
+ — so push them to provide recent stuff.
** Already, swank-asdf doesn't support anything older than 2.014.6
(as provided by the first 2011 iteration of Quicklisp), and
I (fare) am pushing for swank-asdf to not support anything older
@@ -17,7 +17,13 @@
** Make tests stateless and/or make any modifications private,
so that they can be run in parallel.
-* Moving toward cross-compilation (ASDF 3.2? 4?)
+* Testing support
+** Replace test-op with run-test-op and test-report-op ?
+
+* Implement a style-checker, that will issue STYLE-WARNINGs
+ when you don't follow all the recommended guidelines.
+
+* Moving toward cross-compilation (ASDF 3.4? 4?)
This would allow to get most of the benefits of XCVB
(reproducibility, scalability, applicability to mobile platforms, etc.),
except without the backward incompatibility of XCVB.
@@ -112,14 +118,14 @@
make t l=cmucl t=test-encodings.script
which was a different PCL bug from the above (without same workaround)
but possibly also triggered by the non-standard declaim.
-*** `#5(1 ,@`(2 3)))` returns #(1 2 3),
- rather than #(1 2 3 2 3 2 3 2 3) or even better #(1 2 3 3 3).
+*** `#5(1 ,@\`(2 3)))` returns `#(1 2 3)`,
+ rather than `#(1 2 3 2 3 2 3 2 3)` or even better `#(1 2 3 3 3)`.
*** It purports to support Unicode, but it has only 16-bit characters.
Note that (subtypep 'character 'base-char) says T -- everything is a base char(!)
** SCL has bugs:
*** it doesn't like run-program from a modified directory?
-*** it somehow pushes :non-base-chars-exist-p even though +non-base-chars-exist-p+ is NIL???
+*** it somehow pushes `:non-base-chars-exist-p` even though `+non-base-chars-exist-p+` is `NIL`???
** XCL has bad bugs:
*** make-pathname doesn't handle :type nil properly and
@@ -140,6 +146,7 @@
(make-pathname :name "foo" :type :unspecific) ==> #P"foo."
If bug is ever fixed, upgrade *unspecific-pathname-type* in uiop/pathname.
*** XCL doesn't properly process files the name of which contains a *
+*** XCL fails to recognize that specialized method parameters are implicitly ignorable.
** GCL is almost working again; but implementation bugs remain.
See Francois-Rene Rideau's messages on gcl-devel starting November 2013.
@@ -154,33 +161,33 @@
INTERNAL-SIMPLE-ERROR: The package named ASDF/INTERFACE, does not exist
*** Another GCL compiler bug:
when I changed the definition of getcwd from
- (let ((*default-pathname-defaults* #p"")) (truename #p"")) to
- (let ((*default-pathname-defaults* *nil-pathname*)) (truename *nil-pathname*))
+ `(let ((*default-pathname-defaults* #p"")) (truename #p""))` to
+ `(let ((*default-pathname-defaults* *nil-pathname*)) (truename *nil-pathname*))`
to guard against e.g. a logical-pathname context while loading asdf
- and parsing #p"", calls to getcwd result in a segfault.
+ and parsing `#p""`, calls to `getcwd` result in a segfault.
*** An another bug: gcl refuses dynamic-extent declaration on functions.
- uiop/stream.lisp: #-gcl (declare (dynamic-extent ,@(when
- before `(#',beforef)) ,@(when after `(#',afterf))))
-*** (typep p 'logical-pathname) should be T if p has a logical-pathname host.
-*** apropos is case-sensitive and returns a same symbol many times
+ ```uiop/stream.lisp: #-gcl (declare (dynamic-extent ,@(when
+ before `(#',beforef)) ,@(when after `(#',afterf))))```
+*** `(typep p 'logical-pathname)` should be T if p has a logical-pathname host.
+*** `apropos` is case-sensitive and returns a same symbol many times
(presumably once per package where it is present,
instead of just once for its home package)
-*** compile-file fails to return proper secondary values in case of non-style WARNING.
-*** (pathname-directory #p"foo/") is incorrectly ("foo") instead of (:RELATIVE "foo")
+*** `compile-file` fails to return proper secondary values in case of non-style `WARNING`.
+*** `(pathname-directory #p"foo/")` is incorrectly `("foo")` instead of `(:RELATIVE "foo")`
*** Do whatever it takes to pass the asdf tests, add the above?
-*** Trying to uiop:slurp-stream-forms from a stream with #+(or) :foo
- (or read-file-forms from an file with same) results in an error,
- rather than nil. This is probably a bug in #+ processing.
+*** Trying to `uiop:slurp-stream-forms` from a stream with `#+(or) :foo`
+ (or `read-file-forms` from an file with same) results in an error,
+ rather than `nil`. This is probably a bug in `#+` processing.
Unhappily, debian creates such a file in
- /etc/common-lisp/asdf-output-translations.conf.d/01-common-lisp-controller.conf
+ `/etc/common-lisp/asdf-output-translations.conf.d/01-common-lisp-controller.conf`
*** Tests that try to catch an error fail (but catching a warning succeeds),
which suggests brokenness in handler-bind and/or error.
-*** COMPILE-FILE* fails due to handler-bind error in
- with-muffled-compiler-conditions or so.
-*** `#5(1 ,@`(2 3)))` returns #(1 2 3),
- rather than #(1 2 3 2 3 2 3 2 3) or even better #(1 2 3 3 3).
-*** (DIRECTORY #p"*.*") fails to match files with pathname type NIL.
-*** GCL doesn't properly process files the name of which contains a *
+*** `COMPILE-FILE*` fails due to handler-bind error in
+ `with-muffled-compiler-conditions` or so.
+*** `#5(1 ,@`(2 3)))` returns `#(1 2 3)`,
+ rather than `#(1 2 3 2 3 2 3 2 3)` or even better `#(1 2 3 3 3)`.
+*** `(DIRECTORY #p"*.*")` fails to match files with pathname-type `NIL`.
+*** GCL doesn't properly process files the name of which contains a `*`
** ABCL has a few bugs.
*** ABCL fails the timestamp propagation test.
@@ -243,12 +250,6 @@
** Real solution: defer parsing and evaluation of defsystem forms?
Or simply detect circularity and issue an appropriate error?
-* Implement proper incrementality for defsystem-depends-on.
-** For proper incremental redefinitions, systems probably need to
- depend-on (load-asd-op . primary-system) which itself will
- depend-on all the load-op of defsystem-depends-on in that file.
- https://bugs.launchpad.net/asdf/+bug/1500578
-
* Include some ABL test for stassats's (now obsolete?) thing:
(asdf:enable-asdf-binary-locations-compatibility
:centralize-lisp-binaries t :default-toplevel-directory *fasl-dir*)
@@ -382,6 +383,20 @@ It looks like SWANK can be fixed soon, though, so we'll see.
version
* ASDF4: search for this tag, rename things (incompatibly, thus) and cleanup code.
+** Migrate from component-depends-on to action-depends-on
+*** I contend a future version of ASDF will replace
+`(component-depends-on operation component)`
+with `(action-depends-on plan operation component)`.
+This allows for different normalization strategies for dependencies
+(including strategies that avoid resolving things to NIL),
+a reified context for featurep checks, etc.
+*** Easy but long backward-compatible transition:
+**** get all users to use the new gf and keep extending both new and old gf,
+ meanwhile the new gf has an around method that actually calls the old gf
+ except in testing mode and/or if we can check that they're using the new convention
+**** when everyone has migrated, remove the old mode and the short-circuit.
+*** However, we cannot deprecate component-depends-on yet — not until we have
+ some transition in place to a better interface.
* Documentation
** See message from 2014-01-27 05:26:44 GMT for stuff to document.
@@ -417,22 +432,6 @@ It looks like SWANK can be fixed soon, though, so we'll see.
*** There again, a check that a forward-package is not backward
would be very nice.
-* Migrate from component-depends-on to action-depends-on
-** I contend a future version of ASDF will replace
- (component-depends-on operation component)
- with (action-depends-on plan operation component).
- This allows for different normalization strategies for dependencies
- (including strategies that avoid resolving things to NIL),
- a reified context for featurep checks, etc.
-** Easy but long backward-compatible transition:
-*** get all users to use the new gf and keep extending both new and old gf,
- meanwhile the new gf has an around method that actually calls the old gf
- except in testing mode and/or if we can check that they're using the new convention
-*** when everyone has migrated, remove the old mode and the short-circuit.
-** However, we cannot deprecate component-depends-on yet — not until we have
- some transition in place to a better interface.
-
-
* Faster source-registry:
** In addition and/or as a substitute to the .cl-source-registry.cache,
that is meant to be semi-automatically managed, there could be
@@ -457,10 +456,173 @@ It looks like SWANK can be fixed soon, though, so we'll see.
* Properly deprecate all that needs to go
** Get lisa, readable, bourbaki to NOT include ASDF 1 anymore in their sources.
** Identify all the things in ASDF that we want to deprecate, and make
- sure we deprecate them all in 3.2.
+ sure we deprecate them all in 3.4.
** Implement support for deprecating old packages,
by having all functions and macros under the old package name be
proxies to those in the new package name?
(Exception: special variables can't be proxies, have to be imported.)
** Eradicate Quicklisp systems that depend on asdf-driver or
asdf-package-system, make them depend on uiop and asdf3.1 instead.
+
+* Fix plan
+** Goal: Implement proper incrementality for defsystem-depends-on.
+Fix the bug wherein ASDF fails to properly handle incrementality for `defsystem-depends-on`.
+https://bugs.launchpad.net/asdf/+bug/1500578
+Some discussion about ASDF at
+http://ccl.clozure.com/irc-logs/lisp/lisp-2016-10.txt
+
+** DONE Items
+*** Add `DEFINE-OP` for loading system definitions.
+*** A `DEFINE-OP` will depend on other `DEFINE-OP`s
+*** A `DEFINE-OP` depends on all `LOAD-OP` in the file.
+Whether via defsystem-depends-on or manual `LOAD-OP`s.
+*** Store timestamps in `DEFINE-OP` entries, not `*DEFINED-SYSTEMS*`.
+*** In find-system, add condition that all dependencies are up-to-date.
+*** This necessitates another layer of visit states,
+that do not plan, just check planning...
+*** `LOAD-ASD` calls `PERFORM DEFINE-OP` (not the other way around)
+*** Systems probably need depend on `(DEFINE-OP . PRIMARY-SYSTEM)`
+*** These dynamically-discovered dependencies are stored in a slot of `SYSTEM`.
+*** Have a special kind of definition-depends-on dependency.
+Before we reuse a node from a previous session,
+we must make sure it is still properly defined.
+This matters because we do reuse (at least *some* nodes, e.g. system nodes)
+from previous sessions, and the control structure may have changed
+(i.e. the system hasn't been redefined, but one of its defsystem-depends-on
+dependencies has changed).
+*** Handle incomplete definitions properly
+Don't just let the incompletely parsed system be registered
+in a false positive success as if it were an empty system.
+Solution: It is registered, but considered out-of-date,
+so next attempt to use it will cause a refresh attempt.
+https://bugs.launchpad.net/asdf/+bug/1515372
+*** Handle nested calls to `OPERATE`
+We must accept that ASDF isn't really plan-then-perform,
+as the ASDF1 model claimed to be.
+It's an arbitrary nesting of plan-and-perform, like a parenthesization
+with plan (steps) as left parenthesis and perform (steps) as right parenthesis,
+and with operate being a maximal balanced span of plan and perform steps.
+*** What goes in a shared dynamic `*SESSION*`, what in a private lexical `PLAN`?
+Memoization CACHE of various functions, Status of VISITED-NODES, FORCING parameters.
+The session will contain not just the current `*CACHE*`, but also
+all or most of the graph traversal, thus the `:force` options, etc.
+The `*PLAN*` may or may not contain some work queueing.
+But even, e.g. a table of "nodes still directly blocking us"
+is shared at the shared traversal level.
+*** It is an error for a nested `OPERATE` to contradict session flags.
+This raises the question of `:force` and `:force-not` flags.
+An actual operate cannot contradict them and cause things to be performed
+with a wrong subset of dependencies loaded.
+Yet, `REQUIRED-COMPONENTS` (that does cause anything to be performed —
+except via nested `OPERATE` from `defsystem-depends-on`)
+must be able to skip some dependencies
+(though maybe only _in addition_ to the global ones).
+*** What timestamp for `forced` and `forced-not` actions?
+If an action is forced-not, it should be returned with an action stamp from this cache,
+and not with `nil` as is currently wrongly done
+(unless of course the cache entry is missing,
+at which point `nil` is the correct answer).
+Conversely, a force means the stamp should be `t`
+(indicating it needs to be done in the future)
+(that part asdf gets right).
+*** Handle planning things that are now done but were done in a previous phase of the build,
+so dependency on them should trigger the build.
+*** Protocol to not re-perform an action twice
+In PERFORM-PLAN, check that an action wasn't already DONE-P due to
+being needed in a previous phase of recursively calling OPERATE,
+before to PERFORM it.
+To correctly propagate timestamps in across nested calls,
+the graph traversal and timestamp propagation is shared at the session level
+between all nested calls to `OPERATE`.
+During a session and across phases, every action has a status made of
+a stamp and three bits: KEEP-P, DONE-P, NEED-P.
+The stamp also survives across sessions, in `COMPONENT-OPERATION-TIMES`.
+*** Detect circularities across nested calls to `OPERATE`
+This means the circularity detection data is part of the session.
+Notably, the visiting list and set.
+The top of the list can also serve to record the dependencies from `OPERATE` to `DEPEND-OP`.
+*** In UIOP, RECORD-DEPENDENCY must not cross to an upper plan and/or it should
+record the dependency in said upper plan (that matters because
+parallel plans doing extra checking, and this is a problem
+whereby some links to extra actions are left unresolved).
+Maybe in the future, record the dependency in the session, not in a plan.
+
+** TODO items
+*** find-system should probably cause a dependency on define-op, especially inside another define-op
+*** Is it worth it trying to optimize away a dependency on a define-op
+when an action already depends on another operation that "obviously" depends on that define-op?
+*** Document DEFINE-OP and the design in asdf.texinfo and/or in some article.
+
+** Future developments
+*** Grovel all .asd files in Quicklisp and collect all violations of declarativeness,
+where declarativeness is having only defsystem forms. Specially classify in-package,
+maybe also defpackage. Then we can assess the impact of requiring declarativeness.
+*** Also grovel Quicklisp for all extensions to ASDF:
+see if there are new methods on perform, input-files, output-files, operate, etc.
+*** To allow defsystem-depends-on to another secondary system within the
+same file, I can imagine some system to dynamically create a separate action
+(DEFINE-OP foo/x) for each form defsystem foo/x, and have it depend on
+all the preceding actions operated in the .asd file.
+Plus a last action for dangling load-system or operate statements?
+Or one action for every operate statement?
+There would be no action of the entire load-asd, but a separate mechanism
+to detect circularity?
+What about dangling statements? They would be a bit like xcvb weakly-ordered
+run-time dependencies that CAN have circularities? Ouch. Really ugly.
+Each statement would need its own action? Ouch.
+*** Have load-asd clear any invalidated previous secondary system
+before loading, which supposes remembering what these systems are.
+Then, you're not allowed to forward-reference a secondary system that
+hasn't been defined yet while you're loading the .asd, whereas you
+can load the asd if you aren't loading it yet.
+*** Maybe split DEFINE-OP into PREPARE-DEFINE-OP and DEFINE-OP, so that
+secondary systems can be properly loaded?
+*** Add mapping from files to action that creates it and/or uses it?
+Then we can detect conflict between multiple actions creating one file.
+Forbid direct use of files not listed as inputs? What about transitive inputs?
+Build in a container that has only the proper files...
+*** What data structure for scheduling planned actions?
+A doubly-linked to remove entire chunks of the plan after a sub-operate?
+That won't do, because later-discovered defsystem dependency can refer to
+systems that were planned but not performed, so there need be no contiguity
+in the chunks that are performed by subcalls to operate.
+Actually, if we otherwise forbid reference to another system
+from non-system components, then we could move things with system granularity,
+which is one thing; but we still need a general mechanism at the system level.
+So, whether it's an explicit queue or an implicit traversal of the dependency graph,
+we must check whether an action has already been done before we try to perform it.
+And an implicit traversal has the advantage that you can avoid entire subtrees
+when you find that the current node has already been performed during
+a previous phase of the plan.
+In conclusion: "just" traverse the graph at the last minute, and
+perform things serially, and/or if you want some parallelism,
+build a queue dynamically at the last minute for the "now" jobs
+while keeping most of the plan as graph structures.
+
+*** Merging in parts of POIU?
+ASDF internals are nothing to be proud of:
+ASDF1 was optimized for "smallest code that kind of works in the usual case",
+and ASDF3 was optimized for "smallest code that fixes ASDF1 in the general case
+while mostly maintaining backward compatibility".
+
+Because of its optimization constraint, ASDF1 was using and abusing lists a lot,
+rather than defining nice data structures.
+ASDF3 uses somewhat more structured data, with a few more classes, and
+algorithms that are O(n) instead of O(n**3) in simple cases and
+exponential in carefully crafted examples;
+yet it has shunned any definition of more advanced or general data structures.
+By contrast, POIU has a dequeue to represent queued compilation jobs, and
+a general graph representation for the action graph
+(though the original POIU was also O(n**3) because it lacked PREPARE-OP).
+
+I could import the POIU queue and graph representations into ASDF, and
+that would add about 100 lines of code to UIOP for the queue, and
+200 lines to ASDF for the graph, maybe a bit more if the code is
+generalized, commented and largely put in UIOP.
+
+Then, there's the support for actual forking and using forks,
+that would be about 400 lines added to UIOP, and
+the using it in ASDF to consume the plan, which would be under 200 lines.
+But those parts are probably better left out of ASDF itself.
+That said, it all was a lot when ASDF was < 1000 lines of code,
+but isn't all that much now that it is > 12000 lines of code.
--
2.18.1