Attachments (1)

Change History (31)

One problem (perhaps the only one?) is that ECL uses the C compiler for compiling lisp (much like cython) and gives it include paths and library paths that it found out upon configuration. local/bin/sage-location is obviously not updating those. Example in a relocated "sage -ecl":

Since it is a bit unsanitary to have references to old locations lying around (one might link to wrong versions of libraries if the old location still exists!), we should probably strip those variables as well.

Possible solutions:

Patch ECL so that the above code is used to initialize these variables

Wrap our invocation of ECL so that we always update the values of these variables

equip local/bin/sage-location with the logic to update the relevant parts of ECL. That probably means recompiling (part of) ECL, because these values will be stored as a string in a ".fas"-file somewhere, which essentially a ".so".

Making initialization of *ecl-...-directory* variables runtime

;;; The following code sets the default values of the include and library directories used for
;;; invocation of the C-compiler during compilation to values relative to ECL's idea
;;; of the current "system" directory. The logical pathname "SYS:" normally points to
;;; ".../local/lib/ecl", where "..." would normally be "/usr" (if ECL is installed in
;;; "/usr/local"). So, we strip off the last 2 components of "SYS:" and append "include" or
;;; "lib". This results in the values
;;; ".../local/lib/" and ".../local/include/" in the above example.
;;; Since this substitution happens at runtime, these values get adjusted appropriately if
;;; the ".../local" tree gets relocated.
(defvar *ecl-include-directory* (namestring (make-pathname :directory (append (butlast (pathname-directory (translate-logical-pathname "SYS:")) 2) '("include")))))
(defvar *ecl-library-directory* (namestring (make-pathname :directory (append (butlast (pathname-directory (translate-logical-pathname "SYS:")) 2) '("lib")))))

Avoiding patching ECL

The role of defvar is 2-fold: It declares a variable to have dynamic scoping rather than lexical scoping and it sets the value only if the variable wasn't bound already. So, as long as we execute the two defvar statements

before the compiler package is loaded, our values will persist. So, if one finds another file of lisp instructions that are guaranteed to be executed upon ECL startup (~/.eclrc doesn't work because maxima is built by calling ecl -norc) we're still good. I was not able to find such a file.

No changes to relocation script required. The patch to ECL means that the default values for the *ecl-...-library* variables are derived from a quantity that is already aware of ECL's runtime location.

So both changes are necessary and both are to the ecl spkg and both take effect at ECL build-time (well, the patch of course also changes ECL's runtime behaviour. Since a large part of building ECL consists of ECL compiling itself, the two overlap)

(the comment about avoiding patching ECL is just recording possible avenues someone might pursue if they feel really reluctant to patch ECL)

I am for the less intrusive option but I am also pragmatic if it is far more practical to patch ecl itself.
Your option for not patching ecl would involve calling ecl before compiling any lisp code right? Which means before building maxima and sage/libs/ecl.pyx in sage if I understand correctly. The second one especially bother me so patching may be the best avenue.

Actually, for sage/libs/ecl.pyx it's not so much of a problem, since our initialization of ecllib involves sending some instructions there anyway, so we could just add the defvars there. It would add a minuscule overhead for something that is unlikely to have any effect anyway: Who's going to use embedded ECL to compile something? That is one reason to prefer the patch to the compiler package already.

For fixing the behaviour of the ECL executable I haven't actually found a solution other than wrapping it with a shell script, which really does add considerable overhead.

Patching is definitely the cleaner solution, but will likely be with us in perpetuity, with all the extra maintenance of rebasing the patch with any ECL upgrade.

OK. I am putting cutting a patched ecl according to your instruction on my TODO list for the week starting tomorrow. Like I said I'll probably cut a new maxima as well only
one non trivial doctest is being broken by 5.24.0 so it is easy work.

Should we delay making a new spkg until it is released or do you think
we should still do it now with a note to check ecl at the next rev-bump?

That depends on how urgently people think this needs fixing. Given that in practice, it only affects people wanting to build maxima on a moved Sage install and that they have a reasonable workaround of rebuilding ecl first (which builds quickly), I'd say this can wait. But if you're going to make a new ECL package anyway, I'd say patch the flaw, or at least the CCFLAGS thing.

I didn't check, but I assume these paths are still hardcoded and hence not adjusted by relocation. We're putting the "-I" and I think also the "-L" directives into those paths upon ecl config-and-build time. They are just not the right spot to put those. We'd be better off letting the environment be so that these things are found automatically, e.g. via setting the appropriate link paths (already done) and include search paths (for instance via C_INCLUDE_PATH).

Failing that, we need to patch ECL so that these paths get set relative to $SAGE_ROOT, evaluated at ecl invokation time rather than during ecl package build.

As explained above, we can now also avoid patching sage and instead configure sage to include our own customized ecl boot routine to set up the environment with the appropriate path information during ecl invokation.

Cleaning paths from *..-flags* variables

(web documentation indicates that Sun Studio and IBM compilers also support this variable, so although it's not POSIX (neither is CPPFLAGS), it doesn't seem to be GNU exclusive.)

For the record: In contrast to C_INCLUDE_PATH etc., CPPFLAGS (and CFLAGS etc.) are never read by the compiler / preprocessor; their names are just conventions used in Makefiles (and make's default rules).

For the record: In contrast to C_INCLUDE_PATH etc., CPPFLAGS (and CFLAGS etc.) are never read by the compiler / preprocessor; their names are just conventions used in Makefiles (and make's default rules).

Ah, OK. For ECL it does have a meaning: the value gets inherited as hard-wired default value for c::*cc-flags*. There also is c::*ecl-include-directory*, so including -I directives in CPPFLAGS leads to superfluous includes passed to ecl compiler invocations. That would be one reason to try to get rid of it.

The other issue is that ecl by design hard-wires the default value of c::*ecl-include-directory*, which is the real obstacle to relocating ecl. We can change that by patching ecl (but since the current behaviour is likely not considered a bug for ecl, we'd be doing that indefinitely) or we can work around it by configuring ecl to include a different setting (as outlined above, with the --with-extra-files and --with-init-form directives). [Note that this change is only required if sage builds a "private ecl" (as it does normally). If sage were to rely on an ecl supplied via other means, we wouldn't be responsible for relocating it.]

Nils or Leif: can you give an example of an actual compilation which fails because of this? The wrong paths might not matter that much if the proper include and library files are found anyway through other means (e.g. environment variables such as CPATH and LIBRARY_PATH set in sage-env).

Nils or Leif: can you give an example of an actual compilation which fails because of this? The wrong paths might not matter that much if the proper include and library files are found anyway through other means (e.g. environment variables such as CPATH and LIBRARY_PATH set in sage-env).

It's pretty clear how you can make it fail:

build sage and ecl in location <ORIGINAL>

move sage to another location

put a bogus file <ORIGINAL>/local/include/ecl/config.h (or whatever headers are required) in the old location

now trying to compile a file with ecl will use an invalid header. It's the usual risk of keeping references to uncontrolled locations in lookup tables. You may be exposed to arbitrary crap.

(I admit I haven't actually tried. It may be that with the symlink proposed on #14662 in place, the lookup happens at the right place prior to the place searched due to the -I directive. But that would really just be a fluke).

I'm a little surprised this suddenly gets marked as blocker when the issue was known for 2 years.

FWIW, C*PATH and LIBRARY_PATH are searched last, after any dirs specified on the command line.

Especially for libs, searching "random" folders is pretty delicate. (Think e.g. of bdists; we of course still have almost the same problem with R[UN]PATHs, although there's chrpath, but that wouldn't be applicable in this case.)

And it's perhaps not too uncommon to copy a Sage installation for upgrading, i.e., keeping the files at the original location.

put a bogus file <ORIGINAL>/local/include/ecl/config.h (or whatever headers are required) in the old location

now trying to compile a file with ecl will use an invalid header.

Fair enough, if you intentionally want to break it by providing a bogus <ORIGINAL>/local/include/ecl/config.h, you can. But my spkg fixes the problem for people who don't intentionally break their setup.

Note: an old ecl/config.h (as in the case for a copied install) is fine.