ecls-list

Hello,
I've been playing a bit with FFI on a non-threaded ECL 9.7.1 build on
NetBSD 5.
In CFFI tests, cffi:load-foreign-library generates the following error
at compile-time:
LOAD-FOREIGN-LIBRARY requires ECL's DFFI support. Use FFI:LOAD-FOREIGN-LIBRARY with a constant argument instead.
Indeed replacing it by ffi:load-foreign-library works, but this
basically means that every CFFI-based library I try to load fails to
work without local changes. Interestingly, FFI:*USE-DFFI* returns T
so I'm a bit confused as to what is missing relating to DFFI.
Some UFFI tests seemed to work allright, including
CFFI:LOAD-FOREIGN-LIBRARY.
Another test was using an existing library built on CFFI. iolib was
the first candidate but unfortunately failed to build with some "not a
proper list" error in an asdf file which appears fine (I might
eventually look further into this, it also works on SBCL).
So I then tried lispbuilder-sdl, which builds fine once the
CFFI:LOAD-FOREIGN-LIBRARY is replaced with an FFI:LOAD-FOREIGN-LIBRARY
call. Some lispbuilder-sdl examples/tests then work fine, while most
others fail with the error:
4278190080 is not of type FIXNUM.
in SDL-CREATE-RGB-SURFACE
Which function is defined as follows:
(cffi:defcfun ("SDL_CreateRGBSurface" SDL-Create-RGB-Surface) :pointer
(flags :unsigned-int)
(width :int)
(height :int)
(depth :int)
(Rmask :unsigned-int)
(Gmask :unsigned-int)
(Bmask :unsigned-int)
(Amask :unsigned-int))
Unfortunately the backtrace didn't provide the arguments passed to the
function call.
The test that succeeds (sdl-example:bezier) appears to then leave ECL
looping in an endless RUN loop once the window is closed (not even
interruptible via slime, I have to send the process SIGTERM from the
shell. All tests succeed on SBCL, also without the endless loop
issue.
Thanks,
--
Matt

On Tue, Jul 21, 2009 at 4:03 AM, Matthew Mondor<mm_lists@...> wrote:
> Since basic custom UFFI tests seemed to work, and considering that ECL
> appears to have native support for it and that the resulting assembly
> for UFFI stubs seemed much better than for CFFI ones (i.e. 2 vs 6
> function calls, 18 vs 30 instructions for a void (*)(void) function
> wrapper, I decided to then try cl-sdl which uses UFFI.
CFFI has some weird assumptions about what a foreign function
interface can and can not do. It assumes everything has to be dynamic
while ECL prefers to compile things (the resulting code is more
efficient), does not care that functions with varargs ("..." in C) may
be different from other functions (making our compiled code really
really unportable), etc. It was never built with ECL in mind and
development never accepted a twist in this direction, that made my
initial port very difficult and I have never had time to continue
supporting it. I do not know the status of CFFI support for ECL. On
the other hand, UFFI is much much easier to deal with, being closer to
C and keeping really a subset of what can be done on any platform.
> I first removed UFFI ASDF dependencies from the .asd files (ECL
> provides it and UFFI package mentions nothing about ECL support),
> then had to fix a few problems relating to how #+unix conditionals
> weren't being evaluated, and it started building, yet I then get some
> interesting errors at various function definitions using structures as
> arguments, despite those structures being previously defined:
>
> [...]
>
> (def-foreign-struct surface
> (flags uint32)
> (format (:struct-pointer pixel-format))
> (w int)
> (h int)
> (pitch uint16)
> (pixels :pointer-void)
> (offset int)
> (hwdata :pointer-void) ; internally struct (opaque pointer)
> (clip-rect (:struct rect))
> (unused uint32)
> (locked uint32)
> (map :pointer-void) ; internally struct (opaque pointer)
> (format-version unsigned-int)
> (ref-count int))
>
> [...]
>
> (def-foreign-routine ("SDL_Flip" flip) int
> (screen (:struct-pointer surface)))
The following code would be the UFFI version. It compiles, except for
the fact that I do not have SDL installed and the linker complains
about a missing symbol. I presume that the cl-sdl macros do not expand
properly or have some eval-when (:compile-toplevel) statements
missing.
(ffi:def-struct surface
(flags uint32)
(format (:struct-pointer pixel-format))
(w int)
(h int)
(pitch uint16)
(pixels :pointer-void)
(offset int)
(hwdata :pointer-void) ; internally struct (opaque pointer)
(clip-rect (:struct rect))
(unused uint32)
(locked uint32)
(map :pointer-void) ; internally struct (opaque pointer)
(format-version unsigned-int)
(ref-count int))
(ffi:def-function ("SDL_Flip" flip) ((screen int) (surf (* surface)))
:returning int)
Juanjo
--
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com

On Tue, 21 Jul 2009 10:04:12 +0200
Juan Jose Garcia-Ripoll <juanjose.garciaripoll@...> wrote:
> The following code would be the UFFI version. It compiles, except for
> the fact that I do not have SDL installed and the linker complains
> about a missing symbol. I presume that the cl-sdl macros do not expand
> properly or have some eval-when (:compile-toplevel) statements
> missing.
Thanks for taking the time to reply with details.
--
Matt

Juan Jose Garcia-Ripoll
<juanjose.garciaripoll@...> writes:
> CFFI has some weird assumptions about what a foreign function
> interface can and can not do. It assumes everything has to be dynamic
> while ECL prefers to compile things (the resulting code is more
> efficient)
CFFI is indeed inspired on the FFIs of SBCL, CCL, Allegro, Lispworks,
etc, which have similar implementation strategies, and are all very
different from ECL.
> does not care that functions with varargs ("..." in C) may
> be different from other functions (making our compiled code really
> really unportable)
Does ECL's DFFI have any sort of varargs functionality? If so, we could
use it. How is UFFI any different in that regard BTW?
> It was never built with ECL in mind and development never accepted a
> twist in this direction, that made my initial port very difficult and
> I have never had time to continue supporting it.
We're certainly open to patches/suggestions that help us better support
ECL's static and dynamic FFIs.
Cheers,
--
Luís Oliveira
http://student.dei.uc.pt/~lmoliv/

On Wed, Jul 22, 2009 at 4:49 AM, Luis Oliveira<luismbo@...> wrote:
> Juan Jose Garcia-Ripoll
> <juanjose.garciaripoll@...> writes:
>
>> CFFI has some weird assumptions about what a foreign function
>> interface can and can not do. It assumes everything has to be dynamic
>> while ECL prefers to compile things (the resulting code is more
>> efficient)
>
> CFFI is indeed inspired on the FFIs of SBCL, CCL, Allegro, Lispworks,
> etc, which have similar implementation strategies, and are all very
> different from ECL.
The fact that many lisp implementations opt for some FFI strategy does
not mean that is the ideal or the only possibility and it makes our
life (ECL maintainers) more difficult. UFFI, for instance, has a
simpler set of function types and arguments it supports (pointers,
ints, floats, etc) because they are less likely to change with the
ABI.
Work on ECL's DFFI only began because of CFFI. DFFI is a mixture of
assembly and C code to generate dynamically function calls to foreign
functions. I must say I myself do not like it. I was only able to code
the x86/32 bit version, M. Goffioul hacked the x64_64 bits port
without even knowing x86_64 assembler. But that is all. Should we want
to use it on other platforms I would need to contact a skilled (or
reckless) coder that does it for us. But even then I would not trust
it as much as the code generated by a C compiler, which does know the
ABI very well, including the choices of registers it does when passing
structures and other things.
The fact that ECL is forced to use DFFI because all other lisps do is
kind of awkward. ECL uses a C compiler, my gosh, that should put us on
an advantage position, having all the type information, being able to
code a FFI call with just one line of C, no need for building callback
code on the fly, or other hacks... However, as it is, with CFFI this
has not been possible.
>> does not care that functions with varargs ("..." in C) may
>> be different from other functions (making our compiled code really
>> really unportable)
>
> Does ECL's DFFI have any sort of varargs functionality? If so, we could
> use it. How is UFFI any different in that regard BTW?
I just cited an example back from my recollections when initially
working on the CFFI port. CFFI dropped all type information, varargs
was completely ignored, this and other problems made it impossible for
us to produce a single C function call that was type accurate and
could be used by the compiler. You can see traces of the problems in
the current cffi-ecl port, with code like
(let ((types (mapcar #'ecl-type->c-type types)))
;; On AMD64, the following code only works with the extra
;; argument ",...". If this is not present, functions
;; like sprintf do not work
which is due to the fact that C functions are not really called with
the types they should.
To this you should add that CFFI only contemplates dynamically loaded
libraries, no possibility of letting a proper linker resolve symbols:
one has to use always dlopen, etc, as you said, because other lisps
do. In ECL's UFFI we can just compile in the library.
> We're certainly open to patches/suggestions that help us better support
> ECL's static and dynamic FFIs.
Right now I do not have time to work on that. To begin with, it should
be possible to use CFFI as just a kind of lisp-SWIG, in which files
with foreign function calls are compiled before being loaded. That,
and keeping more accurate information, allowing libraries to be
statically specified, not just dynamically loaded, creating compiler
macros for the foreign functions so that they expand into C-INLINE
calls, etc, would be a start towards a more efficient implementation
of CFFI on ECL (and also on GCL, btw)
Juanjo
--
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com

Juan Jose Garcia-Ripoll
<juanjose.garciaripoll@...> writes:
> UFFI, for instance, has a simpler set of function types and arguments
> it supports (pointers, ints, floats, etc) because they are less likely
> to change with the ABI.
Can you be more specific? What CFFI types are problematic?
> Work on ECL's DFFI only began because of CFFI. DFFI is a mixture of
> assembly and C code to generate dynamically function calls to foreign
> functions. I must say I myself do not like it. I was only able to code
> the x86/32 bit version, M. Goffioul hacked the x64_64 bits port
> without even knowing x86_64 assembler. But that is all. Should we want
> to use it on other platforms I would need to contact a skilled (or
> reckless) coder that does it for us. But even then I would not trust
> it as much as the code generated by a C compiler, which does know the
> ABI very well, including the choices of registers it does when passing
> structures and other things.
Perhaps using libffi (for DFFI) would be a good, solid approach?
<http://sourceware.org/libffi/&gt;
> The fact that ECL is forced to use DFFI because all other lisps do is
> kind of awkward. ECL uses a C compiler, my gosh, that should put us on
> an advantage position, having all the type information, being able to
> code a FFI call with just one line of C, no need for building callback
> code on the fly, or other hacks... However, as it is, with CFFI this
> has not been possible.
Having a dynamic FFI seems useful for interpreted code, for libraries
that for some reason or another really do have to be accessed via
dlopen/dlsym, and during development -- e.g., using the REPL.
BTW, regarding the OP's problem, some UFFI libraries such as CLSQL pass
non-constant arguments to UFFI:LOAD-FOREIGN-LIBRARY as well. I assume
that would be a problem for ECL as well.
> I just cited an example back from my recollections when initially
> working on the CFFI port. CFFI dropped all type information, varargs
> was completely ignored, this and other problems made it impossible for
> us to produce a single C function call that was type accurate and
> could be used by the compiler.
At what point does CFFI "drop all type information"? That doesn't sound
right. Again, I'm not aware of any way to define varargs functions in
UFFI/DFFI. If there were, CFFI could use it.
> To this you should add that CFFI only contemplates dynamically loaded
> libraries, no possibility of letting a proper linker resolve symbols:
> one has to use always dlopen, etc, as you said, because other lisps
> do. In ECL's UFFI we can just compile in the library.
I'm pretty sure I've done that with CFFI and ECL on darwin/ppc, so I
don't think that claim is correct.
>> We're certainly open to patches/suggestions that help us better support
>> ECL's static and dynamic FFIs.
>
> Right now I do not have time to work on that. To begin with, it should
> be possible to use CFFI as just a kind of lisp-SWIG, in which files
> with foreign function calls are compiled before being loaded.
That's how CFFI-using libraries are usually loaded, so no problem
there. (I should mention again that having a dynamic FFI seems useful
nevertheless.)
> That, and keeping more accurate information
What information?
> allowing libraries to be statically specified
The only problem I see here is how to specify alternatives.
E.g.: (cffi:load-foreign-library '(:or "libfoo.so.1" "libfoo.so"))
How can we do this why ECL's static FFI?
--
Luís Oliveira
http://student.dei.uc.pt/~lmoliv/

> Having a dynamic FFI seems useful for interpreted code, for libraries
> that for some reason or another really do have to be accessed via
> dlopen/dlsym, and during development -- e.g., using the REPL.
Current ECL ffi (c-inline and clines based) can be used from REPL. One
only need to (compile 'fname) after function definition.
always
It might be useful, not only for this case, to add *always-compile*
feature to ECL to compile function automatically immediately after
definition.
Regards,
Marko

On Wed, Jul 22, 2009 at 3:12 PM, Luis Oliveira<luismbo@...> wrote:
>> UFFI, for instance, has a simpler set of function types and arguments
>> it supports (pointers, ints, floats, etc) because they are less likely
>> to change with the ABI.
>
> Can you be more specific? What CFFI types are problematic?
As I said, passing structures and varargs is not supported by UFFI. To
remove this limitation in a robust and portable way,
> Perhaps using libffi (for DFFI) would be a good, solid approach?
> <http://sourceware.org/libffi/&gt;
I do not know about libffi. Is it just a child of libffcall? Because
that one is no less a hack than our code is, but that is not the
point. Those libraries, just like DFFI, represent an unnecessary
overhead for something that could be done efficiently in ECL.
> Having a dynamic FFI seems useful for interpreted code, for libraries
> that for some reason or another really do have to be accessed via
> dlopen/dlsym, and during development -- e.g., using the REPL.
Yes, but many people are happy with SWIG, even in interpreted
environments (scheme, python, etc) and that is not dynamic at all. As
another user pointed out, you can always compile a particular FFI you
need. CFFI could allow both development models, focusing on the static
one for platforms, such as GCL or ECL, on which this might be
advantageous.
> BTW, regarding the OP's problem, some UFFI libraries such as CLSQL pass
> non-constant arguments to UFFI:LOAD-FOREIGN-LIBRARY as well. I assume
> that would be a problem for ECL as well.
That is not the only possibility. When the library is not constant we
can resort to the non-constant version, which will be much slower, but
UFFI is simple enough that we can intercept the name of the library
and pass it to the linker when the file using UFFI is compiled. This
could also be done with CFFI, but its backend only provides pointers,
not really names.
>> I just cited an example back from my recollections when initially
>> working on the CFFI port. CFFI dropped all type information, varargs
>> was completely ignored, this and other problems made it impossible for
>> us to produce a single C function call that was type accurate and
>> could be used by the compiler.
>
> At what point does CFFI "drop all type information"? That doesn't sound
> right. Again, I'm not aware of any way to define varargs functions in
> UFFI/DFFI. If there were, CFFI could use it.
UFFI does not vararg functions it because it is not really portable.
Even on the same platform different compilers may choose different
conventions. But in any case, CFFI does not make use of the vararg
information either. The code that generates the function call in the
CFFI-ECL port has to compile _all_ function calls as varargs and hope
that works. Please have a look at it. The reason is that the CFFI
backend has not been passed that information to the code that
generates the function call. I explain it below.
>> To this you should add that CFFI only contemplates dynamically loaded
>> libraries, no possibility of letting a proper linker resolve symbols:
>> one has to use always dlopen, etc, as you said, because other lisps
>> do. In ECL's UFFI we can just compile in the library.
>
> I'm pretty sure I've done that with CFFI and ECL on darwin/ppc, so I
> don't think that claim is correct.
As I said, I do not know the full status of CFFI as of now. I am
talking about what I saw at the time and what I have read by a cursory
inspection of the code you have at the repo right now.
>> Right now I do not have time to work on that. To begin with, it should
>> be possible to use CFFI as just a kind of lisp-SWIG, in which files
>> with foreign function calls are compiled before being loaded.
>
> That's how CFFI-using libraries are usually loaded, so no problem
> there. (I should mention again that having a dynamic FFI seems useful
> nevertheless.)
In this case, as I said before, if instead of using the prototypes
based on dlopen + dlsym (I mean loading library and searching for
symbols), if the current backend was allowed to use the C names of the
functions and produce a C code that is compiled once and used, then I
believe it should be possible to create better prototypes and even
keep compiler macros that expand later calls, making them more
efficient.
>> That, and keeping more accurate information
>
> What information?
The accurate list of argument types and prototype of a function, such
that we can produce an valid C function call. Currently CFFI uses a
PRODUCE-FUNCTION-POINTER-CALL that takes a pointer and the types of
the arguments the user declares to pass, not the actual declaration of
the function. Hence if you call SPRINTF with a string and a float you
are not using the fact that it takes a variable number of arguments,
as I said before, but rather treating it as a function that takes a
string and a float, which is incorrect.
Ideally, CFFI should also allow specifying C headers that provide the
same information and let the backend either use the header or the type
information which is provided.
>> allowing libraries to be statically specified
>
> The only problem I see here is how to specify alternatives.
> E.g.: (cffi:load-foreign-library '(:or "libfoo.so.1" "libfoo.so"))
> How can we do this why ECL's static FFI?
Is it really such a big problem in the C world? Don't we have -l flags
that take care of resolving library names? The backend could be taught
to strip the library names, passing the search path to the linker
together with a list of -lA -lB ... etc
Juanjo
--
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com

On Wed, Jul 22, 2009 at 9:34 AM, Juan Jose
Garcia-Ripoll<juanjose.garciaripoll@...> wrote:
>> Perhaps using libffi (for DFFI) would be a good, solid approach?
>> <http://sourceware.org/libffi/&gt;
>
> I do not know about libffi. Is it just a child of libffcall? Because
> that one is no less a hack than our code is, but that is not the
> point. Those libraries, just like DFFI, represent an unnecessary
> overhead for something that could be done efficiently in ECL.
I emphatically agree.
ECL's way of doing things may look odd, but when you aim at
portability and don't control native code generation as SBCL does,
you ought to aim for simplicity and efficiency while actively
supporting the most advanced uses. Wherever the function names and
arities are known, it should not take more indirections than it would
in normal C code.
-- Gaby

On Wed, 22 Jul 2009, Juan Jose Garcia-Ripoll wrote:
>> The only problem I see here is how to specify alternatives.
>> E.g.: (cffi:load-foreign-library '(:or "libfoo.so.1" "libfoo.so"))
>> How can we do this why ECL's static FFI?
>
> Is it really such a big problem in the C world? Don't we have -l flags
> that take care of resolving library names? The backend could be taught
> to strip the library names, passing the search path to the linker
> together with a list of -lA -lB ... etc
I have seen recent Redhat/Fedora installs and security updates with
/usr/lib/libXYZ.so.N but no symlinks for /usr/lib/libXYZ.so, where XYZ
takes values like freetype. Thus -lfreetype fails on these platforms. I
have no idea why they chose to do this.
- Daniel

On Fri, Jul 24, 2009 at 8:21 AM, Daniel Herring<dherring@...> wrote:
>
> I have seen recent Redhat/Fedora installs and security updates with
> /usr/lib/libXYZ.so.N but no symlinks for /usr/lib/libXYZ.so, where XYZ
> takes values like freetype. Thus -lfreetype fails on these platforms. I
Do they have development and runtime versions? On Debian if you do not
install the development version, files for linking and compiling might
be missing.
Juanjo
--
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com

Juan Jose Garcia-Ripoll
<juanjose.garciaripoll@...> writes:
>> Can you be more specific? What CFFI types are problematic?
>
> As I said, passing structures and varargs is not supported by UFFI.
CFFI doesn't support passing structures by value either, I'm
afraid. Liam Healy has implemented such support using CFFI bindings to
libffi, though.
<http://common-lisp.net/pipermail/cffi-devel/2009-April/003090.html&gt;
Varargs is indeed not very well supported by most backends, but seems to
work well enough on x86 (and PPC, IIRC) for stuff that really needs it,
like cl-objc. However, as you mention, the cffi-ecl backend is in a
perfect position to support it properly.
>> Perhaps using libffi (for DFFI) would be a good, solid approach?
>> <http://sourceware.org/libffi/&gt;
>
> I do not know about libffi. Is it just a child of libffcall? Because
> that one is no less a hack than our code is, but that is not the
> point. Those libraries, just like DFFI, represent an unnecessary
> overhead for something that could be done efficiently in ECL.
There doesn't seem to be a relation between libffi and libffcall but I
wouldn't be surprised if the implementations are similar. It seems that
the advantage in using libffi -- as opposed to rolling your own -- is
that it is widely used by other language implementations such as
CPython, Squeak and Java Native Access.
> CFFI could allow both development models, focusing on the static
> one for platforms, such as GCL or ECL, on which this might be
> advantageous.
Sure.
>> BTW, regarding the OP's problem, some UFFI libraries such as CLSQL pass
>> non-constant arguments to UFFI:LOAD-FOREIGN-LIBRARY as well. I assume
>> that would be a problem for ECL as well.
>
> That is not the only possibility. When the library is not constant we
> can resort to the non-constant version, which will be much slower, but
> UFFI is simple enough that we can intercept the name of the library
> and pass it to the linker when the file using UFFI is compiled.
> This could also be done with CFFI, but its backend only provides
> pointers, not really names.
The cffi-ecl backend discards the name for some reason I don't
understand.
> UFFI does not vararg functions it because it is not really portable.
> Even on the same platform different compilers may choose different
> conventions. But in any case, CFFI does not make use of the vararg
> information either. The code that generates the function call in the
> CFFI-ECL port has to compile _all_ function calls as varargs and hope
> that works. Please have a look at it. The reason is that the CFFI
> backend has not been passed that information to the code that
> generates the function call. I explain it below.
Indeed not trying to support varargs is much easier than trying to. :-)
But we can fix this for cffi-ecl very easily by passing the varargs
information to cffi-sys:%foreign-funcall since the mechanism for passing
other similar options is already there.
> In this case, as I said before, if instead of using the prototypes
> based on dlopen + dlsym (I mean loading library and searching for
> symbols), if the current backend was allowed to use the C names of the
> functions and produce a C code that is compiled once and used, then I
> believe it should be possible to create better prototypes and even
> keep compiler macros that expand later calls, making them more
> efficient.
I have no idea why the cffi-ecl backend doesn't use the C names. It
certainly has access to them.
> Ideally, CFFI should also allow specifying C headers that provide the
> same information and let the backend either use the header or the type
> information which is provided.
That sounds like a good recipe to have CFFI code only work properly on
ECL but not on the other Lisps.
> Is it really such a big problem in the C world? Don't we have -l flags
> that take care of resolving library names? The backend could be taught
> to strip the library names, passing the search path to the linker
> together with a list of -lA -lB ... etc
Good point. I'll look at ECL's FFI:LOAD-FOREIGN-LIBRARY to see how it
works.
It seems to me that your objections to CFFI pertain to the current
implementation of the cffi-ecl backend and not to the architecture of
CFFI per se. I suppose that's good news since the shortcomings you
described sound straightforward to fix.
Thanks for your comments,
--
Luís Oliveira
http://student.dei.uc.pt/~lmoliv/