clisp-list

This is rube goldbergian. I know there are a 100 better ways to solve
this "problem" but my son wants to do it a particular way and I think we
are pretty close.
We've got a Raspberry Pi. He wants to write a program to run on it in
lisp. But he doesn't want to have to install Linux and run in
userspace. He wants his program to be the operating system, or at least
the only thing running (as another project he wants to boot right into a
game written in lisp).
We are already able to cross-compile C/ASM from i386 to ARM. Today I
figured out how to use clisp to compile lisp to our intel/amd machines
(we are both running Debian, btw). I know clisp runs on the arm
chip(s). What I want to be able to do is cross-compile to arm with
clisp. Is this possible?
Do I have to tell clisp where my cross-compilation toolchain is? Or will
it only build binaries for the host architecture?

David Rysdam <david@...> writes:
> This is rube goldbergian. I know there are a 100 better ways to solve
> this "problem" but my son wants to do it a particular way and I think we
> are pretty close.
>
> We've got a Raspberry Pi. He wants to write a program to run on it in
> lisp. But he doesn't want to have to install Linux and run in
> userspace. He wants his program to be the operating system, or at least
> the only thing running (as another project he wants to boot right into a
> game written in lisp).
AFAIK, it won't be simple to make clisp run on bare metal.
Also, I don't remember anybody working on this problem.
Somebody has worked on making sbcl run on bare metal.
But this probably involved ix86, I'm not sure how this work would be
useful for ARM.
The CL implementation that runs on bare metal and which is designed to
write OS in, is Movitz. But again, it's ix86 specific. You'd have to
add an ARM backend to the Movitz compiler.
Finally, there are also ccl and ecl that are able to run on ARM. CCL
runs quite well there (as long as it's the right version of ARM). But
like clisp, they both require a linux kernel, making them run on the
bare metal would involve the same work as with clisp.
This work consists in implementing the POSIX features that are used and
necessary to the implementation.
The easiest path to boot a CL running on the bare hardware on ARM today,
would be, IMO, to use an ix86/arm emulator, and to run Movitz on it.
Yes, it would not run at top-speed, but it would be usable and doable
quickly enough.
Have a look at this project:
http://dmitry.gr/index.php?r=05.Projects&proj=07.%20Linux%20on%208bit
Using a ix86/arm emulator to boot Movitz would be much easier than that.
Otherwise, making clisp bootable on the bare hardware would have the
advantage of being able to target any processor for which we have a C
compiler.
On the other hand, making ccl bootable on the bare hardware would give
you a better ARM compiler.
I don't know your objective, but it may still be a good enough plan to
run the lisp implementation on a bare Linux kernel. Just something
like: https://www.informatimago.com/linux/emacs-on-user-mode-linux.html
With a CL implementation you wouldn't need the external mount(1)
command, since we can do that with a syscall thru CFFI.
Running clisp or ccl on a "bare" linux this way can be archived in an
afternoon, and then you can have fun developping the rest of the OS (not
the kernel, but the more fun parts, including AI user interface).
> We are already able to cross-compile C/ASM from i386 to ARM. Today I
> figured out how to use clisp to compile lisp to our intel/amd machines
> (we are both running Debian, btw). I know clisp runs on the arm
> chip(s). What I want to be able to do is cross-compile to arm with
> clisp. Is this possible?
More than just cross-compile, you want also to target the bare
hardware. This means that you want to use the tool chain used to
compile kernels (like linux), not the tool chain used to compile user
space programs.
Here is a good resource for kernel writers:
http://wiki.osdev.org/Main_Page
Of course, if you don't want to use linux, you may also want to avoid
grub, and rather write your own boot loader in lisp too ;-)
> Do I have to tell clisp where my cross-compilation toolchain is? Or will
> it only build binaries for the host architecture?
ISTR that it's possible. In any case, it shouldn't be too hard to do,
just specifying CC as an environment variable to configure. See the
tail of:
./configure --help
--
__Pascal Bourguignon__ http://www.informatimago.com/
A bad day in () is better than a good day in {}.
You know you've been lisping too long when you see a recent picture of George
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

"Pascal J. Bourguignon" <pjb@...> writes:
> David Rysdam <david@...> writes:
>
>> This is rube goldbergian. I know there are a 100 better ways to solve
>> this "problem" but my son wants to do it a particular way and I think we
>> are pretty close.
>>
>> We've got a Raspberry Pi. He wants to write a program to run on it in
>> lisp. But he doesn't want to have to install Linux and run in
>> userspace. He wants his program to be the operating system, or at least
>> the only thing running (as another project he wants to boot right into a
>> game written in lisp).
>
> AFAIK, it won't be simple to make clisp run on bare metal.
> Also, I don't remember anybody working on this problem.
I hadn't even considered booting to CL. That could also be cool. Thanks
for the ton of info on that idea.
>> We are already able to cross-compile C/ASM from i386 to ARM. Today I
>> figured out how to use clisp to compile lisp to our intel/amd machines
>> (we are both running Debian, btw). I know clisp runs on the arm
>> chip(s). What I want to be able to do is cross-compile to arm with
>> clisp. Is this possible?
>
> More than just cross-compile, you want also to target the bare
> hardware. This means that you want to use the tool chain used to
> compile kernels (like linux), not the tool chain used to compile user
> space programs.
>
> Here is a good resource for kernel writers:
> http://wiki.osdev.org/Main_Page
>
> Of course, if you don't want to use linux, you may also want to avoid
> grub, and rather write your own boot loader in lisp too ;-)
Yes. Or maybe. I was kind of imagining a simple bootloader in asm, based
on some info here:
http://www.cl.cam.ac.uk/projects/raspberrypi/tutorials/os/#lessons
But doing everything in relatively pure Lisp would be interesting.
>> Do I have to tell clisp where my cross-compilation toolchain is? Or will
>> it only build binaries for the host architecture?
>
> ISTR that it's possible. In any case, it shouldn't be too hard to do,
> just specifying CC as an environment variable to configure. See the
> tail of:
>
> ./configure --help
I'm not convinced we are talking about the same thing, so let me be
clear: clisp running natively on x86, producing a binary for arm.
I can do this with a native x86 gcc producing arm binaries. But I don't
know how clisp produces it's binaries. By calling CC? Or does it do it
itself?

David Rysdam <david@...> writes:
> "Pascal J. Bourguignon" <pjb@...> writes:
>> David Rysdam <david@...> writes:
>>
>>> This is rube goldbergian. I know there are a 100 better ways to solve
>>> this "problem" but my son wants to do it a particular way and I think we
>>> are pretty close.
>>>
>>> We've got a Raspberry Pi. He wants to write a program to run on it in
>>> lisp. But he doesn't want to have to install Linux and run in
>>> userspace. He wants his program to be the operating system, or at least
>>> the only thing running (as another project he wants to boot right into a
>>> game written in lisp).
>>
>> AFAIK, it won't be simple to make clisp run on bare metal.
>> Also, I don't remember anybody working on this problem.
>
> I hadn't even considered booting to CL. That could also be cool. Thanks
> for the ton of info on that idea.
Oh, so you meant to write a kernel, but not necessarily to include CL in
that kernel?
Then you don't need to cross compile the CL implementation to ARM. You
only need to have an ARM code generator written in lisp. You can easily
write one (since ARM has quite simple and orthogonal instruction sets),
or you can find one in free
software. http://www.cliki.net/machine%20code
You could also perhaps use an ARM code generator found in a CL compiler
like in ccl.
Then you design a language (lisp-like) that use this code generator to
generate the ARM binaries you need to boot your computer.
While your lisp-like language can be compiled to ARM, you can also very
easily write functions and macros to have it run directly in CL, so you
can also develop and debug the program in the lisp environment.
Now, I say lisp-like, it could be a superset of some subset of Common
Lisp itself. A subset because you may not want, and don't need to
implement a whole CL implementation, to write an OS. A superset,
because you may need some primitives not provided by CL, like register
memory, or device accesses.
Actually, you could just use ccl, by modifying its code generator, so
that you can generate ARM binary files in the format you need to boot
your kernel, instead of saving the usual .fasl files ccl saves when it
works on normal systems. But you'd still have to do the work of
implementing the dependencies that you'll lack on the bare metal, or
take extreme care not to use any of CL in your system.
See what I mean. If you write a little function to keep track of
processes:
------------------------------------------------------------------------
(defpackage "MYOS-PROCESSES"
(:use "CL") ; dangerous. it would be better to (:use "MYOS-LISP")
(:export "ADD-PROCESS"))
(in-package "MYOS-PROCESSES")
(defvar *processes* '())
(defun add-process (new-process)
(push new-process *processes*))
------------------------------------------------------------------------
Now, when we compile this, we get a .fasl file that contains
instructions to create the *PROCESSES* variable and initialize it to
NIL, and to create a function named ADD-PROCESS, that will call the
function CL:CONS. You can generate it in your specific binary file
format to load it as a kernel.
But if you want to run this you will need to include a function named
CONS (as well, as all the internal functions used to create global
special variables, functions, to manage symbol bindings, etc).
The function CONS is rather simple, but it calls out to the garbage
collector, so you will have to include also the garbage collector.
The CL garbage collector may report memory collected, or may signal an
STORAGE-CONDITION, so you need to include PRINC, TERPRI, and the
condition system. When a STORAGE-CONDITION is signaled, if it is not
handled, then the debugger is called, so you need to include the
debugger along. The default debugger uses the I/O system (print, read,
etc), so you need to include the lisp printer, the lisp reader. It then
evaluates the expressions it read, so you need to include EVAL. That
is, you need to include an interpreter or a compiler. Since the user
may want to use any CL operator, and you may want to provide them this
ability to use any CL operator to debug their kernel code, then you will
want to include all the CL system.
Therefore you just want to modify an implementation that has an ARM
compiler to be able to generate a kernel image instead of generating a
lisp image or an ELF executable image.
A kernel image, is just a binary blob that can be loaded by a boot
loader, it can be in any format. (Including ELF, so the modifications
are not necessarily big on this part).
But since you get the whole CL implementation, you must re-implement the
parts that depend on linux, since you won't have linux. Instead of
using write(2) to print bytes to the terminal, you must call the
function that will output the characters to the device you use to show
text (it can be a serial chip, a USB device, a frame buffer bit map,
whatever). Since you don't have open(2) to open files, you need to
write the function and the device driver that will read data from the
sdcard, or a UBS disk, etc.
I would advise you to try out Movitz, and see how it's organized.
>>> We are already able to cross-compile C/ASM from i386 to ARM. Today I
>>> figured out how to use clisp to compile lisp to our intel/amd machines
>>> (we are both running Debian, btw). I know clisp runs on the arm
>>> chip(s). What I want to be able to do is cross-compile to arm with
>>> clisp. Is this possible?
>>
>> More than just cross-compile, you want also to target the bare
>> hardware. This means that you want to use the tool chain used to
>> compile kernels (like linux), not the tool chain used to compile user
>> space programs.
>>
>> Here is a good resource for kernel writers:
>> http://wiki.osdev.org/Main_Page
>>
>> Of course, if you don't want to use linux, you may also want to avoid
>> grub, and rather write your own boot loader in lisp too ;-)
>
> Yes. Or maybe. I was kind of imagining a simple bootloader in asm, based
> on some info here:
>
> http://www.cl.cam.ac.uk/projects/raspberrypi/tutorials/os/#lessons
>
> But doing everything in relatively pure Lisp would be interesting.
>
>>> Do I have to tell clisp where my cross-compilation toolchain is? Or will
>>> it only build binaries for the host architecture?
>>
>> ISTR that it's possible. In any case, it shouldn't be too hard to do,
>> just specifying CC as an environment variable to configure. See the
>> tail of:
>>
>> ./configure --help
>
> I'm not convinced we are talking about the same thing, so let me be
> clear: clisp running natively on x86, producing a binary for arm.
>
> I can do this with a native x86 gcc producing arm binaries. But I don't
> know how clisp produces it's binaries. By calling CC? Or does it do it
> itself?
clisp is written in C, hence the "c" in "clisp".
So indeed, it runs the compiler referenced in the environment variable
CC, when there's one, or some other default (cc or gcc) otherwise.
clisp has an interpreter (written in C), and it has a virtual machine
(written in C) that interprets a lisp-friendly byte-code, and it has a
lisp compiler (written in lisp) that generates this lisp-friendly
byte-code (you can see that with CL:DISASSEMBLE or in the .fas files).
ccl has native compilers for various processors (x86, ppc, arm). It is
written almost entirely in lisp, plus some assembler and some C.
ecl has an interpreter, a byte code virtual machine and a byte-code
compiler, and a native compiler that works by generating C code, fed to
gcc to produce elf .o files that are dynamically loaded into the ecl
process. ecl is almost entirely written in lisp (plus some C). You can
run ecl without gcc, but then you can't compile to native code.
sbcl is like ccl, but it also has an interpreter, which is not used by
default but that applications may activate in the case it would gives
them a speed boost. Yes, an interpreter let you run code faster than
compilers! (Notably when you have a very careful and slow compiler like
sbcl).
--
__Pascal Bourguignon__ http://www.informatimago.com/
A bad day in () is better than a good day in {}.
You know you've been lisping too long when you see a recent picture of George
Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin

"Pascal J. Bourguignon" <pjb@...> writes:
> David Rysdam <david@...> writes:
>
>> "Pascal J. Bourguignon" <pjb@...> writes:
>>> David Rysdam <david@...> writes:
>>>
>>>> This is rube goldbergian. I know there are a 100 better ways to solve
>>>> this "problem" but my son wants to do it a particular way and I think we
>>>> are pretty close.
>>>>
>>>> We've got a Raspberry Pi. He wants to write a program to run on it in
>>>> lisp. But he doesn't want to have to install Linux and run in
>>>> userspace. He wants his program to be the operating system, or at least
>>>> the only thing running (as another project he wants to boot right into a
>>>> game written in lisp).
>>>
>>> AFAIK, it won't be simple to make clisp run on bare metal.
>>> Also, I don't remember anybody working on this problem.
>>
>> I hadn't even considered booting to CL. That could also be cool. Thanks
>> for the ton of info on that idea.
>
> Oh, so you meant to write a kernel, but not necessarily to include CL in
> that kernel?
Right.
> But since you get the whole CL implementation, you must re-implement the
> parts that depend on linux, since you won't have linux. Instead of
> using write(2) to print bytes to the terminal, you must call the
> function that will output the characters to the device you use to show
> text (it can be a serial chip, a USB device, a frame buffer bit map,
> whatever). Since you don't have open(2) to open files, you need to
> write the function and the device driver that will read data from the
> sdcard, or a UBS disk, etc.
Yes, this is what I was envisioning. I had gotten as far as realizing
I'd need to implement (a small subset of) glibc but hadn't really
realized that I'd have to either do this outside of lisp OR modify my
lisp.
Actually, wait. If there was the equivalent of inline ASM, that would
work. Make a (write)/(open)/etc that are just thin lisp wrappers around
some inline ARM and call those.
> I would advise you to try out Movitz, and see how it's organized.
OK.
>> I can do this with a native x86 gcc producing arm binaries. But I don't
>> know how clisp produces it's binaries. By calling CC? Or does it do it
>> itself?
>
> clisp is written in C, hence the "c" in "clisp".
>
> So indeed, it runs the compiler referenced in the environment variable
> CC, when there's one, or some other default (cc or gcc) otherwise.
>
>
> clisp has an interpreter (written in C), and it has a virtual machine
> (written in C) that interprets a lisp-friendly byte-code, and it has a
> lisp compiler (written in lisp) that generates this lisp-friendly
> byte-code (you can see that with CL:DISASSEMBLE or in the .fas files).
>
> ccl has native compilers for various processors (x86, ppc, arm). It is
> written almost entirely in lisp, plus some assembler and some C.
In that case, I think this is what I want.
I think I have enough information to take a few stabs at it, fail
miserably and then come crawling back to ask more intelligent
questions. :) Thanks!