[Sbcl-devel] Getting your feet wet with SBCL hacking

Hi SBCL hackers,
I have downloaded the SBCL source a few times in an attempt to get familiar
with the code so that eventually I can port SBCL to new architectures or
make optimizations. Unfortunately, every time I download the source I have
a hard time getting going and eventually give up. I am writing for tips
about getting started hacking on SBCL.
This is a pretty general inquiry so I will give some questions that might
yield helpful responses:
* How did you get started hacking SBCL?
* Can I avoid knowing much about the bootstrapping process and still learn a
lot about how the compiler works. For example, how (+ 1 2) is compiled into
assembly?
* Was the Internals wiki helpful for first-timers? for me it gives a
high-level understanding of what SBCL is up to, but it doesn't teach me
function names or what files to seek out.
* Are there good pet projects for beginners to work on?
* Does most development happen through the REPL on an active SBCL instance?
I can imagine there is a lot of restarting for some types of work.
* What's the best way you have found to search the mailing list to avoid
repeating questions?
Some of these questions may be dumb, so appologies galor. It just seems
like it will be fun to play around with SBCL every once in a while, but I
don't know how to get started with this massive piece of code.
Also there are some random questions I have about SBCL that have popped into
my mind every now and again:
* How ugly is the source compared to OpenMCL, a fast Scheme implementation,
a commercial lisp implementation, and GCC?
* Has anyone attempted to port to Java bytecode or some other unconventional
assembly-like language?
* Are there portions of the compilation process that are currently obviously
suitable for optimization via machine learning algorithms? I do not mean
really dreamy AI as in "hey computer, optimize my program for me," but more
deciding which functions to inline or something.
Thanks for any answers. I know there are more SBCL would-be hackers like me
that will benefit greatly from this thread.
Red

Thread view

Hi SBCL hackers,
I have downloaded the SBCL source a few times in an attempt to get familiar
with the code so that eventually I can port SBCL to new architectures or
make optimizations. Unfortunately, every time I download the source I have
a hard time getting going and eventually give up. I am writing for tips
about getting started hacking on SBCL.
This is a pretty general inquiry so I will give some questions that might
yield helpful responses:
* How did you get started hacking SBCL?
* Can I avoid knowing much about the bootstrapping process and still learn a
lot about how the compiler works. For example, how (+ 1 2) is compiled into
assembly?
* Was the Internals wiki helpful for first-timers? for me it gives a
high-level understanding of what SBCL is up to, but it doesn't teach me
function names or what files to seek out.
* Are there good pet projects for beginners to work on?
* Does most development happen through the REPL on an active SBCL instance?
I can imagine there is a lot of restarting for some types of work.
* What's the best way you have found to search the mailing list to avoid
repeating questions?
Some of these questions may be dumb, so appologies galor. It just seems
like it will be fun to play around with SBCL every once in a while, but I
don't know how to get started with this massive piece of code.
Also there are some random questions I have about SBCL that have popped into
my mind every now and again:
* How ugly is the source compared to OpenMCL, a fast Scheme implementation,
a commercial lisp implementation, and GCC?
* Has anyone attempted to port to Java bytecode or some other unconventional
assembly-like language?
* Are there portions of the compilation process that are currently obviously
suitable for optimization via machine learning algorithms? I do not mean
really dreamy AI as in "hey computer, optimize my program for me," but more
deciding which functions to inline or something.
Thanks for any answers. I know there are more SBCL would-be hackers like me
that will benefit greatly from this thread.
Red

Hi Red,
> Hi SBCL hackers,
I'm not a contributor but I do sometimes dive into the SBCL
sources to find out more about a problem.
So take it all with a grain of salt:
> * How did you get started hacking SBCL?
Be curious and download sbcl.git :)
> * Can I avoid knowing much about the bootstrapping process and still learn a
> lot about how the compiler works. For example, how (+ 1 2) is compiled into
> assembly?
DISASSEMBLE surely is enough to start working on optimization.
> * Was the Internals wiki helpful for first-timers? for me it gives a
> high-level understanding of what SBCL is up to, but it doesn't teach me
> function names or what files to seek out.
Yeah, grep is more helpful here.
Also checkout DESCRIBE which will report the source file
of a built-in function.
> * Are there good pet projects for beginners to work on?
No clue. Likely.
Improving stuff for the Debian Shootout Benchmarks seems
fun to me if you got the time.
> * Does most development happen through the REPL on an active SBCL instance?
> I can imagine there is a lot of restarting for some types of work.
I always work directly on the REPL.
> * What's the best way you have found to search the mailing list to avoid
> repeating questions?
Google, the sf.net interface is obnoxious.
> * How ugly is the source compared to [...] a commercial lisp implementation,
How would one know? They're all closed source.
I haven't looked at the source of the other projects you mentioned,
I'm afraid.
>From what I've seen SBCL's code can be quite pretty (e.g. the printer
functions) but also complicated (irX compiler code).
I haven't seen *really* ugly code in SBCL so far, but it's bound to
be there I guess. ;)
I will leave the other two questions to people more knowledgable than me.
Leslie
--
LinkedIn Profile: http://www.linkedin.com/in/polzer
Xing Profile: https://www.xing.com/profile/LeslieP_Polzer
Blog: http://blog.viridian-project.de/

There's also the sbcl internals wiki: http://sbcl-internals.cliki.net/index
Chaitanya
Red Daly wrote:
> Hi SBCL hackers,
>
> I have downloaded the SBCL source a few times in an attempt to get
> familiar with the code so that eventually I can port SBCL to new
> architectures or make optimizations. Unfortunately, every time I
> download the source I have a hard time getting going and eventually
> give up. I am writing for tips about getting started hacking on SBCL.
>
> This is a pretty general inquiry so I will give some questions that
> might yield helpful responses:
> * How did you get started hacking SBCL?
> * Can I avoid knowing much about the bootstrapping process and still
> learn a lot about how the compiler works. For example, how (+ 1 2) is
> compiled into assembly?
> * Was the Internals wiki helpful for first-timers? for me it gives a
> high-level understanding of what SBCL is up to, but it doesn't teach
> me function names or what files to seek out.
> * Are there good pet projects for beginners to work on?
> * Does most development happen through the REPL on an active SBCL
> instance? I can imagine there is a lot of restarting for some types
> of work.
> * What's the best way you have found to search the mailing list to
> avoid repeating questions?
>
> Some of these questions may be dumb, so appologies galor. It just
> seems like it will be fun to play around with SBCL every once in a
> while, but I don't know how to get started with this massive piece of
> code.
>
> Also there are some random questions I have about SBCL that have
> popped into my mind every now and again:
>
> * How ugly is the source compared to OpenMCL, a fast Scheme
> implementation, a commercial lisp implementation, and GCC?
> * Has anyone attempted to port to Java bytecode or some other
> unconventional assembly-like language?
> * Are there portions of the compilation process that are currently
> obviously suitable for optimization via machine learning algorithms?
> I do not mean really dreamy AI as in "hey computer, optimize my
> program for me," but more deciding which functions to inline or something.
>
> Thanks for any answers. I know there are more SBCL would-be hackers
> like me that will benefit greatly from this thread.
>
> Red
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
> -Strategies to boost innovation and cut costs with open source participation
> -Receive a $600 discount off the registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> ------------------------------------------------------------------------
>
> _______________________________________________
> Sbcl-devel mailing list
> Sbcl-devel@...
> https://lists.sourceforge.net/lists/listinfo/sbcl-devel
>

Chaitanya Gupta wrote:
> There's also the sbcl internals wiki:
> http://sbcl-internals.cliki.net/index
>
> Red Daly wrote:
>> * Was the Internals wiki helpful for first-timers? for me it gives a
>> high-level understanding of what SBCL is up to, but it doesn't teach
>> me function names or what files to seek out.
My bad. I see you already know about the internals wiki..
Chaitanya
>> * Are there good pet projects for beginners to work on?
>> * Does most development happen through the REPL on an active SBCL
>> instance? I can imagine there is a lot of restarting for some types
>> of work.
>> * What's the best way you have found to search the mailing list to
>> avoid repeating questions?
>>
>> Some of these questions may be dumb, so appologies galor. It just
>> seems like it will be fun to play around with SBCL every once in a
>> while, but I don't know how to get started with this massive piece of
>> code.
>>
>> Also there are some random questions I have about SBCL that have
>> popped into my mind every now and again:
>>
>> * How ugly is the source compared to OpenMCL, a fast Scheme
>> implementation, a commercial lisp implementation, and GCC?
>> * Has anyone attempted to port to Java bytecode or some other
>> unconventional assembly-like language?
>> * Are there portions of the compilation process that are currently
>> obviously suitable for optimization via machine learning algorithms?
>> I do not mean really dreamy AI as in "hey computer, optimize my
>> program for me," but more deciding which functions to inline or
>> something.
>>
>> Thanks for any answers. I know there are more SBCL would-be hackers
>> like me that will benefit greatly from this thread.
>>
>> Red
>> ------------------------------------------------------------------------
>>
>> ------------------------------------------------------------------------------
>>
>> Open Source Business Conference (OSBC), March 24-25, 2009, San
>> Francisco, CA
>> -OSBC tackles the biggest issue in open source: Open Sourcing the
>> Enterprise
>> -Strategies to boost innovation and cut costs with open source
>> participation
>> -Receive a $600 discount off the registration fee with the source
>> code: SFAD
>> http://p.sf.net/sfu/XcvMzF8H
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Sbcl-devel mailing list
>> Sbcl-devel@...
>> https://lists.sourceforge.net/lists/listinfo/sbcl-devel
>>
>
>

Some quick notes on my own methods, that are very welcome for optimization.
* How did you get started hacking SBCL?
+ I think trying to know your compiler under the hood makes you write
better code. Also sbcl's many feature like compiling to machine code
makes it complete.
SBCL is the leading platform to do misc hacks on :)
* Can I avoid knowing much about the bootstrapping process and still
learn a lot about how the compiler works. For example, how (+ 1 2) is
compiled into assembly?
SBCL recompiles itself very straight-forward. I say that meaning that
you have stages where you can do tests inbetween.
How much you need to know would depend on what type of hacking is done.
+ To port to a new arch/os a minimal knowledge is needed to bootstrap
hacking attemps. Atleast know that we build and "assemble" an
cold-core that is cold-booted to further inject itself with CL and
finally dump itself as an finished product.
Setup for Cross-compiling (look at wiki) is also good, to learn what
goes where.
+ To explore the FOP enging minimal knowledge of the build process is
needed.
Here you could just take a copy of fop.lisp and load.lisp and change
the package to be able to load them into any sbcl without interfering
with sbcl's own
FASL loader.
+ Dig around in the git archive, some patches are very clean with
complete interfaces in on how to implement things. Those makes up great
learning material.
* Was the Internals wiki helpful for first-timers? for me it gives a
high-level understanding of what SBCL is up to, but it doesn't teach me
function names or what files to seek out.
Somewhat, it seems little outdated if I may say so. But some section
has nice detail-level descriptions.
* Are there good pet projects for beginners to work on?
Sooner or later you would end up using gdb, so I guess that can be a
good start.
Try on a fresh sbcl build: "gdb -x /tmp/z --args ./src/runtime/sbcl
--core output/cold-sbcl.core"
And in the /tmp/z you can put your sbcl things, to start out put:
"break call_into_lisp"
That would let the sbcl-runtime do its thing and if you Continue, it
gives control to lisp function cold-init ( src/cold-init.lisp ).
To try out VOP hacking, that is enough to see if your VOP assembles
good code.
At the cold-init function you can put in test forms that triggers the
VOP code.
Using "si" in gdb to single-step if the code branches, lookup the new
entry-point in output/cold-core.map
Watch out: If you go down in cold-init you will see GC being
initiated. After that the core is "scrambled" and cold-core.map isn't
usable directly.
Also dont put any breakpoints after gc has initiated.
+ Look for isolated things, like that FASL loader (where hacking could
be contained to src/code/fop.lisp, src/code/load.lisp,
src/compile/dump.lisp and little of genesis.lisp ).
* Does most development happen through the REPL on an active SBCL
instance? I can imagine there is a lot of restarting for some types of
work.
+ Alot of stuff is probably reloadable in sbcl. Havn't figured that
out yet. But I guess doing a port involves around 200-400 builds from
scratch.
* What's the best way you have found to search the mailing list to avoid
repeating questions?
+ IRC is quicker when you are stuck :)
* How ugly is the source compared to OpenMCL, a fast Scheme
implementation, a commercial lisp implementation, and GCC?
+ Dont know about the others, but SBCL is the standard that I try to
follow when it comes to design, code-documentation etc..
* Has anyone attempted to port to Java bytecode or some other
unconventional assembly-like language?
+ Also LLVM or maybe S/390 would be interesting to hear if it is
feasible to port to.
* Are there portions of the compilation process that are currently
obviously suitable for optimization via machine learning algorithms?
+ I guess the thing that produces "IR1" and converts to "IR2" is
itself a big chapter. Luckily we can look at "IR2" using trace-files
to know if the compiler is behaving.
best regards,
/larry