A previous post covered the subject of the initial LLDB and Dylan
integration.

While much of this post discusses what we did to improve Dylan integration,
the overall techniques are broadly useful for many languages. In fact,
if anyone is interested, I am available for consulting on this topic.
Just email me.

Catching Up

In the previous post, we talked about changing the base representation
from void* to uintptr_t to work around a bug in LLDB. This was,
in part, due to our goal to try to have the integration work with the
currently shipping version of LLDB.

Unfortunately, this work won't be completed and merged. In separate
developments, it has emerged that it can be unsafe to treat a pointer
as an integer when working with a garbage collector. The details
of this are quite interesting, so out of an abundance of caution,
we will not be going down this route.

Expanding Objects

We are able to use the "pointer depth" parameter to commands like
expression and frame variable to show the "expanded" form of
objects:

As we look to the future and what we would like for Dylan to become and
investigate how we would like for Dylan to evolve, it is helpful to look
at some of the current work and how Dylan compares, where Dylan falls
down and whether or not we can improve it.

One of those areas is in the guarantees offered by the type system. While
Dylan is seen as a dynamic language, it has a number of features that
help provide optional static type checking. As we'll see, there is a lot
of room for improvement in this area.

In this post, using a missing compile-time warning as the driver, we'll
walk through some details of the Dylan type system and then see how it
differs from a gradually typed system. We'll see that type annotations
are interpreted very differently under a gradual typing regime versus
the Open Dylan compiler.

I've previously written a Type System Overview which may be useful,
but hopefully this post can stand on its own.

Dylan is often seen as a descendent of the Lisp family of languages. It
was designed (and implemented) by people from the Common Lisp world and
borrowed lots from Scheme, EuLisp and other Lisp dialects. On the
other hand, it was given a syntax from the ALGOL tradition rather than
using s-expressions.

It is interesting to compare some of what is present within the Dylan
language design with ALGOL though and see if perhaps some of the
ALGOL influence wasn't just in the syntax. It is difficult today to
know the extent to which ALGOL 68 was a direct influence versus having
been filtered through Common Lisp and other languages. After all, Dylan
was designed some 25 years or so after ALGOL 68. (At the time of this
writing, Dylan itself is over 20 years old.)

It is clear though that many languages have picked up ideas from
ALGOL over the years, including languages that were direct influencers
of Dylan such as Scheme.

This post isn't intended to say "this concept came from ALGOL" but just
to look at some of the interesting similarities.

Orthogonal Design

Many compilers are now being outfitted with LLVM backends. There are
a variety of ways to do this, and we'll take a look at some of them
here.

Calling LLVM APIs Directly

One common and easy way to integrate with LLVM is to directly link
against the LLVM libraries and invoke the LLVM API directly.

C++ API

LLVM's native APIs are C++. Examples of languages that talk to LLVM
via the C++ APIs are Julia and CLASP.

This is also the approach taken by Clang, however, Clang lives
within the same code repository as LLVM and shares many developers.
It is part of the LLVM project rather than a separate compiler
that is using LLVM as a backend.

While this seems like an attractive option, there are some issues
with it:

Your code must either be written in C++ or be able to invoke C++
code via an FFI (foreign function interface).

You are tied to a particular version of the LLVM API at compile
time.

It is harder to re-use someone's existing installation of LLVM
as you are tied to a particular version.

C API

This begins a new series of blog posts that will continue over the next
few months as we say "Good-bye!" to parts of Open Dylan.

Freedom comes when you learn to let go
Creation comes when you learn to say no
-- Madonna, The Power of Good-Bye

We're beginning a process by which we'll start slimming down the compiler
and the libraries, letting go of some major chunks of code, with the goal
of improving the hackability of the compiler and enabling us to make
new leaps in functionality.

What is HARP?

HARP is the Harlequin Abstract RISC Processor and was designed and developed
at Harlequin in the late 1980s. It was used in Harlequin's LispWorks and
later translated to Dylan for use in Harlequin's DylanWorks (which is now
Open Dylan).

The compiler targeted an instruction set known as HARP (Harlequin
Abstract RISC Processor), and then HARP instructions were translated
into machine instructions using a template matching scheme. HARP had
an infinite set of registers, and the register colouring happened as
part of this templating processing.

The type system and how it is used is a commonly misunderstood aspect
of the Dylan language. Although it lacks some forms of expressiveness
in the current incarnation, it also has some features that aren't
found in many languages, such as singleton types. It is also very
important in helping the compiler to generate faster yet still safe code.

One interesting feature in Dylan is that it is optionally typed. While
this is more common today and sometimes has fancy names applied like
'gradually typed', the overall point is the same: Your code can start
out untyped and looking like code does in Ruby or Python. However,
when you want or need additional performance or correctness guarantees,
you can supply type annotations that the compiler can use. The compiler
can also infer some types from the values used or other type annotations.

In this post, we'll explain some of the basic concepts of the Dylan
type system and show how it is used by the compiler.

Type and Value Relationships

There are 2 important relationships between values and types in Dylan.

They are instance? and subtype?. Other relationships, such as
known-disjoint? are used within the compiler to assist ...

Moving towards Dylan 2016, the Dylan community would like to address
some weaknesses in the language specification and what can be readily
expressed in Dylan code. In this post, we'll look at function types
as well as provide a brief introduction to some details of the type
system implementation within the Open Dylan compiler.

Function Types

One of the big holes in the Dylan type system is the inability to specify
function types. What this means is that you can only say that a value is
of type <function> and can't indicate anything about the desired
signature, types of arguments, return values, etc. This is unfortunate
for a number of reasons:

Poor static type safety. The compiler can verify very little
involving a function value. It can't warn when the wrong number
of arguments or the wrong types of arguments are passed.

Less clear interfaces. The type signature of a function must
be documented clearly rather than being expressed clearly within
the code.

Optimization is more difficult. Since the compiler can't
perform as many checks at compile time, more checks need to be
performed at run-time, which limits the amount of optimization
that can be performed ...

As we discussed in the previous post, we are thinking about a new design
and implementation for the streams library in Open Dylan.

While the examples in this post are in Dylan and are using code from our
HTTP server, these issues exist in HTTP frameworks in other languages.
The code should be clear enough that little to no Dylan knowledge is required
to understand the points being made here.

What does this have to do with HTTP? There are several pain points in our
HTTP stack as it is currently written:

Requests are read in their entirety into memory, so a large request (such
as a file upload) takes a significant amount of memory.

Responses often buffer their entire output in memory as well.

Because of the use of the existing streams library, we don't handle
non-blocking sockets and require a thread per socket.

We don't have a good model for handling long-lasting connections such as
might be used with Server Sent Events or WebSockets without tying
up a thread for the duration of the socket being open.

We don't know yet what the new streams API will look like, but we can
look at our ...

Dylan's current streams library has served us moderately well
over the years. However, it has some issues which can be addressed
by a new design, expanding the range of problems for which it is
suited.

How things are now

A generic, easy-to-use interface for streaming over sequences and files.
The same high-level interface for consuming or producing is available
irrespective of the type of stream, or the types of the elements being
streamed over.

A comprehensive range of I/O facilities for using memory-mapped files,
network connections, and so on.

Unfortunately, the primary interface to our current network library
is based on these very streams for which network connections were not
a design goal. While this works in practice, it imposes some important
limitations on our networking code. The biggest of these is that sockets
can not be non-blocking as it is expected that reads and writes will
complete ...

I spend a lot of time debugging Dylan code. Up until now, this has been a
somewhat painful process when not using the IDE on Windows. (And I don't
really use the IDE on Windows as it doesn't fit well into my workflow.)
I finally reached the point where I decided that I wanted to improve our
debugger integration.

Much of what is described below may be applicable to people working on
debugging support for other languages.

Current State of Dylan Debugging

We have a debugger on Windows integrated with our IDE. This facility is
not available on the other platforms that we support. There are many reasons
for this:

The debug info that Open Dylan generates is only done on Windows.

The debugger code for interacting with the OS is only implemented for Windows.

The IDE itself is only available on Windows.

This means that debugging on Linux, FreeBSD and Mac OS X has traditionally
been more challenging. We often resort to "printf debugging" and have
some basic debug printing functions that can be invoked from the compiler
so long as they don't crash. Debugging is really only effective with the
C back-end as the HARP ...