When I first learned about the PyPy project, it took me a while to figure out
exactly what it was about. For those that don't already know, it's two things:

A set of tools for implementing interpreters for interpreted languages

An implementation of Python using this toolchain

The second part is probably what most people think PyPy is, but this tutorial
is not about their Python interpreter. It is about writing your own
interpreter for your own language.

This is the project I undertook to help myself better understand how PyPy works
and what it's all about.

This tutorial assumes you know very little about PyPy, how it works, and even
what it's all about. I'm starting from the very beginning here.

What PyPy Does

Here's a brief overview of what PyPy can do. Let's say you want to write an
interpreted language. This involves writing some kind of source code parser, a
bytecode interpretation loop, and lots of standard library code.

That's quite a bit of work for moderately complicated languages, and there's a
lot of low level work involved. Writing the parser and compiler code usually
isn't fun, that's why there are tools out there to generate parsers and
compilers for you.

Even then, you still must worry about memory management in your interpreter,
and you're going to be re-implementing a lot if you want data types like
arbitrary precision integers, nice general hash tables, and such. It's enough
to put someone off from implementing their idea for a language.

Wouldn't it be nice if you could write your language in an existing high level
language like, for example, Python? That sure would be ideal, you'd get all the
advantages of a high level language like automatic memory management and rich
data types at your disposal. Oh, but an interpreted language interpreting
another language would be slow, right? That's twice as much interpreting going
on.

As you may have guessed, PyPy solves this problem. PyPy is a sophisticated
toolchain for analyzing and translating your interpreter code to C code (or JVM
or CLI). This process is called "translation", and it knows how to translate
quite a lot of Python's syntax and standard libraries, but not everything. All
you have to do is write your interpreter in RPython, a subset of the Python
language carefully defined to allow this kind of analysis and translation, and
PyPy will produce for you a very efficient interpreter.

Because efficient interpreters should not be hard to write.

The Language

The language I've chosen to implement is dead simple. The language runtime
consists of a tape of integers, all initialized to zero, and a single pointer
to one of the tape's cells. The language has 8 commands, described here:

>

Moves the tape pointer one cell to the right

<

Moves the tape pointer one cell to the left

+

Increments the value of the cell underneath the pointer

-

Decrements the value of the cell underneath the pointer

[

If the cell under the current pointer is 0, skip to the instruction after
the matching ]

]

Skip back to the matching [ (evaluating its condition)

.

Print out a single byte to stdout from the cell under the pointer

,

Read in a single byte from stdin to the cell under the pointer

Any unrecognized bytes are ignored.

Some of you may recognize this language. I will be referring to it as BF.

One thing to notice is that the language is its own bytecode; there is no
translation from source code to bytecode. This means that the language can be
interpreted directly: the main eval loop of our interpreter will operate right
on the source code. This simplifies the implementation quite a bit.

First Steps

Let's start out by writing a BF interpreter in plain old Python. The first step
is sketching out an eval loop:

As you can see, a program counter (pc) holds the current instruction index. The
first statement in the loop gets the instruction to execute, and then a
compound if statement decides how to execute that instruction.

The implementation of [ and ] are left out here, but they should change the
program counter to the value of the matching bracket. (The pc then gets
incremented, so the condition is evaluated once when entering a loop, and once
at the end of each iteration)

Here's the implementation of the Tape class, which holds the tape's values as
well as the tape pointer:

As you can see, the tape expands as needed to the right, indefinitely. We
should really add some error checking to make sure the pointer doesn't go
negative, but I'm not worrying about that now.

Except for the omission of the "[" and "]" implementation, this code will work
fine. However, if the program has a lot of comments, it will have to skip over
them one byte at a time at runtime. So let's parse those out once and for all.

At the same time, we'll build a dictionary mapping between brackets, so that
finding a matching bracket is just a single dictionary lookup. Here's how:

If you're following along at home, you'll also need to change the signature of
mainloop() and implement the bracket branches of the if statement. Here's the
complete example: example1.py

At this point you can try it out to see that it works by running the
interpreter under python, but be warned, it will be very slow on the more
complex examples:

$ python example1.py 99bottles.b

You can find mandel.b and several other example programs (not written by me) in
my repository.

PyPy Translation

But this is not about writing a BF interpreter, this is about PyPy. So what
does it take to get PyPy to translate this into a super-fast executable?

As a side note, there are some simple examples in the pypy/translator/goal
directory of the PyPy source tree that are helpful here. My starting point for
learning this was the example "targetnopstandalone.py", a simple hello world
for PyPy.

For our example, the module must define a name called "target" which returns the
entry point. The translation process imports your module and looks for that
name, calls it, and the function object returned is where it starts the
translation.

In the example above, you'll see a few things have changed. I'm now using low
level file descriptors with os.open and os.read instead of file objects.
The implementation of "." and "," are similarly tweaked (not shown above).
Those are the only changes to make to this code, the rest is simple enough for
PyPy to digest.

That wasn't so hard, was it? I still get to use dictionaries, expandable lists,
and even classes and objects! And if low level file descriptors are too low for
you, there are some helpful abstractions in the rlib.streamio module included
with PyPy's "RPython standard library."

Translating

If you haven't already, check yourself out the latest version of PyPy from
their bitbucket.org repository:

$ hg clone https://bitbucket.org/pypy/pypy

(A recent revision is necessary because of a bugfix that makes my example
possible)

The script to run is in "pypy/translator/goal/translate.py". Run this script,
passing in our example module as an argument.

[A note added much later: this script has been moved to "rpython/bin/rpython".]

$ python ./pypy/pypy/translator/goal/translate.py example2.py

(You can use PyPy's python interpreter for extra speed, but it's not necessary)

PyPy will churn for a bit, drawing some nice looking fractals to your console
while it works. It takes around 20 seconds on my machine.

The result from this is an executable binary that interprets BF programs.
Included in my repository are some example BF programs, including a mandelbrot
fractal generator, which takes about 45 seconds to run on my computer. Try it
out:

$ ./example2-c mandel.b

Compare this to running the interpreter un-translated on top of python:

$ python example2.py mandel.b

Takes forever, doesn't it?

So there you have it. We've successfully written our own interpreter in RPython
and translated it with the PyPy toolchain.

(more in the next blog post...)

This is a guest blog post written by Andrew Brown, with help from the PyPy developers
on the pypy-dev mailing list.

When I first learned about the PyPy project, it took me a while to figure out
exactly what it was about. For those that don't already know, it's two things:

A set of tools for implementing interpreters for interpreted languages

An implementation of Python using this toolchain

The second part is probably what most people think PyPy is, but this tutorial
is not about their Python interpreter. It is about writing your own
interpreter for your own language.

This is the project I undertook to help myself better understand how PyPy works
and what it's all about.

This tutorial assumes you know very little about PyPy, how it works, and even
what it's all about. I'm starting from the very beginning here.

What PyPy Does

Here's a brief overview of what PyPy can do. Let's say you want to write an
interpreted language. This involves writing some kind of source code parser, a
bytecode interpretation loop, and lots of standard library code.

That's quite a bit of work for moderately complicated languages, and there's a
lot of low level work involved. Writing the parser and compiler code usually
isn't fun, that's why there are tools out there to generate parsers and
compilers for you.

Even then, you still must worry about memory management in your interpreter,
and you're going to be re-implementing a lot if you want data types like
arbitrary precision integers, nice general hash tables, and such. It's enough
to put someone off from implementing their idea for a language.

Wouldn't it be nice if you could write your language in an existing high level
language like, for example, Python? That sure would be ideal, you'd get all the
advantages of a high level language like automatic memory management and rich
data types at your disposal. Oh, but an interpreted language interpreting
another language would be slow, right? That's twice as much interpreting going
on.

As you may have guessed, PyPy solves this problem. PyPy is a sophisticated
toolchain for analyzing and translating your interpreter code to C code (or JVM
or CLI). This process is called "translation", and it knows how to translate
quite a lot of Python's syntax and standard libraries, but not everything. All
you have to do is write your interpreter in RPython, a subset of the Python
language carefully defined to allow this kind of analysis and translation, and
PyPy will produce for you a very efficient interpreter.

Because efficient interpreters should not be hard to write.

The Language

The language I've chosen to implement is dead simple. The language runtime
consists of a tape of integers, all initialized to zero, and a single pointer
to one of the tape's cells. The language has 8 commands, described here:

>

Moves the tape pointer one cell to the right

<

Moves the tape pointer one cell to the left

+

Increments the value of the cell underneath the pointer

-

Decrements the value of the cell underneath the pointer

[

If the cell under the current pointer is 0, skip to the instruction after
the matching ]

]

Skip back to the matching [ (evaluating its condition)

.

Print out a single byte to stdout from the cell under the pointer

,

Read in a single byte from stdin to the cell under the pointer

Any unrecognized bytes are ignored.

Some of you may recognize this language. I will be referring to it as BF.

One thing to notice is that the language is its own bytecode; there is no
translation from source code to bytecode. This means that the language can be
interpreted directly: the main eval loop of our interpreter will operate right
on the source code. This simplifies the implementation quite a bit.

First Steps

Let's start out by writing a BF interpreter in plain old Python. The first step
is sketching out an eval loop:

As you can see, a program counter (pc) holds the current instruction index. The
first statement in the loop gets the instruction to execute, and then a
compound if statement decides how to execute that instruction.

The implementation of [ and ] are left out here, but they should change the
program counter to the value of the matching bracket. (The pc then gets
incremented, so the condition is evaluated once when entering a loop, and once
at the end of each iteration)

Here's the implementation of the Tape class, which holds the tape's values as
well as the tape pointer:

As you can see, the tape expands as needed to the right, indefinitely. We
should really add some error checking to make sure the pointer doesn't go
negative, but I'm not worrying about that now.

Except for the omission of the "[" and "]" implementation, this code will work
fine. However, if the program has a lot of comments, it will have to skip over
them one byte at a time at runtime. So let's parse those out once and for all.

At the same time, we'll build a dictionary mapping between brackets, so that
finding a matching bracket is just a single dictionary lookup. Here's how:

If you're following along at home, you'll also need to change the signature of
mainloop() and implement the bracket branches of the if statement. Here's the
complete example: example1.py

At this point you can try it out to see that it works by running the
interpreter under python, but be warned, it will be very slow on the more
complex examples:

$ python example1.py 99bottles.b

You can find mandel.b and several other example programs (not written by me) in
my repository.

PyPy Translation

But this is not about writing a BF interpreter, this is about PyPy. So what
does it take to get PyPy to translate this into a super-fast executable?

As a side note, there are some simple examples in the pypy/translator/goal
directory of the PyPy source tree that are helpful here. My starting point for
learning this was the example "targetnopstandalone.py", a simple hello world
for PyPy.

For our example, the module must define a name called "target" which returns the
entry point. The translation process imports your module and looks for that
name, calls it, and the function object returned is where it starts the
translation.

In the example above, you'll see a few things have changed. I'm now using low
level file descriptors with os.open and os.read instead of file objects.
The implementation of "." and "," are similarly tweaked (not shown above).
Those are the only changes to make to this code, the rest is simple enough for
PyPy to digest.

That wasn't so hard, was it? I still get to use dictionaries, expandable lists,
and even classes and objects! And if low level file descriptors are too low for
you, there are some helpful abstractions in the rlib.streamio module included
with PyPy's "RPython standard library."

Translating

If you haven't already, check yourself out the latest version of PyPy from
their bitbucket.org repository:

$ hg clone https://bitbucket.org/pypy/pypy

(A recent revision is necessary because of a bugfix that makes my example
possible)

The script to run is in "pypy/translator/goal/translate.py". Run this script,
passing in our example module as an argument.

[A note added much later: this script has been moved to "rpython/bin/rpython".]

$ python ./pypy/pypy/translator/goal/translate.py example2.py

(You can use PyPy's python interpreter for extra speed, but it's not necessary)

PyPy will churn for a bit, drawing some nice looking fractals to your console
while it works. It takes around 20 seconds on my machine.

The result from this is an executable binary that interprets BF programs.
Included in my repository are some example BF programs, including a mandelbrot
fractal generator, which takes about 45 seconds to run on my computer. Try it
out:

$ ./example2-c mandel.b

Compare this to running the interpreter un-translated on top of python:

$ python example2.py mandel.b

Takes forever, doesn't it?

So there you have it. We've successfully written our own interpreter in RPython
and translated it with the PyPy toolchain.

@temptemptemp13: I think you are missing something much more substantial. This article is not about Python at all. It is about how to use the PyPy toolchain to implement a different language - in this case the brainfuck programming language.

While BF isn't a very useful language, it has the nice properties of being very small. Almost all of the language fits in a blog post.

Amazing! Thanks for posting. I was wondering, what's about a pure C or C++ implementations, as close as reasonable to the python one? So I wrote them. You can read more details here, but the bottom line is that PyPy is (marginally) faster than C++, and (marginally) slower than C :-O

I think that your "," implementation is incorrect, os.read returns an empty string on EOF, thus [0] triggers an exception.According to Wikipedia, setting the cell to 0, -1 or leaving the cell unchanged each may be used to tell EOF apart from other characters.