What is Cython?

Cython is the language that lxml is written in. It is a very
Python-like language that was specifically designed for writing Python
extension modules.

The reason why Cython (or actually its predecessor Pyrex at the time)
was chosen as an implementation language for lxml, is that it makes it
very easy to interface with both the Python world and external C code.
Cython generates all the necessary glue code for the Python API,
including Python types, calling conventions and reference counting.
On the other side of the table, calling into C code is not more than
declaring the signature of the function and maybe some variables as
being C types, pointers or structs, and then calling it. The rest of
the code is just plain Python code.

The Cython language is so close to Python that the Cython compiler can
actually compile many, many Python programs to C without major
modifications. But the real speed gains of a C compilation come from
type annotations that were added to the language and that allow Cython
to generate very efficient C code.

Even if you are not familiar with Cython, you should keep in mind that
a slow implementation of a feature is better than none. So, if you
want to contribute and have an idea what code you want to write, feel
free to start with a pure Python implementation. Chances are, if you
get the change officially accepted and integrated, others will take
the time to optimise it so that it runs fast in Cython.

Where to start?

First of all, read how to build lxml from sources to learn how to
retrieve the source code from the GitHub repository and how to
build it. The source code lives in the subdirectory src of the
checkout.

The main extension modules in lxml are lxml.etree and
lxml.objectify. All main modules have the file extension
.pyx, which shows the descendence from Pyrex. As usual in Python,
the main files start with a short description and a couple of imports.
Cython distinguishes between the run-time import statement (as
known from Python) and the compile-time cimport statement, which
imports C declarations, either from external libraries or from other
Cython modules.

Concepts

lxml's tree API is based on proxy objects. That means, every Element
object (or rather _Element object) is a proxy for a libxml2 node
structure. The class declaration is (mainly):

cdef class _Element:
cdef _Document _doc
cdef xmlNode* _c_node

It is a naming convention that C variables and C level class members
that are passed into libxml2 start with a prefixed c_ (commonly
libxml2 struct pointers), and that C level class members are prefixed
with an underscore. So you will often see names like c_doc for an
xmlDoc* variable (or c_node for an xmlNode*), or the above
_c_node for a class member that points to an xmlNode struct
(or _c_doc for an xmlDoc*).

It is important to know that every proxy in lxml has a factory
function that properly sets up C level members. Proxy objects must
never be instantiated outside of that factory. For example, to
instantiate an _Element object or its subclasses, you must always call
its factory function:

A good place to see how this factory is used are the Element methods
getparent(), getnext() and getprevious().

The documentation

An important part of lxml is the documentation that lives in the
doc directory. It describes a large part of the API and comprises
a lot of example code in the form of doctests.

The documentation is written in the ReStructured Text format, a
very powerful text markup language that looks almost like plain text.
It is part of the docutils package.

The project web site of lxml is completely generated from these text
documents. Even the side menu is just collected from the table of
contents that the ReST processor writes into each HTML page.
Obviously, we use lxml for this.

The easiest way to generate the HTML pages is by calling:

make html

This will call the script doc/mkhtml.py to run the ReST processor
on the files. After generating an HTML page the script parses it back
in to build the side menu, and injects the complete menu into each
page at the very end.

Running the make command will also generate the API documentation
if you have epydoc installed. The epydoc package will import and
introspect the extension modules and also introspect and parse the
Python modules of lxml. The aggregated information will then be
written out into an HTML documentation site.

lxml.etree

The main module, lxml.etree, is in the file lxml.etree.pyx. It
implements the main functions and types of the ElementTree API, as
well as all the factory functions for proxies. It is the best place
to start if you want to find out how a specific feature is
implemented.

At the very end of the file, it contains a series of include
statements that merge the rest of the implementation into the
generated C code. Yes, you read right: no importing, no source file
namespacing, just plain good old include and a huge C code result of
more than 100,000 lines that we throw right into the C compiler.

The main include files are:

apihelpers.pxi

Private C helper functions. Except for the factory functions,
most of the little functions that are used all over the place are
defined here. This includes things like reading out the text
content of a libxml2 tree node, checking input from the API level,
creating a new Element node or handling attribute values. If you
want to work on the lxml code, you should keep these functions in
the back of your head, as they will definitely make your life
easier.

classlookup.pxi

Element class lookup mechanisms. The main API and engines for
those who want to define custom Element classes and inject them
into lxml.

docloader.pxi

Support for custom document loaders. Base class and registry for
custom document resolvers.

extensions.pxi

Infrastructure for extension functions in XPath/XSLT, including
XPath value conversion and function registration.

Namespace implementation and registry. The registry and engine
for Element classes that use the ElementNamespaceClassLookup
scheme.

parser.pxi

Parsers for XML and HTML. This is the main parser engine. It's
the reason why you can parse a document from various sources in
two lines of Python code. It's definitely not the right place to
start reading lxml's source code.

parsertarget.pxi

An ElementTree compatible parser target implementation based on
the SAX2 interface of libxml2.

proxy.pxi

Very low-level functions for memory allocation/deallocation
and Element proxy handling. Ignoring this for the beginning
will safe your head from exploding.

public-api.pxi

The set of C functions that are exported to other extension
modules at the C level. For example, lxml.objectify makes use
of these. See the C-level API documentation.

readonlytree.pxi

A separate read-only implementation of the Element API. This is
used in places where non-intrusive access to a tree is required,
such as the PythonElementClassLookup or XSLT extension
elements.