Sunday, 14 June 2009

Python standard library in Native Client

By that I mean that it is possible to import modules from the standard library, but a lot of system calls won't be available. Sqlite works for in-memory use.

Here's a screenshot of the browser-based Python REPL:

The changes needed to make this work were quite minor:

Implementing stat() in glibc so that it does an RPC to the Javascript running on the web page. Python uses stat() when searching for modules in sys.path.

Changing fstat() in NaCl so that it doesn't return a fixed inode number. Python uses st_ino to determine whether an extension module is the same as a previously-loaded module, which doesn't work if st_ino is the same for different files.

Some tweaks to nacl-glibc-gcc, a wrapper script for nacl-gcc.

Ensuring that NaCl-glibc's libpthread.so works.

I didn't have to modify Python or Sqlite at all. I didn't even have to work out what arguments to pass to Sqlite's configure script - the Debian package for Sqlite builds without modification. It's just a matter of setting PATH to override gcc to run nacl-glibc-gcc.

Unladen Swallow could work under Native Client but whether it's worthwhile to use would depend on how fine-grained the JIT compilation is.

If it's a one-off compilation when a module is loaded, that should work fine. The compiler would present the generated code to NaCl, which would make a copy and run the validator on it before mapping it as executable. The dynamic loading support in my branch of NaCl already has a validate-and-map-code operation.

JITting smaller fragments of code repeatedly might not work so well because there would be more calls to the validate-and-map-code operation, which has some overhead. To avoid wasting memory you'd probably want this operation to work at finer granularity than a page (which is 64k on Windows rather than 4k). Deallocating code could be tricky - when reallocating a page in the multithreaded case NaCl would have to make sure that no thread is currently executing instructions on that page.

Where would I find the code to compile Python under Native Client? I would like to use the code for this project https://chrome.google.com/webstore/detail/gdiimmpmdoofmahingpgabiikimjgcia hosted on Google code.

@Ben: There isn't currently a port of Python to run under NaCl in Chromium, as far as I know. This blog post is quite old, and it predates Chrome's PPAPI interface, NaCl's x86-64 sandbox, the newer nacl-glibc port, etc. - and I haven't got around to refreshing the Python port to work with all of that. I'd suggest asking on native-client-discuss.