The Python library includes a hand-written Lua parser with support for Pico-8 features, with API access to the token stream and abstract syntax tree. Additional modules provide access to the graphics, map, sound, and music data. All game data can be transformed or created with Python code and written to a .p8 or .p8.png cart file.

I've included the Lua minifier (luamin) as an example tool, though I don't actually think it's a good idea to use it. Statistically, you'll run out of tokens before you run out of characters. The Pico-8 community benefits more from published carts with code that is easy to read and well commented. All luamin does is make the code difficult to read, without much benefit.

Here is a chart of the character count, minified character count, and token count of 496 carts that have been posted to the BBS, relative to the Pico-8 maximums:

As shown, even un-minified code tends to stay below the token count, percentage-wise. Minifying reduces the (uncompressed) character count to 65% of the original size on average.

The inspirational use case for this library is actually luafmt, which re-formats the Lua code of a cart to be easier to read. The current version of this tool is simple and only adjusts indentation. There is much more luafmt can do by analyzing the AST, but this is not yet implemented.

As of today, this is very much a v0.1 early release. There are known issues regarding parsing and token counting, and the library APIs are incomplete and messy. This project has already gotten away from me a bit, so I'm not sure how far I'll take it. But if you do want to build against it, patience with non-backwards compatible changes will be appreciated. :)

hseiken: By definition, minifying the code as luamin and JavaScript minifiers do does not change the token count. They only reduce the number of characters used to represent long tokens, such as by renaming variables. Pico-8's token count already excludes comments and whitespace, and only counts each name as a single token, regardless of its length. (Try the listtokens tool for an illustration of how tokens are defined and counted.)

There may be a few token optimizations that a tool could do automatically, but they'd be pretty smart compiler-style optimizations that actually rewrite the semantic structure of the code to a shorter equivalent. I'm thinking about building a tool that does dead code elimination, for example. (No promises, I might not have time, but it's the kind of thing you can do with access to the AST.)

I've also started a new build tool. Its features are modest for now: it can replace sections of a cart (Lua, spritesheet, sfx, etc.) with sections from other carts, and it can also read Lua code from a .lua file and add it to a cart. It can also create carts from scratch and erase sections.

I have plans for more compelling features for the build tool (require() support, dead code elimination). My summer sabbatical is coming to an end so I might not get to it right away, but I thought I'd push what I have so far.

The new .p8.png code includes the code compression routine, the dual of the decompression routine that we've had for a while. Thanks to zep for providing the last few missing pieces!

[Update: fixed] Oof, just noticed picotool's compressor is compatible but not efficient, and is not producing the same result as Pico-8's compressor. It's bad enough that I don't think this is quite ready for the build workflow of a real (large) game. I'll update again once I've fixed it, but I'm mentioning it in case I don't get around to it right away. (My vacation is over! :( )

I made a quick fix by temporarily replacing my compression code with a direct port of the Pico-8 version. It'll be worth going back and looking for differences, if only to speed it up. But this'll do for now.

It's likely that the cart just doesn't compress under the limit and there's not much to be done, but I'd like to clean up the error messages at least. If we're lucky, minification will get it just under the wire, but no guarantees. :)

Replied on the bug but will summarize here for reference: I need to add support for glyphs and labels before I can properly parse your cart. These are likely unrelated to the symptom you're seeing but I can't repro otherwise, and they're notable shortcomings regardless. I'll keep the bug open and revisit once I've had a chance to make these changes. Thanks for the report!

I'm having a strange problem; I'm attempting to use writep8 on Mac OS Sierra/Python 3.5 to convert p8.png carts to .p8 carts (to be edited in Sublime Text). However, every time I've attempted to do so, p8tool reports:

../carts/cartname.p8.png -> ../carts/cartname_fmt.p8.png

and sure enough, when I check the resulting output file, it's still a *.p8.png cart, and attempting to read it with cat or Sublime only yields byte data. Any ideas what could be wrong?

I'm looking forward to trying out luamin over the weekend. I must be one of the rarer cases that have hit the character limit before the token limit. I am parsing long strings in order to reduce token count which is where most of it would be going.

I'm going to blame Pico-8 for this one because it appears to be a shortcoming in its short-form "if" statement that luamin doesn't take into account. luamin will collapse space between a non-word non-space character and a word character. This is valid for Lua, but breaks Pico-8's short-if, which is implemented as a preprocessor replacement.

x=true
y=(0)
if (x) y=1
print(y)

minifies to

x=true y=(0)if (x) y=1
print(y)

Pico-8 doesn't recognize the "if" as a short-if and fails to insert the "then" and "end" keywords before parsing the Lua. (picotool's own parser doesn't see the problem because I implemented short-if as a formal part of the grammar before I realized that Pico-8 was using a preprocessor replacement.)

I see a similar issue for a space before a "local" keyword being collapsed and causing problems. I'm less sure of the cause within Pico-8 but the solution is similar: it needs leading spaces preserved.

You can work around the short-if issue by avoiding short-if (using if-then-end) in problem locations. Unfortunately the workaround for "local" would be to make sure the previous line ends with a word character (and not in a comment), which is not a good workaround. As a cheap hack, you could write a little script that replaces "local" with " local" after the minification.

If you want to experiment, I found the first problematic short-if on line 454 of your original cart. Let me know if you find other cases beyond short-if and local. Sorry for the inconvenience.

BREAKING CHANGE: All Lua code is now handled in the API as bytestrings and not text strings. Pico-8 uses a custom text encoding equivalent to lower ASCII plus glyphs as high bytes. Instead of trying to manage conversion between these high bytes and Python text strings via an encoder, I just converted the whole thing to keep Lua code in bytestring form. This is a breaking change for the parts of the API that accept or return strings for/from the Lua code (token data, cart title, etc.). I probably busted stuff in the process so let me know if you see bugs.

With thanks to juanitogan, the minifier knows about more Pico-8 built-ins (and so doesn't munge their names). Also, a few bits were using an API only available in Python 3.5. I replaced those bits to use juanitogan's suggested Python 3.4-compatible equivalent, so Cygwin users can play. (Cygwin is currently still using Python 3.4 as its 3.x package.)

I added support for loading and saving .p8 files that have a label. This does not yet know how to convert labels between .p8 and .p8.png carts. (That'd be cool, especially since the same code could also support importing and exporting PNG images as spritesheet or label data.)

@Sascha217: Thanks for the report. I submitted a few fixes to get your cart to go through luamin, including adding "stop" to the keyword list. There's still a syntax error in the result, and I believe this is related to issue #11 regarding luamin being too aggressive with eliminating space between non-word chars and keywords. I can edit the result and insert spaces to resolve some syntax errors (though this needs to be done throughout). I'll look at this issue more this evening (PT).

One more thread bump to say that I've committed another batch of changes and small improvements. This includes support for a few popular but unofficial or accidental quirks in Pico-8's Lua syntax. if-do is now supported, as are C++-style // line comments. If you pulled since Feb 22, please pull again.

I'm pleased to say that with this last set of improvements, picotool successfully recognizes as valid all valid carts published to the BBS up to cart ID 28280. In this set there are four carts with errors that I've jailed for obscure reasons where Pico-8 accepts them but I don't see a good reason. I'll do a fresh crawl and test again on more recent carts. (This testing methodology only shows that picotool accepts these carts as valid. it does not prove that it produces correct parse trees. :) )

Sascha217: I submitted a fix, please try again. I can minify your cart and it runs successfully. mario005.p8 goes from 41096 uncompressed to 233353 uncompressed, and from 15743 compressed to 10389 compressed.

It's a quick fix and may or may not be a complete solution to the problem. (In theory it might actually add chars in trivial extreme cases, but I'm not worried.)

Thanks. The issue was just in the error message. The actual token limit is 8192, so 27797 definitely exceeds that. Also I can't save your cart as a .p8.png because the compressed code size is too large. You may need to consider a multi-cart solution.

Sorry, it was still reporting chars as tokens as of my previous message. You don't have 27797 tokens. :)

picotool does still appear to have a significant disparity from Pico-8 in token counting. Other than doing rigorous black box testing (with a lot of manual effort) through various parts of Lua syntax, there's not much I can do to nail down the last few differences. I'd like picotool to be accurate so it can report when it's building a cart that won't fit, but maybe that's an impractical feature for a weekend hobby project.

p8tool stats mario010.p8 reports 8560 tokens, 47866 chars, 18710 compressed chars. Pico-8 reports 7719 tokens, 47865 chars, 18720 compressed chars. So picotool erroneously overcounts and warns that it might be over the limit. (Then on top of that I had a broken message, which I'm about to fix.) Shrug emoji.

Nice tool! I want to use it to modularize my project, splitting code into different files and separating code from data. However, the PICO-8 editor directly puts sprite and music data into the game file (p8 or p8.png). I need a method to extract those data chunks and export them into separate files. Then, I can call "p8tool build" with the options to import the data files back into the final game file.

This would allow me to version control each data file alongside the code, and to exclude the built game file from version control. Currently, I would need to track the main game file because it's the only file containing my data, which means I'm tracking the code twice: in the source lua files and in the game file.

Without a data export script, I would need to copy-paste the data chunks to data files manually before each commit.

From what I can see, exporting data is feasible with the current API. list_lua uses g.lua.to_lines() to access the lua chunk, and we can similarly access other chunks.

Ex: for l in g.gfx.to_lines(): print(l)

We could output these lines in the correct format in some game.gfx file. Same for other data chunks.

For now I'll just make a simple data export script for my own game, but since you offer a tool to build a game from both code and data, I think such a feature would be a nice addition to the core tool package.

If "mygame_final.p8.png" does not exist, it'll create it, using empty sections for any section you don't specify. If it does exist, it'll preserve any section you don't specify.

It sounds like you want to track source files in version control, and not track the final built output. In that case, you could either 1) specify all options to p8tool build to create a fresh cart during the build, or 2) keep everything but Lua in a source cart, copy that cart to the build location, then run p8tool build to add the Lua to that cart.

Yes you can write your own tool using the library if you'd like. You can look at the source for p8tool build for ideas on how to use the API. But it sounds like your use case is covered by the existing tool.

Hm, I hadn't considered working on a cartridge made purely of data. That would allow me to work in the P8 editor, save the data cartridge then build the final game, without even the need to extract data. I will try this today, thanks!

I have another problem now. I'm trying to unit test my modules with pico-test (https://github.com/jozanza/pico-test). As you explained in the README, we should be able to create lua scripts containing their own tests. Except it doesn't work if you use the "local module = {} / return module" pattern, because building a cartridge directly from the script will preserve the "return" which causes a parsing error (you're not supposed to do that in a normal game script).

So either you need to use one of the 2 other export methods (global function or global module), or you need to build from a main testmodule.lua script which will require the said module. In this case, you can either keep the test on the module side and just call them, or move the tests to the testmodule script. But this method requires an extra test runner script for each module.