Today I've had a little time to try your viewer. It works very well and is fast indeed. Even the gamma calculation, which as I understand from the commit log implies some performance penalty, doesn't seem to cause a noticeble slowdown. Are you working on zooming and scrolling already? Say something like shift+FiveWay_UP to scale up the document with 30% and alt+FW_UP to scale up with 5% ...?
Perhaps I can help with this, but I will need some time to familiarize myself with Lua and the codebase as a whole and I don't know if I will have enough free time this week.

About the GUI I still haven't made any experiments with Qt but I think that there should be a way to draw on the framebuffer without performance loss for example using QDirectPainter or even a more direct(hacky) way if needed. After all we are providing the screen driver plugin. What I want to try is to use some kind of direct rendering for the pages, and return the control to Qt to visualize the dialogs. This way even if the painting of the dialogs is somewhat sluggish it won't affect the user experience most of the time.