Broadway adventures in Gtk4

One of my long running side projects is a Gtk backend called “Broadway”. Instead of rendering to the screen this backend creates a HTTP server that you can connect to, and then exposes the UI remotely in the browser.

The original version of broadway was essentially streaming image frames, although there were various ways to optimize what got sent. This matches pretty well with how Gtk 3 rendering works, particularly on Wayland. Every frame it calls out to all widgets, letting them draw on top of a buffer and then sends the final frame to the compositor. Broadway just inserts some image delta computation and JavaScript magic in the middle of this.

Enter Gtk 4, breaking everything!

However, time moves on, and the current development branch of Gtk (which will be Gtk 4) has completely changed how rendering works, with the goal of doing efficient rendering on modern GPUs.

In the new model widgets don’t directly render to a buffer. Instead they build up a model of how the final result should look in terms of something called render nodes. These describe rendering as a tree of highlevel operations. The backend (we have software, OpenGL and Vulkan backends) then knows how to take this description and submit it to the GPU in an efficient way. This is somewhat similar to the firefox WebRender project.

Its would be possible to implement the broadway backend by hooking up the software renderer, letting it generate a buffer and then send that to the browser. However, that is pretty lame!

CSS comes to the rescue!

Instead I’ve been looking at making the browser actually draw the render nodes. Gtk defines a lot of its UI in terms of CSS these days, and that means that the render nodes actually are very close to the CSS rendering model. For example, the basic drawing operation are things like rounded boxes with borders, shadows, etc.

So, I was thinking, could we not take these render node and turn them into actual DOM nodes with CSS styles and send them to the browser. Then every frame we can just diff the DOM trees, sending the minimal changes necessary.

Sounds crazy right? But, it turns out to work pretty well.

Check out this example page which I created with the magic of “save as”. In particular, try zooming into that page in the browser, and play with the developer tools inspector to see the nodes. Here is a part of it zoomed in:

The icons and the text are not CSS, so they don’t scale, but look at those gorgeous borders, shadows and gradients!

Entering the 3rd dimension!

Particularly interesting is the support in Gtk for general 3D transforms. This maps well to the CSS transform on the browser style.

Some technical notes

If you look at the examples above they all use data: uris for images. This is a custom mode that lets “screenshots” like the above work. Normally broadway uses blobs for the images.

Also, looking at the examples they seem very heavy in terms of images, as all the text are images. However, in a typical frame most of the render tree is identical to the previous frame, meaning any label that was used in the last frame need not be sent again. In fact, even if it changes position in the tree due to a parent node changing (scrolling, cube-switching, etc) it can still be reused as-is.

However, text is clearly the weak point in here. Unfortunately HTML/CSS has no low-level text rendering APIs we could use. I’m considering generating a texture atlas with pre-rendered glyphs that can be reused (like CSS sprites) when rendering text, that would mean we will have to download less data at least. If anyone has other ideas I would love to hear about it.

Also With “Cascading Style Sheets” (CSS) You can Specify Multiple Metric-compatible fonts and they will Cascade

on a User Script I am working on I use
“color: #0000EE; color: -moz-hyperlinktext; color: -webkit-link;”
Will First set the color to 0000EE
Then Override it with -moz-hyperlinktext if you are on Firefox
and then override it again with -webkit-link if you are on Chrome
So only the Last Valid Style is used.

TLDR
My Posted Examples above and below aren’t tested and may not actuarially Work.
Google Fonts has Cantarell but i don’t think/know if it has any Metric-compatible fallbacks
Consider Pros And Cons of Browser input types and Submits
Consider Pros And Cons of strip down simple DOM with Custom Attributes and A Separate CSS.css file
You can Create or substitute Widgets with CSS defined globally (It doesn’t need to be specified in every element)
SCSS to Web Browser CSS with cascading fallbacks
Consider a11y
Broadway, Electron, emscripten, and webassembly Synergies
(SDL Qt and unity3d) still using canvas
Client/Front-end/Browser (Interactive JS in Browser) + BroadwayServer/WebAssembly/Backend
————————————–
PS

I don’t know if anything is Metric-compatible with Cantarell but if the Font and size are flexible It shouldn’t matter too badly if it falls back to something else (that is if the page dynamically changes). End users are already allowed to modify their CSS themes.

img[GtkWidgetType=”GtkCheckButton”] {src: “GtkCheckButton.png”; src: “GtkCheckButton.svg”; width: 14px; height: 14px;}
(I Haven’t checked that this works but it is just an idea)
(You could use class instead of (or in addition to) attribute but for userscripts I avoid conflicts)
(Actually you could do exactly what you are doing now and add class and custom attributes for a11y and CSS UserScript UserStyle Mods to use latter Remember the Alt text for images)

So if I understand this correctly GTK uses
“HTML-like DOM structure, with nodes and classes.”
and
“SCSS”
“Adwaita is a complex theme, so to keep it maintainable it’s written and
processed in SASS. The generated CSS is then transformed into a gresource file
during gtk build and used at runtime in a non-legible or editable form.”

If you Can Get your hands on the “SASS” file You can change the SASS to CSS generator to add Browser Specif Fallbacks without mucking around in the clean SASS file.
then GTK can take the “HTML-like DOM structure” and make it into Renderable HTML5/CSS
The DOM can use either standard or Custom Attributes that the CSS.css styles
Put the two Together and the user can override the CSS file if they want.
So you Make “DOM nodes with CSS styles”

“So, I was thinking, could we not take these render node and turn them into actual DOM nodes with CSS styles and send them to the browser. Then every frame we can just diff the DOM trees, sending the minimal changes necessary.”

Then the Question Arises If you want to use Real Browser input types and Submits or Define Your own Attributes
Ether way you can strip the CSS and formatting out of the “DOM trees” and have the CSS in it’s own File.
CSS in the DOM tree can override the CSS file.

Maybe you can have the inputs and drop down menus and dynamic elements on the Client Browser and submit the inputs back.
(Rather than have a Static DOM tree that is constantly updating)
Is it possible to have a normal DOM Interactive web page (Client/Front-end/Browser) that just communicates with the BroadwayServer/WebAssembly/Backend
Some interactive elements need their own JS but others don’t need js
Broadway can inject the CSS and JS into the DOM
but alternatively it can keep the DOM clean and have the CSS in in css.css and the JS in a js.js
the DOM would need Class and/or Attributes
Adwaita can have it’s own CSS.css and JS.js
the Broadway Front end can handle standard GTK interacting with the user
And the Broadway/WebAssembly Server/Backend can handle the backend stuff
Does the WebBrowser really need to wait for updates (New DOM Frames) from the backed/server for some of the user interactions? Can some of this be done Browser side JS? GTK3 Broadway Canvas/bitmaps has no choice but to get new frames, but you aren’t using canvas.

How does Pure DOM + CSS (No JS) & Constant DOM Frame updates compare with
DOM + CSS + JS & Rare DOM Frame updates
If the JS has to loop through every element by Class/Attributive that could slow down every time there is a new Frame but it could eliminate the need for new dom frames. (New DOM frames can even get their own Window/Tab/Frame) Data could be piped in and out and synergize with WebAsembly.

So it is possible to convert non Javascript into JS/WASM or have a JS/WASM interpreter not js scripts

Broadway, Electron, emscripten, and webassembly seem to all need to render GTK themes in the Web Browser
webassembly (SDL Qt and unity3d) as far as i can tell right now just dumps bitmaps into canvas but maybe it doesn’t have to be that way.
Electron I’m assuming uses REAL HTML and Javascript and CSS (but might use webassembly + canvas for parts that are games)
Maybe Most of it (Or at least the GTK Widgets) can be real DOM + CSS and then only the parts that can’t be HTML DOM CSS JS WASM can be bitmaps in canvas.
(So i’m assuming SDL Qt and unity3d all put everything as bitmap in canvas but these are “Games” that don’t translate into DOM+CSS)

If you are using real DOM + CSS does the DOM even need to be updated (like canvas bitmaps) that often and can it change with user inputs directly in the browser. Can More of the Dynamic Rendering and CSS be done Client side in the Browser rather than Server side.
(I think CSD is the opposite of true GTK + Broadway with Client side theming despite using the term “client side”)

Is emscripten/webassembly + GTK a thing yet? and can it be accomplished in DOM instead of canvas?
Can Broadway bring back the old server/client model that Xserver used to have and Wayland Got rid of?
Can the GTK program run Completely in the Browser as an WASM Offline progressive web app with Add to home screen?

You are the Expert This is only my 2 cents
I’m sorry if i wasted too much of you time. I Hope Broadway doesn’t get killed off before we figure out how to use it (Mainstream Adoption).
Vector DOM vs Bitmap Canvas sounds really interesting.
Thank you for your time
E-Bod

Warning!https://www.w3schools.com/about/about_copyright.asp
w3schools is “Fair Use” “Favorable Use” “non-profit teaching” Only
Kind of Defeats the whole point of posting code snippets in the first place and then saying “Fair Use” only.
You can redact those links if that is a problem.
(Or collapse remove shorten my comments if they are too long)

Hi! Do you have any information about webassembly support in the near future? QT has announced this week the webassembly support, and provided a few impressive online demo. I would like to stick with GTK, I feel it more intuitive for me, but from a technology perspective, webassembly support is something to concider.

I don’t personally have any plans for WebAssembly, and I don’t actually think it is very interesting either for Qt or Gtk. If your goal is to run something entirely in the browser, then it should be a native web design, anything else will just be too heavy and large.