GUI Toolkits Suck - But they don't have to

2024-01-15 1559 words 8 minutes

Contents

For years, graphical widget sets, or UI toolkits, or whatever you want to call them, have sucked.

They suck for a wide variety of reasons, but broadly speaking there are a few classes of problems that UI toolkits fail at some or all of:

Platform specificity: They don’t work on all platforms.
Language specificity: They’re written in a language that is either difficult to make bindings for (like C++, because name mangling), or is interpreted, making binding to it undesirable if not practically impossible.
Look-and-feel mismatch: Their visual design conflicts with the design motif of the host operating system.
Theming problems: It’s either hard to make custom themes and styling for the widget set, or it’s practically impossible.
Incompatibility with assistive tech: Most UI toolkits unfortunately don’t play well with screen readers, high contrast settings, and other assistive tech.
Difficulty in making custom widgets: A lot of the time it’s hard to access draw routines, hook into event mechanisms, and so on, making it difficult to make custom widgets.

Because of these and other problems, the computer industry has progressed towards the worst of all possible outcomes: people have largely moved away from making desktop programs. And when they do exist, they are annoyingly frequently just a chromeless browser running a pre-packaged web site. Programmers and designers alike shy away from systems like QT, GTK+, and so on. To say nothing of the platform-specific toolkits.

There are other factors involved too. The biggest two are that making web pages has become desirable from platform-control, business model and marketing reasons (i.e., stripping users of any control over the software they’re using), and that users have turned out to be reluctant to install programs, but very happy to visit websites.

I know the proponents of whatever $WIDGET_SET are going to read this and go “but that’s what we’re doing!!!” But no. Sorry. I know you’re trying, but if you were succeeding we probably wouldn’t see people unironically releasing programs made in Electron ffs. More accurately: the thing you’re working on is probably excellent on some level, but it’s probably not addressing exactly the issues I’m trying to get at. Anyway.

The Stupid Antidote to Suckage

And so, UI toolkits for websites have blossomed. To the point that there are hundreds of them. Many are good, many more are bad, but they all come with their own customizability limitations, framework association, and so on and so forth.

But ultimately, it’s people using web technologies to make application UIs. There are downsides to this:

Limited, if any, consistency.
Users need to adapt (knowingly or unknowingly) to the idiosyncracies of each component set.
Massive efficiency hit associated with using a powerful general purpose layout engine where a fast specialized layout set with possibly a constraint solver for declarative layouts would suffice.+
An (often annoying) disconnect between the user’s operating system and the execution environment of the software.
Very variable support for assistive technologies.

… and more, probably.

On the upside:

A huge number of visual, UI and UX designers are familiar with web technologies.
The customizability is enormous.
Because the apps are websites, it becomes trivial to roll out new versions of the software to all users or a chosen subset of them.

But what’s also been happening is that, because the native(ish) widget sets are so bad, it’s quite common for software projects to roll their own. Text editors (at least those that aren’t just chromeless websites) are particularly guilty here, with editors like Sublime Text and Focus having their own UI framework.

Blender, similarly, has its own widget set - one I actually really really like. If anything, I wish that widget set had just been broken out into its own thing.

Challenge to any Blender or Focus developers who read this: Set your widgets free!

So yeah. Anyway. It’s a bit of a mess.

It gets worse

Once upon a time, you could at least roughly expect the form factor and resolution of the underlying system to be roughly within a small range of options. No more. Now we have phones (vertical and horizontal), tablets, laptops and desktops; we have examples users still unironically using 1024x768 pixel resolutions, and others blasting a quad of 4K monitors. Or whatever.

So things like DPI matter a ton more than they used to, as does media-aware layout flexibility.

And you also have mice, touch, multi-touch, stencils, and any number of other input modes…

Some nostalgia

Recently I’ve found myself thinking back to Delphi’s Visual Component Library (VCL), and how simple, elegant and flexible it was, while also being relatively full featured and not difficult to extend.

I miss stuff like that.

I never felt that way about GTK+ or QT or WxWidgets, or anything else really. I kind of liked the BeOS widget set and the QNX Photon GUI wasn’t bad, but both of those were quite niche. The Android default widget set wasn’t terrible either, but it was probably always a bit too much Java for my taste.

I really like Dear ImGUI and some other immediate mode GUIs, but I think in practice it’s going to be a lot less difficult for most people to work with a retained mode GUI, where widgets are largely pre-defined in a scene graph and rendered as such. The tradeoffs are significant, but one benefit to not overlook is that with retained mode, you can calculate widget layout constraints only when needed (such as on window resize) rather than having to do so on every frame, which means that more complex layout mechanics are feasible, even if the more crazy forms of constraint based layout should be avoided.

Working with web-based component libraries is always a bit of a pain. On the one hand, you have near limitless flexibility. But it’s hard to get things right. The visual language has such a loose grammar that it’s inevitable that some component will be used inappropriately, and even if everything is well formed, you’re still almost certainly going to run afoul some nuances in tab order, screen reader or braille interface, colorblindness and contrast issues, or what have you. Keeping all those ducks in a row is just so much effort.

I have projects I work on now using Blueprint and Ant Design. The latter is clearly the superior widget set from both a technical and visual design perspective, but it’s got quite a few gotchas.

A wishlist for a new component set

What I find myself wanting is a widget set that:

Runs as native code; no browser.
Is easy to use from many programming languages.
Works on Linux, Windows, MacOS, Android and iOS.
Can get look-and-feel hints from the underlying operating system, so it never seems out of place.
Is adaptable to the different form factors and resolution ranges.
Is highly performant.
Uses GPU acceleration natively, because drawing pixels on a screen in 2024 without using the GPU is stupid.
Has good standard support for high contrast and other similar display modes.
Supports both manual layout and lightweight widget packing mechanisms; stays away from heavier constraints models as much as is practical.
Supports animations, sounds and such, without being over-the-top by default.
Supports RTL, i18n and l10n out of the box.
Should be easy to overlay over other graphical viewports for use in games and such.
Easily extensible.
Has a toolchain for enabling designers to easily define UIs without the output of that slowing down the resulting native code.

In other words, I want a modern, GPU accelerated version of the Delphi VCL.

Is that too much to ask?

Thoughts on how to do this

First off, a next generation graphical widget set should be built on a next generation programming language. It can be Zig or Rust or Jai or whatever, but it needs to be simple.

As mentioned above, I really like immediate mode GUIs, but I think the default should be retained mode, where you construct a scene graph where components are either “Containers” or “Widgets”. A Container may have some amount of chrome, but it is expected to have some number of children. A Widget cannot have children, or, if it does, it has to manage them itself.

The main reason for doing it this way is that it is easier to think about – the emphemerality of immediate mode GUIs is a bit wild for people who’ve never seen them before. But the additional benefits are opportunities for more advanced layout and pre-compute features without it being a hit to performance.

I don’t actually have clear ideas on the implementation beyond this.

To make this, you’d need to have a rich set of basic widgets and they need to be pretty flexible. Here’s a quick list I threw together:

Widgets

Label
Button
ToggleButton
RangeSlider
Scrollbar
Checkbox
Dropdown
Radio Button
Image
Video
Text Input
Toggle Switch
Progress Bar
Date Picker
Time Picker
Slider
Color Picker
Calendar
File Picker
Table
List
Buttonbar
Menubar
Menu
Menu Item
Tooltip
Badge
PixelCanvas

Containers

Window
Panel
Row
Column
Grid
Tabs
Pills
Dock
SlidePanel
Modal
Accordion

Utilities

Translation Provider
Animation Provider
Localization Provider
Theme Provider

So, what now?

I dunno. I’m just thinking out loud. I doubt I’ll do anything about this. I simply don’t have the time. And if I do, it’ll probably be half-arsed. Which is a fantastic reason not to do it.