The only reason [3d user interfaces] haven't caught on, is because the hardware interface to the computer is inherently 2D. A mouse moves only in 2D-- sure, you can fudge it, but there needs to be a proper 3D interface in order for the 3D UIs to work right.

I think this kind of comment is putting the cart before the horse.
The problem with 3d user interface is that they don't really use the
third dimension in a way that's useful or convenient.

The trick is to come up with a useful metaphor for the third dimension. Once you have that, it'll be obvious how to map that into two-dimensional mouse movements that don't frustrate the user.

The model most often used is a geographical model for a hierarchy... usually a file hierarchy. "See, here's the directory you're looking at. And then behind it, there's the sub directories, and when you pop up a level, everything fades back..." which is cool as hell, but when I play with it I find myself wishing for a nice isometric or plan view, which takes us back to two dimensions again.

What I think might be useful is mapping attention into three dimensions. The windows you're actually working on are in the foreground, and other windows are further back, like in Mac OS X. But instead of just having this be implemented using a 2d isometric view that's just a bit of flash on the normal layered windows model... let's use the GPU. Windows further in the background are a little smaller. They really move back in the 3d viewpoint, gradually fogging out (which may look nicer on your nVidia cards), and finally turning into a little dock-style minimized icon. Clicking on them brings them to the foreground and starts the current foreground window cycling back...

To see behind them, you hold down the navigation key (pick one), and moving the mouse moves your viewpoint. Like full-time Expose. The scroll wheel lets you drive forward over or through your windows... as you approach iconized windows they turn back into the real thing (small, in the distance, then growing to full size).

I don't think you'd want "free flight" mode. You're always looking "forward", like in a fixed-attitude video game. Scroll all the way back and you're looking at the active application. Click on something, and by default you "warp" back to home.

Now you have an environment that real 3d widgets (like a 3d "dashboard" or "konfabulator") can live in, like active icons sitting on the ground in fixed locations that windows drift past or over as they fade out of your attention...

You could have modes where windows stay partly in view all the time, moving to the side as they shrink so you can always see them around the edge of the foreground window, iconifying against the edge of the screen if they get "pushed into" it.

Once you have the software, you can experiment with things like "free flight" modes, or modes where the world is a 4d donut so the windows stay in place and you move forward or back... eventually going so far back that you start seeing the previously active windows in the distance...

I've received several responses to this that start something like
"I don't want a 3d environment, I just want a better 2d one"... and
proceed to describe an environment that's pretty much identical to the
one described above, without "walk around" or "free flight" modes. The
thing that I need to make clear is that this is that kind of
environment... there is no "walk around" or "free flight" mode, the
third dimension is not a "spatial" dimension, it's exactly the kind of
layering KillerBob is asking for.