I’m a Gesture Junkie

So today I watched Minority Report that bear lent to me and noticed, apart from that Philip K. Dick liked to write stories about future prediction, the computer interface Tom Cruise used was reminicient of what Keanu Reeves used in Johnny Mnenomic, only without the goggles. That is, not only did they use their hands as if they were a computer mouse, but gestures were added that performed functions you’d normally find in some hotkey or drop-down menu list. I really think that this kind of interface (albeit rather exaggerated in the movies to be efficient), along with voice commands, is the future. Except, they aren’t the future at all. Gestures are already here.

My prefered web browser, Opera, for its speed, low system resources and customability, turned me on to mouse gestures (a system which Mozilla subsequently copied). Mouse gestures are mouse movements, including clicks, wheel rolls and drags, that allow quick access to functions normally without having to use a keyboard (also useful for those who have trouble using a keyboard). According to Wikipedia, Mac actually invented the first mouse gesture, the drag and drop. Since then, it’s been all Opera for pushing the bill and popularizing it. I can go forward, back, reload, stop, change/close/open new browsing tabs, open link in new tab, etc., including things I’m not even aware of yet. This is to the point that, thanks to Opera’s configurability, I don’t even have a browsing toolbar. No arrows, stop signs or anything, because I wouldn’t use any of it; the gestures are simply quicker and more convenient.

Back to the point, while watching the movie I thought how cool it would be to interface my system like I do my web browser. Actually, it’s a common habit of browser gesture users to try to navigate Windows like they do their web browser, and just end up openning a drop-down menu instead of doing something else. But, again, this technology is already here. StrokeIt is a pretty powerful Win app that puts gestures into your OS, but goes from application to application, so it doesn’t interfere with gestures in programs you already have. Some gestures are pretty arbitrary, but it also incorporates Graffiti, that Palm OS way of writing letters with a stylus. It can do anything a hotkey can, which is apparently alot, with just a flick of the wrist. I didn’t even know there was a hotkey that could close MDI windows, or windows within windows. Anyway, I’m using and customizing it already, but I’m just scratching the surface, so I don’t have too much to say at the moment.

You Mac-ies should look into what kind of gesture software is available.

Also, yes, I have tried the voice commands thing, like I mentioned above, with a program called Dragon. While, according to a co-worker who worked with people with physical impairments, it supposedly works great. She says that some un-impaired people use it to dictate papers instead of having to type them. Though, I didn’t find that kind of ease of use to warrant me using it over a keyboard.

About dick

Post navigation

7 thoughts on “I’m a Gesture Junkie”

Funny, I saw Minority Report a long time ago and just remember being vaguely unimpressed with the plot. Totally forgot about the gesture interface.

Actually I think there might have been some sci-fi story like that, where “magic” was actually some sort of high tech phenomena and the arcane gestures were supposed to be computer activation sequences. Maybe I’m just making that up, but the idea sounds familiar.

Really though the whole “gesturing in the air” interface sounds like a neat idea, but it does have it’s drawbacks. First of all there’s nothing to support your arms so you just have to hold them in front of you and flail away. Might be tiring in the long run (but at least data entry people will be beafy armed from the exercise). Also there’s no tactile feedback, so it might be difficult to gague the exact position of your fingers or if you’ve activated a command properly even if the screen provides some indicator.

Me, I like a keyboard. Gives my fingers something to do. I definitely type faster than I can graffiti in a palm-type-device. But I can see the use of gesture software, especially for people who have trouble with keyboards for whatever reason.

The thing that bugs me isn’t the mouse or the keyboard or some other device, but having to move my hand between one and the other. Maybe I need a chord keyboard or a data hand or something.

While I realize computers and the interfaces we use to interact with them have evolved a lot since the 8-bit Nintendo days, I find myself using the keyboard more than ever. I typically navigate my computer with LaunchBar. While it might look like a simply application launcher, in addition to opening programs, I can open files with, navigate my music library, open web pages, do google searches (or any other web search), etc. Its a really powerful program, and using it involves typing in a few letters when you want something.

I also notice when using “professional” programs like Logic, Photoshop, or Final Cut Pro, it really helps to have one hand on the keyboard and one on the mouse. The mouse hand is certainly useful in these programs, especially the visual ones, but when it comes to changing tools, changing colors, changing location in the song/video, playing/stopping, the keyboard can’t be beat. And don’t even get me started about using the keyboard over the mouse in a powerhouse text application like BBEdit.

Obviously (and hopefully), computing won’t be stuck on the mouse and keyboard forever. I’m happy that people are pushing the envelope in terms of interfaces, graphical and otherwise. However, we’s gots a ways to go still before I give up my mouse and keyboard.

And yes, I use the one button Apple mouse. I’ve certainly spent enough on computers, the aforementioned applications, and boatloads of others technical stuff to purchase a mouse with 85 buttons on it. Though I do sometimes think it might be nice to have a scroll wheel, for the most part, I’m perfectly happy with one button and the other 100+ on my keyboard.

IMHO the scroll wheel is pretty over-rated. I have no problem using the mouse to do the same thing.

I know what you mean about the mouse being useful for visual stuff though (Photoshop, etc.). As mice go I’m starting to prefer a trackball. Tastes differ heavily on trackball vs. mouse, but with mice my hand ends up slowly migrating across the desk in the course of a day.

Yeah, the point the mouse gestures isn’t to replace the keyboard. If anything, that’s the job of voice recognition; that’s why I mentioned that. People really are “writing” papers by dictation, so that technology is already here as well. If done right, that’s most likely faster than anyone can type. The point of graffiti with a mouse certainly isn’t for writing, but for an added set of motions to use in making a gesture (other than just drag up, drag down, drag left, drag left, drag up then left, drag up then right, drag up then down, etc.). Here’s an example of using the letter C to close (or reverse C to MDI close). Actually, I think the simpler gestures are better than writing graffiti since they are alot faster. I expect to reconfigure things to like the Winamp defaults because I can just sub in a drag down for play and a drag up for stop (that’s intuitive if you’ve ever used Opera gestures, drag up to stop downloading a page, drag down start a new tab). While writing a letter in graffiti to do a function may seem not too much easier than hitting a few keys, flicking the wrist once is alot quicker and more efficient than bringing your arm up from your side if it happens to be there, let alone clicking a drop-down menu, waiting for it to graphically fall down and then hunting for the option.

I don’t use the middle wheel to browse either, dragon. It’s too slow compared to dragging the side bar. I do, however, use it to open up links browsing tabs and to scroll through those tabs very quickly. Your mention of your mouse making its way to the end of your desk is exactly the reason I use mouse gestures. Instead of having to reach for little Xs or drop-down menus while I’m at my computer, my mouse hovers in one spot performing functions via small jerks quicker, with far less movement.

That’s one point. But back to Minority Report, the keyboard/mouse at Cruise’s desk didn’t suggest he was doing the boring part of his job, what he no doubt would do during 99% of his time in the office, with those fancy power gloves. The point of the gloves was speed when one of those balls drops and you’ve only got 13 minutes to figure out who the future murderer is, then catch the arm a split second before he stabs a pair of scissors into his wife’s face. Could you imagine Cruise moving his mouse pointer to that frame and then right clicking and waiting for a drop down menu so he can choose zoom? Then doing that again, and again, and again? No, he just waved the back of his hand forward several times in the direction of the photo and it picked up the gesture, taking 4 seconds instead of 20 seconds of precious Spielburg dramatic film time. No, I don’t expect everyone to be using power gloves in the air instead of the mouse. Maybe a glove on desk with support, but who knows. As far as nothing tactile, computers aren’t so rigid now-a-days. Voice recognition programs all learn now, and adapt to your voice. StrokeIt also learns what you want to teach it. If your C doesn’t look quite like a circle, then you can teach it your flattened version.

Again, don’t make this seem like an attack on the keyboard. If you’re typing, there’s no way a mouse is going to replace that. Speaking maybe, but not a mouse. It can, however, whittle down the need to use all those Ctrl, Alt, Apple, Function, etc. keys, at least for the most common functions and reduce the need to move the mouse to click Xs and windows/tabs. Of course, this ain’t so professional. But I expect it to get more standard with advanced users, just like how browser gestures are becoming now.

I wondered if and when eyeball gesture recognition software and/or hardware will be commonplace. I’ve drawn a quick picture of what I hope the dark and distopian future holds for borg-eyes. Click here for BORG VISION.

It’s like Issac Asimov and Austin Powers had a baby which grew up and became one of the cyclopses from The City of Lost Children.

On the other hand the borg were sort of cool in their own way. They had this kind of techno-gothy piece-meil style to their accoutrement that was kind fo appealing. If being a borg drone was less of a 24/7 job and more of a time-share thing I might be down with it.