Microsoft gives an accidental sneak peek of touch Office’s evolution

And the company's research into pen-driven interfaces remains ongoing.

Microsoft may have accidentally given the public a look at its hotly anticipated touch interface for Office, codenamed Gemini. Microsoft Research published—temporarily, at least—a presentation looking at ways that pen input could be better incorporated into the platform. Within it were a number of screenshots portraying some kind of touch interface for Word, Excel, PowerPoint, and Outlook.

Excel

Excel

Outlook

Outlook

PowerPoint

Word

Word

Word

Word

Word

Word

Though the document has been removed from Microsoft's servers, Paul Thurrott took a lot of screenshots. The document shows a number of ways in which stylus/pen-based annotations can be used to enrich the Office experience: sketching diagrams, adding annotations, making notes, and so on. The Office apps serve as a backdrop for this experimentation, and unlike the poor touch features in Office 2013, this time around, they appear to be true finger-driven applications.

A video presentation accompanied the document. In it, William Vong of the Xbox team and Tucker Hatfield of the Office team described the Office apps as "early prototypes." The different apps look quite different from each other, with PowerPoint and Word having a different concept from Excel, and Outlook being completely different again. Notably, the PowerPoint picture looks very different from the touch PowerPoint briefly demoed at the Build conference earlier this year.

The images also appear to be quite old. A few include the Windows Start screen, and they date back to the original Windows 8 start screen rather than 8.1 (the presence of the old Messenger app betrays their age).

The research indicates that Microsoft still isn't giving up on pen computing. The company's long dependence on the stylus on both smartphone and tablet devices arguably limited their mass-market appeal and made Microsoft slow to make the shift toward finger-based interfaces. Even in the presentation, Hatfield notes how painful it is that Microsoft went from a pen-based leadership position to an also-ran.

An inked out area in Internet Explorer...

An inked out area in Internet Explorer...

... is used to construct a reminder in Outlook.

Angry Birds doesn't support ink directly, but the overlay layer is always available.

Nonetheless, the pictures present a compelling picture of integrating pen input into the rest of the application. The concept moves away from Microsoft's old attempts to use the pen to drive the user interface, tapping buttons and so on, and instead uses the pen for what it is actually good at: laying down digital ink. Driving the interface would be the job for fingers, but the researchers envisage a kind of annotation layer above every application that can be used to sketch and write on. Some applications, such as Word, would recognize and preserve this layer. Others might not, but the inking could still be useful for making notes or other operations.

There's also a depiction of what's codenamed Oz, a natural language command system for Office. In Excel, the user writes (in cursive script) "find images of saddles" and is shown a set of Bing image search results.

Oz natural language commands.

Oz natural language commands.

The presentation makes mention of Microsoft's One Microsoft organization, suggesting that this research was a product of the company's newly collaborative structure.

The research indicates that Microsoft still isn't giving up on pen computing.

That's been its defining characteristic lately. But what fraction of apps on what fraction of its platforms are going to really run with stylus-based interactions? Isn't there somebody at Microsoft saying, “rather than having 5 alternative ways to italicize a section of text—5 different ways that devs will have to guarantee they don't run afoul of, accidentally mis-trigger, document and debug; 5 different ways that users will need to discover and remember gestures, shortcuts and etc for—let's make these two the primary ways on these platforms?

I wouldn't want Microsoft to be NOT thinking about 2020, but is optional, sorta-half-hearted stylus input the best way to differentiate and make Microsoft devices and services the must-haves?

The research indicates that Microsoft still isn't giving up on pen computing.

That's been its defining characteristic lately. But what fraction of apps on what fraction of its platforms are going to really run with stylus-based interactions? Isn't there somebody at Microsoft saying, “rather than having 5 alternative ways to italicize a section of text—5 different ways that devs will have to guarantee they don't run afoul of, accidentally mis-trigger, document and debug; 5 different ways that users will need to discover and remember gestures, shortcuts and etc for—let's make these two the primary ways on these platforms?

I wouldn't want Microsoft to be NOT thinking about 2020, but is optional, sorta-half-hearted stylus input the best way to differentiate and make Microsoft devices and services the must-haves?

From what I can see, a stylus is currently very important to the experience of the 8" Windows tablets, especially since they are all x86 models (until tomorrow?). If you actually want a non-frustrating desktop mode experience, that stylus is critical since the desktop is still a high-dpi interface. Currently, the 10" Windows tablets can work okay without the need for a stylus, but it is still a little cludgey with just a finger. With that in mind, the note taking is a bonus.

I wouldn't want Microsoft to be NOT thinking about 2020, but is optional, sorta-half-hearted stylus input the best way to differentiate and make Microsoft devices and services the must-haves?

I saw the video mentioned and I think the point is that they are trying to make this more of a core of Windows and less an add-on feature. It was mentioned that new development apis were being created (called direct write I think) that will hopefully make it easier for devs to do things with inking.

Microsoft is so damn stubborn, when will they realize that pen interfaces WILL NOT EVER be useful?! nobody likes them because they suck, and it's nothing to do with technology, using a pen sucks when you can just use your finger.

But that's crap. Pens are much better for sketching and writing with, which is why people continue to pay Wacom a pretty penny for their tablets.

I've been using stylus input on windows since 2008 and it's been great. The evolution of the operating system to make the pen more versatile and more importantly improve touch response has only improved the situation. Pen is simply better than touch for productivity - taking notes with a stylus is awesome.

So far in the comments, some people do not get that what we are seeing here is an early prototype/build of MS Office, touch edition. The overlay stuff is more a system-wide thing that you wouldn't actually need but which you may want given your use-case or interaction preferences.

Maybe it is the articles fault, but some people in here seem like they are being deliberately dense and bias about things. Current Windows 8.1(.1) works well on a tablet and a desktop - I use it on both form factors daily. Adding touch office to that AND this pen-driven overlay system makes it even better, and no more or less touch-focused as before.

The traditional desktop UI may be a suboptimal using fingers on screens under 11" but what point is there in heavy desktop UI use on form factors that are practically designed around consumption-only? Never mind that with the x86 small tablets, you can use docks if you actually did want to do "serious work" on them regardless.

Microsoft is so damn stubborn, when will they realize that pen interfaces WILL NOT EVER be useful?! nobody likes them because they suck, and it's nothing to do with technology, using a pen sucks when you can just use your finger.

But that's crap. Pens are much better for sketching and writing with, which is why people continue to pay Wacom a pretty penny for their tablets.

K, they're better for drawing, but you've got a keyboard for writing so that claim is bullshit too.

So... it's ok to have a nice keyboard dock option for tablet but not also a stylus? Also, sometimes just writing something down quick on the screen IS faster/better. In fact, given the right tech, for many use cases, why should there ever be a keyboard?

For something like the Surface RT, I would take great stylus support over the keyboard support, but better would not being forced to choose in the first place.

Microsoft is so damn stubborn, when will they realize that pen interfaces WILL NOT EVER be useful?! nobody likes them because they suck, and it's nothing to do with technology, using a pen sucks when you can just use your finger.

But that's crap. Pens are much better for sketching and writing with, which is why people continue to pay Wacom a pretty penny for their tablets.

K, they're better for drawing, but you've got a keyboard for writing so that claim is bullshit too.

I guarantee you that you are significantly underestimating the percentage of people who can hand write faster than they can type. For many people, a keyboard is a great way of inputting information (I happen to prefer it to handwriting, myself) and I'm sure it will be a primary means of interacting with computers for quite some time, but there's no reason to act like it's some kind of inevitable force of history.

Microsoft is so damn stubborn, when will they realize that pen interfaces WILL NOT EVER be useful?! nobody likes them because they suck, and it's nothing to do with technology, using a pen sucks when you can just use your finger.

But that's crap. Pens are much better for sketching and writing with, which is why people continue to pay Wacom a pretty penny for their tablets.

K, they're better for drawing, but you've got a keyboard for writing so that claim is bullshit too.

That really depends on your use case. Personally, the primary reason I got the Surface Pro was note taking, because that is way too slow with a keyboard if you need to use things like mathematical symbols for anything beyond arithmetic (good luck finding the "definite integral from 1 to pi" key), or anything where you also want to insert drawings and diagrams made on the fly side by side with the text.

Plus the pen can serve as a more accurate pointer than a finger then I don't feel like folding out the keyboard to use the touch pad mouse. Arguably, that can be said to be a flaw in Window's touch interface rather than a pro of the pen interface though.

I'd like to see MS make it so the stylus can act as a universal/generic mouse input when running non-stylus optimized programs.

For instance, running FTL (a game) in full screen mode will cause the stylus to be ignored/disabled. Running that same game in windowed mode will allow the stylus to work properly, though. Curious, that.

Microsoft is so damn stubborn, when will they realize that pen interfaces WILL NOT EVER be useful?! nobody likes them because they suck, and it's nothing to do with technology, using a pen sucks when you can just use your finger.

But that's crap. Pens are much better for sketching and writing with, which is why people continue to pay Wacom a pretty penny for their tablets.

From his comment, it sounds like he's unaware of the distinction between a typical capacitive stylus and an active digitizer like Wacom's in the Surface Pro.

Microsoft is so damn stubborn, when will they realize that pen interfaces WILL NOT EVER be useful?! nobody likes them because they suck, and it's nothing to do with technology, using a pen sucks when you can just use your finger.

But that's crap. Pens are much better for sketching and writing with, which is why people continue to pay Wacom a pretty penny for their tablets.

From his comment, it sounds like he's unaware of the distinction between a typical capacitive stylus and an active digitizer like Wacom's in the Surface Pro.

I've been using stylus input on windows since 2008 and it's been great. The evolution of the operating system to make the pen more versatile and more importantly improve touch response has only improved the situation. Pen is simply better than touch for productivity - taking notes with a stylus is awesome.

I don't know either. If the latest OneNote is as good as the older software, and the battery of the Surface Pro (III?) is decent I may migrate back to the Windows meadows. At least for my portable needs.

OneNote is how a computer interface should work, even if it needs some visual love. That said, I've virtually integrated with Evernote due to its platform agnostic

Microsoft is so damn stubborn, when will they realize that pen interfaces WILL NOT EVER be useful?! nobody likes them because they suck, and it's nothing to do with technology, using a pen sucks when you can just use your finger.

But that's crap. Pens are much better for sketching and writing with, which is why people continue to pay Wacom a pretty penny for their tablets.

From his comment, it sounds like he's unaware of the distinction between a typical capacitive stylus and an active digitizer like Wacom's in the Surface Pro.

From his comment, it sounds like he's unaware.

So, people see Microsoft working on styli and Office and they're right to be skeptical. These are THE TWO obsessions that have held Microsoft back in mobile. Styli have encouraged lazy UI design, and Office is a system seller - on PCs, not mobile.

Perhaps a fresh start like Courier might have cut through early in mobile. Smartphones came with styli at one point, after all. It's too late now - the agenda has been set by other companies and anything new would be easily copied, anyway. I look at this and just think it's useful for some business customers at best (as it always was) and a crutch to bad UI design at worst. It's a distraction unless this is all they're going to be.

I think, too, it's ultimately a limited time tech. Once screens are dirt cheap and measured in feet - perhaps literally painted on walls - your fingers will be enough. You don't need smaller fingers, in other words, if you have larger screens. Look at the xbox projector tech, look at kinect. We're not there, yet, but neither are styli mainstream.

I think artists will always have them. Maybe gamers will always have joysticks and programmers will always have keyboards. But the point is, specific tools for specific scenarios. Microsoft needs to move the needle in the mass market, not just in business edge cases like tablet PCs always did.

With all this discussions about "love and hate" of the stylus, I don't see anything in the article that says Microsoft is sticking to stylus only input. Perhaps you could use fingers, perhaps you could use on screen keyboards, perhaps you could also use keyboard docks. Perhaps you could even use Kinect for your input.

The point being, the article talks about a "TOUCH OPTIMIZED MS OFFICE". Its not just stylus only input right?

If you need to use a pen for your UI, your widgets are too small, or you have not made effective use sof multi-touch/gestures.

Not to say that there is no place for pen-baed input, of course there is.

But reliance on a pen for general purpose use is a symptom that your UI is not touch friendly.

For text input that isn't drawing/scribbling though? I can type faster than i can write by a long shot - even on an a smartphone's touch keyboard.

That said - OneNote (or similar note-taking app) on the Surface or other Windows tablet with a pen interface is a killer app. Which makes it fairly inexcusable that we're 2 years after the release of surface and Microsoft still hasn't pulled their finger out and made a proper touch friendly version of Office for surface.

I have users crying out for Office on a tablet, and at the moment, VDI with View on an ipad is a superior office experience to Office on Surface natively for the simple reason that View allows you to zoom into the screen to hit tiny office widgets reliably with your finger. You can't zoom in on UI widgets, with locally running Office on Surface - and they are ridiculously small on a 10" 1920x1080 screen.

MS needs to push this not to just business but education. Every kid learns maths, chemistry and physics and these are the subjects where a keyboard is simply not sufficient due to all the symbols which doesn't have a dedicated key and switching to software symbol keyboards is cumbersome and non-intuitive. As we push more e-learning into the classroom, the stylus is going to be an essential tool where traditional laptops is not going to cut it no matter how many times haters cry "I can type faster than I write" for these subjects because there is no way you can type an algebra, calculus or balancing a chemical equation faster than writing it with current keyboard and mouse combo. This sector will also never go away and will always have new users (unless we stop teaching kids maths, chemistry and physics, or voice technology gets smart enough to dictate such equations). It's not a niche usage case, it's going to be an essential extra input tool, it's surprising so many technology guys don't see it considering they are the ones whom had to be head deep in all the maths subjects during school.

Whatever they do they need to do it fast. As an owner of a Surface Pro and full Office 2013 it was a little bit insulting to see the Office for iPad come out with far better touch compatibility than I could get!

Excellent work in reinforcing the impression many have that the only good tablet round is an iPad Microsoft!

MS needs to push this not to just business but education. Every kid learns maths, chemistry and physics and these are the subjects where a keyboard is simply not sufficient...

There is no evidence that tablets help kids learn, its all wasted money. One-on-one interaction with a teacher (and a meal) are the largest factors in improving learning.

It doesn't matter - computers and tablets are making their way into schools no matter what, and people will want to get more value out of them - it's a trend that will not reverse, just like no matter how feature poor the original iPhones were people made them work in a corporate envrionment. A lot of rich private schools where the family has no problem feeding their kids are already mandating each child having their own laptop for use in school. And let's not even get into university level study where you can't live without a laptop/tablet. So MS making their programs/OS more useful by deep integrating stylus support where it has a real usage case such as in the subjects I've listed as its differentiator with iPads/non-touch enabled laptop is only going to be beneficial and leading to better business for MS.

With all this discussions about "love and hate" of the stylus, I don't see anything in the article that says Microsoft is sticking to stylus only input. Perhaps you could use fingers, perhaps you could use on screen keyboards, perhaps you could also use keyboard docks. Perhaps you could even use Kinect for your input.

The point being, the article talks about a "TOUCH OPTIMIZED MS OFFICE". Its not just stylus only input right?

I'm guessing most people didn't actually watch the presentation, because it actually starts out with the presenter covering most of the problems with pen input in Windows so far. Namely that it's treated too much like a mouse, watering down the experience. It's used inconsistently and so makes people unaware of the fact that you can ink at all. And even when you know you can, it takes far to long to get to a point that you can usefully take notes or annotate something. One of the key points about rethinking pen usage is that it should always be an accompaniment to touch or keyboard/mouse and not a replacement.

The research indicates that Microsoft still isn't giving up on pen computing.

That's been its defining characteristic lately. But what fraction of apps on what fraction of its platforms are going to really run with stylus-based interactions? Isn't there somebody at Microsoft saying, “rather than having 5 alternative ways to italicize a section of text—5 different ways that devs will have to guarantee they don't run afoul of, accidentally mis-trigger, document and debug; 5 different ways that users will need to discover and remember gestures, shortcuts and etc for—let's make these two the primary ways on these platforms?

I wouldn't want Microsoft to be NOT thinking about 2020, but is optional, sorta-half-hearted stylus input the best way to differentiate and make Microsoft devices and services the must-haves?

FYI. My wife uses the pen to drive web pages using Google Chrome. Sure, it works with fingers, but she actually prefers using the pen that was supplied with her Surface

I wouldn't want Microsoft to be NOT thinking about 2020, but is optional, sorta-half-hearted stylus input the best way to differentiate and make Microsoft devices and services the must-haves?

I saw the video mentioned and I think the point is that they are trying to make this more of a core of Windows and less an add-on feature. It was mentioned that new development apis were being created (called direct write I think) that will hopefully make it easier for devs to do things with inking.

My big gripe about Windows since XP is that they've trashed keyboard shortcuts that I knew without thinking, provided bloated ribbons, monster icons that steal screen real estate for functions of zero interest to me, and otherwise tried to make it more accessible to everybody besides the expert user.

If stylus input is not just an add-on feature, but a bedrock UI idiom, bring it on; it'll be useful and we'll all (learn to) appreciate it. But it seems ultra-bizarre that I'll suddenly want to use a stylus on my four-screened desktop device as a primary way of interacting with it, after (1) Alt-Tab and the right shortcut keys, (2) mouse-based menus and icons, and (3) those friggin ribbons.

I suppose it'd be better than having Microsoft spec Kinect as a required UX element, so I'd need to roll my head in a counterclockwise circle when I wanted to go up a page or two in a document. But not much.

I do a lot of basic desktop publishing for my kindergarten using publisher. I used to use more advanced design programs but they are too advanced and expensive for the teaching staff to use. Lately I have moved most of my office work onto a tablet so I can work while visiting and observing the classes and while it's still usable, I find the publisher implementation of touch hugely inferior to how it works in excel. I'm waiting for tomorrow's announcements before buying but I am leaning heavily towards a surface pro just because the pen will make it much more usable.

On a side note, next time you are on the phone with ms can you guys ask if there are any future plans for publisher? It doesn't seem to be progressing as fast as the rest of office.

where traditional laptops is not going to cut it no matter how many times haters cry "I can type faster than I write"

Almost every student and worker can type FAR faster than they can write - a pen cannot replace a keyboard. For pure text entry, a stylus SUCKS.

That said, Surface Pro (2) with the TypeCover works better than a laptop in almost any not-in-your-lap situation - it's decent for typing on and also has a stylus for things that aren't text - eg: equations, pictures, etc.

I wouldn't want Microsoft to be NOT thinking about 2020, but is optional, sorta-half-hearted stylus input the best way to differentiate and make Microsoft devices and services the must-haves?

I saw the video mentioned and I think the point is that they are trying to make this more of a core of Windows and less an add-on feature. It was mentioned that new development apis were being created (called direct write I think) that will hopefully make it easier for devs to do things with inking.

My big gripe about Windows since XP is that they've trashed keyboard shortcuts that I knew without thinking

Windows 8(.1) has more keyboard shortcuts than any other version of Windows - ever. Off the top of my head I'm not actually aware of any keyboard shortcuts being removed in versions of Windows after XP - care to give some examples?

Quote:

provided bloated ribbons

Oh, I see - you're talking about Office, not Windows.

Quote:

monster icons that steal screen real estate for functions of zero interest to me, and otherwise tried to make it more accessible to everybody besides the expert user.

Pre Office-2007 there was almost no such thing as an expert user - virtually the only people who thought they were experts were the ones that had no clue of the scope of the Office applications. Office 2003 (and earlier) was a disgusting mess of bad UX decisions. Those one or two REAL expert users that did exist exclusively used the keyboard - and thankfully there were no keyboard shortcuts removed (in fact even more were added) after the much-needed overhaul of Office in 2007.

where traditional laptops is not going to cut it no matter how many times haters cry "I can type faster than I write"

Almost every student and worker can type FAR faster than they can write - a pen cannot replace a keyboard. For pure text entry, a stylus SUCKS.

Good for you for taking my entire argument out of context, however I'm not talking about pure text entry:

the stylus is going to be an essential tool where traditional laptops is not going to cut it no matter how many times haters cry "I can type faster than I write" for these subjects because there is no way you can type an algebra, calculus or balancing a chemical equation faster than writing it with current keyboard and mouse combo.

Show me someone who could type out a 3rd order differential equation (this is typical highschool level maths) faster on a keyboard and mouse with accompanying notes than drawing with a stylus. These are the situation where a pure keyboard and mouse laptop absolutely SUCKS. A stylus has a HUGE usage scenario in maths/sciences/diagram heavy subjects, and there are no shortage of these classes in school. Haters just keeps ignoring it saying stylus has no place anywhwere or that the usage is so niche, but is just plain not true as again, every kid must learn some maths/chemistry/physics some time in their life at school, and as I've said learning going electronic is not a reversible trend. And once we get to university, then every science and engineering student also greatly benefit from such input. This segement of the market will always be there (unless you think we stop teaching our children), so again, aiming and developing pen input has real, actual usages which haters is just choosing to ignore.

where traditional laptops is not going to cut it no matter how many times haters cry "I can type faster than I write"

Almost every student and worker can type FAR faster than they can write - a pen cannot replace a keyboard. For pure text entry, a stylus SUCKS.

Good for you for taking my entire argument out of context, however I'm not talking about pure text entry:

the stylus is going to be an essential tool where traditional laptops is not going to cut it no matter how many times haters cry "I can type faster than I write" for these subjects because there is no way you can type an algebra, calculus or balancing a chemical equation faster than writing it with current keyboard and mouse combo.

Show me someone who could type out a 3rd order differential equation (this is typical highschool level maths) faster on a keyboard and mouse with accompanying notes than drawing with a stylus. These are the situation where a pure keyboard and mouse laptop absolutely SUCKS. A stylus has a HUGE usage scenario in maths/sciences/diagram heavy subjects, and there are no shortage of these classes in school. Haters just keeps ignoring it saying stylus has no place anywhwere or that the usage is so niche, but is just plain not true as again, every kid must learn some maths/chemistry/physics some time in their life at school, and as I've said learning going electronic is not a reversible trend. And once we get to university, then every science and engineering student also greatly benefit from such input. This segement of the market will always be there (unless you think we stop teaching our children), so again, aiming and developing pen input has real, actual usages which haters is just choosing to ignore.

Not to mention, a stylus is (in principle) silent. If you've ever been in a smallish classroom with everybody clattering away, you'll appreciate this is a great feature .

I like this. I use a Livescribe and sync (via Onenote2010) as note taking with a pen is a lot quicker than typing and the notes make far more sense than typing alone (something about the way you can scribble quick diagrams next to things for one)

I would love to replace my Livescribe notepad with this. Some pen features work on the RT with a capacitive pen, but it lacks definition.