Apple seeks patent on resolution independent user interface

Apple Computer is looking to secure exclusive rights to techniques required to produce resolution independent user interfaces, one of the supported features of its forthcoming Mac OS X Leopard operating system.

The Cupertino, Calif.-based Mac maker described several approaches to the technique in a July 21, 2006 filing with the United States Patent and Trademark Office titled "Resolution Independent User Interface Design", which was first published on Thursday.

Essentially, the technology breaks the software assumption that all display output is to be rendered at 72 dots per inch (DPI), allowing an operating system like Mac OS X to draw or represent user interface elements using a scale factor.

"The invention provides a method to represent a graphical user interface object's material map in a procedural and, therefore, resolution independent manner," Apple wrote in the filing. "The method includes receiving values for each of a plurality of attributes associated with a material map object, associating a value for each of the plurality of attributes, and storing the plurality of attributes and their associated values in a file."

The company said 'recipe' files for resolution independent user interfaces may be "flat" or hierarchically-ordered, where a collection of attribute-value pairs form a complete description of the graphical user interface object's material map, hence allowing a rendering module to create a visual representation of the material map at any number of resolutions.

In one interesting aspect of the otherwise predominately procedural filing, Apple notes that "because material maps in accordance with the invention are represented procedurally, they may be encrypted to prevent unauthorized inspection or use."

Another benefit of the technology, the company said, is that each user interface attribute may be associated with a plurality of values, thereby permitting the designer to optimize an object's design for each of a specified number of resolutions.

In a case where the displayed resolution of the graphical object falls between two of the resolutions specified by the designer, a rendering engine may interpolate between the two values -- a technique that generally provides a significantly improved display -- Apple added.

Earlier this year, Apple informed developers through documentation for Leopard that the introduction of resolution independence means that they may have some work to do in order to make their applications look as good as possible.

"For modern Cocoa and Carbon applications, most of the work will center around raster-based resources," the company said. "For older applications that use QuickDraw, more work will be required to replace QuickDraw-based calls with Quartz ones."

Apple's intention to support resolution independence with Mac OS X was first detailed in an August 2004 AppleInsider report.

As long as everyone is playing dirty, it is essential that Apple patent any innovations, regardless of whether or not they should be patented! Whether or not they should go chasing other people is another story, but at least they aren't getting screwed over or having to settle like they have had to in the past.

That's nice. I hope it offers the ability to make all the text larger without breaking window layouts or downresing the monitor. Then that would allow higher pitch displays and still be readable to more people.

If the patent is granted, Microsoft finally won't be able to steal/copy another MacOS feature.

Patents are 2 fold: 1 is getting the patent which ANYONE can get on ANYTHING.

2 is suing anyone who violates it, and winning.

I could have a patent on the wheel from 2000 BC, but it doesn't do any good if I don't have the scratch to sue firestone.

You have your patents, they have theirs. Many times, people are given patents for something that have already been patented. In that case, enforcement of the patent lies on the original person. That person has to go to court to get rid of the second patent, or they can go to court when the johnny-come-lately starts suing others (far more likely).

So Apple patents this resolution independent thing. M$, 10 years from now, incorporates it. A few years after that, M$ sues Apple for fast user switching. Apple counter-sues because M$ is using the resolution independent thing. Sun Microsystems sues everyone because they're going bankrupt and they actually own patents on everything. Hilarity ensues.

It's all about leverage and NOTHING about intellectual property. There is simply too much sh¡t out there to figure out if you have something they patented or visa versa.

If the patent is granted, Microsoft finally won't be able to steal/copy another MacOS feature.

Did you ever wonder why Apple never sued Microsoft (or vice versa) and probably never will?!

It is because patents are not only about intellectual property but they are primarely a powerful tool for negotiating! Meaning, Apple will allow Microsoft to use some of his patents in exchange for Microsoft patents that Apple might need.

So, patents are just a part of the game and very important if you wanna do real business with someone!

Well, just based on the article posted here, it doesn't sound much different than what Palm OS does to handle multiple resolutions (you can have a collection of different images, etc. that are appropriate for a specific scale/resolution). It's really unclear what they are trying to patent though: a file structure, a way of organizing files, a collection of attribute value pairs??? I mean resolution independence by itself is really, really, really old news ... try 15-20 years ago.

Well, just based on the article posted here, it doesn't sound much different than what Palm OS does to handle multiple resolutions (you can have a collection of different images, etc. that are appropriate for a specific scale/resolution). It's really unclear what they are trying to patent though: a file structure, a way of organizing files, a collection of attribute value pairs??? I mean resolution independence by itself is really, really, really old news ... try 15-20 years ago.

I don't see how it applies to Palm OS here since thats only running 1 application at a time at full screen. It's the same thing as running a game on a PC that is set to a different resolution than the desktop.

BTW, I'm part of the staff at PalmInfocenter and Palm Inc's Support Forums.

From the screenshot, to me, it's nothing like PalmOS. What PalmOS does is store bitmap images at various resolutions and it picks the most appropriate. That's also what MacOSX does now. Try changing the size of icons in the Finder for instance and you'll see that they don't simply scale down from huge 128x128 bitmaps - there's a few steps along the way.

What Apple are patenting here is a method of storing procedural markup that draws an interface element at a specific resolution. They can store a different procedure at different resolutions which is useful because scaling a 150dpi image down to 72dpi often doesn't look good.

So, they can describe how to create a button at 72dpi differently from at 150dpi and even map several descriptions (or 'recipes') to different resolutions if they're all similar enough.

As some of the screenshots of the last Leopard build show, parts of the Finder has got new scalable interface elements now. They look terrible too, standing out from the old bitmaps, so one hopes Apple has a whole new procedural recipe squad cooking up a new UI before Leopard ships. I'd hope we'd see that at MWSF as it'll otherwise give developers no time at all to redo their UIs.

This sure sounds an awful lot like vector-based graphics (such as what can be created in Adobe Illustrator or Flash) taking over the whole display, or to go back even farther, it sounds like EPS images for the entire display. I always thought EPS was a great format, but it seems that it is hardly, if ever, used today.

Resolution independence is going to be extremely important for the OS base of the iTV, whether it's a closed OS, such as a un-exitable update to FrontRow, or an actual OS X variation (I guess it will be an OS X variation at some level regardless because that's what FrontRow runs on).

I've got Mac minis hooked up to both of my TVs and while one looks ok, but not really good (57" rear projection HD-ready Toshiba connected via DVI, which won't center properly on the screen), the other is nearly unreadable (Philips 27" widescreen flat tube HD-ready TV connected via a DVI to HDMI converter cable, but it centers properly). FrontRow looks good, but it's icons and text are huge compared to what is displayed on the Mac desktop. You can't really lower the resolution until it's easily readable because each resolution centers on the display differently, particularly on a TV's DVI input in my experience, sometimes unacceptably, and many application windows just aren't designed to be displayed at less than 800x600 at the very minimum. My Philips TV won't go to any lower resolution and a higher resolution where the text is even smaller is just going from bad to worse. It's really funny to boot up my old P'Mac 7500 in OS X with the original 640x480 display. You literally cannot access everything in the System Prefs window. At least, you couldn't back in 10.1 or 10.2 -- it's been awhile since I've even tried that, and the reason was that the prefs window wouldn't resize small enough and/or become scrollable (I think it becomes scrollable in Tiger).

Maybe they can get a patent for this, but on the surface, it appears to me that the concepts for displaying graphics in a completely resizable manner, such as they are describing to achieve "resolution independence," have been around a very long time (as others have noted). Still, I can't wait to see if Leopard will look better on my Mac mini-driven TVs.

I don't know much about the math behind the concepts noted here, but I do know that procedural textures are used in 3D graphics to simulate extremely realistic textures. I would almost assume that this knowledge is a carryover from Steve's Pixar involvement.

This sure sounds an awful lot like vector-based graphics (such as what can be created in Adobe Illustrator or Flash) taking over the whole display, or to go back even farther, it sounds like EPS images for the entire display. I always thought EPS was a great format, but it seems that it is hardly, if ever, used today.

Well, I still think this sounds more like having different images for different resolutions (and that's what it looks like in the figure), but in any case, it just sounds like a mapping of UI elements to multiple methods of rendering based on the resolution. A fairly basic computer science data structure.

If they wanted to patent something for vector-based resolution independence, I thought they already owned all the NeXT Display PostScript stuff (i.e. over 15 years old).

Quote:

Maybe they can get a patent for this, but on the surface, it appears to me that the concepts for displaying graphics in a completely resizable manner, such as they are describing to achieve "resolution independence," have been around a very long time (as others have noted). Still, I can't wait to see if Leopard will look better on my Mac mini-driven TVs.

Resolution independence is vector graphics, but with the aide that Quartz will rasterize the view, on the fly as you scale the object.

Scalar factors with the vector make it straight forward; and since screens are dumb devices they don't know anything about vectors which requires the Window Server to coordinate with Quartz PDF primitives to do the work.

Say you have a line 1 inch long that is 1/72 of an inch thick. On Mac OS X (and Mac OS before it) that one inch line would be defined as 72 points long and one point thick since Mac OS X defines it point space (what you draw in) as having 72 points per inch.

Currently Mac OS X defines that one point maps to one pixel on a display.

So if you had a display that had 72 pixels per inch (physical pixels crammed into one inch of length along the display's surface) then a 72 point long line would appear to be 1 inch long on that display. Now if you had a display with 144 pixels per inch then a 72 point long line would appear to be 1/2 inch long on that display and on a 288 pixel per inch display it would appear to be 1/4 inch long.

So as you get displays that can cram more and more pixels per inch your on screen drawing will appear to be physically smaller and smaller unless you adjust the scale of what you are drawing. In other words as the fidelity of displays approaches that of common laser printers you need to adjust what you draw to use more pixels so it maintains the same appearance in terms of physical dimensions.

This is what resolution independent UI is about... Mac OS X in the near future will allow you to (or possibly automatically) set the scale factor for the visual environment to match the physical size of pixels of your display.

For example if you had a 144 pixel per inch display you would ideally set the scaling factor to 2x. That would mean a 72 point long line (aka 1 inch long line) that is 1 point tall would be displayed on your screen using a rectangle that is 144 pixels long (72 x 2) and 2 pixels tall (1 x 2). In other words that 1 inch long line would appear as 1 inch long on your display instead of just a half inch long and it would be displayed using twice as many pixels... which means you have a higher fidelity image.

It should be noted that since day one Mac OS X has utilized a resolution independent drawing environment (Quartz / Quartz2D) and that is how what you see on screen is appropriately scaled when you print it out on say a 600+ DPI printer without out much of a through by a programmer (at least for those using modern APIs based on Quartz or using Quartz directly). What Apple is currently doing is starting to utilizing this existing capability to scale what is drawn to the screen since screens are improving in DPI.

As long as everyone is playing dirty, it is essential that Apple patent any innovations, regardless of whether or not they should be patented! Whether or not they should go chasing other people is another story, but at least they aren't getting screwed over or having to settle like they have had to in the past.

Actually I happened to patent the basic idea of the resolution independent UI one month before Apple - I first submitted it to the USPTO on May 31. 2004, and I see now that Apple's first appilcation on this subject was on June 24, 2004. This is not a joke.

So I guess I can now make a deal either with Microsoft or with Apple. Whichever one is going to be more reasonable.