Apple: pixels as touch sensors for brighter, thinner screens

How can you make a touchscreen thinner and brighter at the same time? Use the …

Touchscreens and multitouch technology make up a significant majority of Apple's research into future user interface improvements, and the iPhone introduced some of those UI paradigm shifts into our increasingly mobile computing. Since almost all interaction with the iPhone—and presumably the hopefully imminent Apple tablet—involves a touchscreen, Apple hopes to improve on touchscreen technology by using each individual LCD pixel as a touch sensor.

Apple has filed a patent application, published today, for a "display with dual-function capacitive elements." By mixing display and sensing functions into each individual pixel, it would make touchscreens thinner, lighter, and brighter than they currently are today.

The way current touchscreens found on most smartphones work is by overlaying a touch-sensitive panel on top of a traditional LCD panel. The touch-sensitive panel is essentially a grid array of capacitors, most commonly made from the transparent conductor indium tin oxide (ITO). When your fingertip comes in contact with the small magnetic fields present in the capacitors, it causes the voltage along those capacitors to fluctuate. A processor translates these fluctuations into touch positions.

The need for additional layers covering the LCD screen means it is thicker, and despite the fact that ITO is transparent, the touch layer does block some light coming from the LCD display underneath. Apple's solution involves using each individual pixel as a capacitive sensor, eliminating the need for an additional layer for a separate touch sensor.

Part of the magic of Apple's patent relies on forming an IPS LCD display using a low temperature polycrystalline silicon instead of the more common amorphous silicon. Materials engineering nerds may want to look at the patent for a more detailed explanation, but suffice it to say that the poly-Si allows for a much faster switching frequency for driving the individual pixels. (For those unaware, the individual pixels in an LCD panel switch on and off at a rate much faster than we can perceive—it's this same switching that can cause eye fatigue from staring at your screen all day.)

Apple's idea takes advantage of the faster switching of poly-Si to drive the pixels one instant, and use the capacitive properties of the individual pixels as touch sensors the next. The switching happens fast enough to give a clear, bright display, as well as responsive touch sensing. The elimination of the separate touch-sensing layer also makes for a thinner, lighter, brighter, and simpler touchscreen unit.

Apple proposes its solution for mobile devices, making references to iPhones, iPods, and even MacBooks, but don't be surprised if such an innovation also makes its way into an Apple tablet.

24 Reader Comments

If this works as well as one might think it would, I see this as a major advance. Especially if it can be extended to support OLED screens. Now all Apple has to do is to go one step further and replace those glass screens with some sort of break resistant ceramic. Time to build all of this on transparent aluminum.

This is freakin' cool. I might not personally be into Macs myself, but they are always coming with the innovations (not sure which people you're listening to, wizard69, but they sound like they don't know what they're talking about ).

Originally posted by wizard69:If this works as well as one might think it would, I see this as a major advance.

I don't see how it's a major advance. How thick is the ITO layer anyway? A few nanometers? A few micrometers at worst? The reduction in thickness and transparency will be negligible.

quote:

Especially if it can be extended to support OLED screens.

The basic idea - interleaving the functionality of a pixel - won't work as well with OLEDs (at least not for the materials used today) because the carrier mobility of organic materials and thus the switching speed is even slower than that of amorphous silicon.

Maybe flicker can cause eye fatigue, although at 60hz or above, more likely causes include lack of blinking, long duration close focusing, and lack of scanning eye (and head and neck) movements.

Anyway, this invention doesn't suggest using poly-si's faster switching for improving flicker rate; it's for the extra time needed for capacitive sensing while using the same circuit elements as for display.

Also, if the patent were to be used to increase display brightness, it would actually make the flicker more perceptible at the same refresh rate. However, brightness is always a tradeoff with power/battery life, so the displays wouldn't *necessarily* be brighter if, say, Marketing believes longer battery life would sell more units.

Although this doesn't sound like an inexpensive display to produce, the extra display cost may compare favorably to the cost of integrating the existing extra layers, especially if Apple can leverage intellectual property like this in its supplier negotiations. Cynic that I am, that seems like the most probable motivation from Apple's point of view.

Originally posted by grimlog:I don't see how it's a major advance. How thick is the ITO layer anyway? A few nanometers? A few micrometers at worst? The reduction in thickness and transparency will be negligible.

The author of the article is a bit misleading---the reduction in brightness is mostly caused by the Fresnel loss due to the index mismatch between the ITO and substrate and ITO and air. You are correct in saying that the absorptive loss will be small. However, some of the light will be reflected off the ITO back into the device, and will probably be absorbed by the rear wall behind the screen, or some of the other components. This is where the majority of the reduction in light will come from.

I could have sworn that I read about a Phillips/LG display that was rocking this tech earlier. AFAIK the idea isn't new, just apples specific planned implementation. But seeing how I can't actually find the link right now, I guess you can file my post under trollbait.

Wouldn't this also improve precision? I mean, if you have two layers, you'll have to calibrate them to align. If it's all in one layer, it will be aligned no matter what.

I've only used the iPhone's multitouch displays but I must say that I find it amazing how well it works. With new tech like this I'd expect glitches like loss of alignment og loss of control, but I've never encountered and iPhone that didn't work perfectly with regards to the multitouch input. Even those with broken displays still work flawlessly. I find that amazing – esp. since the tech is so relatively new.

Originally posted by Sneakyness:I don't think anybody's realized this would mean full 1:1 resolution on touch input. That's good enough for drawing.

Actually, a higher than 1:1 ratio is better for drawing. Check out the specs of a Wacom tablet. It allows for better anti-aliasing for raster image drawing at 100%. It also means more accuracy when drawing at less than 100% magnification.

For those unaware, the individual pixels in an LCD panel switch on and off at a rate much faster than we can perceive.

Something I've noticed: Under direct sunlight, I often see flickering on some parts of the iPhone's screen, but never indoors. I'm guessing it's the sheer brightness that makes it more noticeable to the eye.

Hm... Maybe I'm wrong but don't the capacitors need to be on or very near the surface of the touch screen? In that case, wouldn't this sort of integration put the pixels at risk from scratches and the like?

Originally posted by grimlog:I don't see how it's a major advance. How thick is the ITO layer anyway? A few nanometers? A few micrometers at worst? The reduction in thickness and transparency will be negligible.

I dunno. When I was doing a laptop hunt a while ago, I was going to buy a HP Touchsmart, specifically because it had a touchscreen and could be switched into tablet mode. However, when I got to play with one in the shop, the display completely put me off: the screen was dull and grainy.

Admittedly, this may just have been due to the technology HP was using, but I think we tend to be more forgiving with mobile tech - the screens are smaller, the resolution lower and the battery-life impact of boosting the brightness is comparitively less than on a laptop.

Originally posted by daemonios:Hm... Maybe I'm wrong but don't the capacitors need to be on or very near the surface of the touch screen? In that case, wouldn't this sort of integration put the pixels at risk from scratches and the like?

The voltage applied to capacitive touch sensors can actually be adjusted to project a magnetic field to whatever necessary distance is needed. In the case of the iPhone, for instance, the coated glass covers the ITO touch layer, and it is calibrated to extend the touch sensitivity to just beyond the surface of the glass.

The effect is (I'm told by Synaptics) due to the water content in your finger-tip, which is why the iPhone (and other capacitive touch screens) don't work with fingernails. It's also why you sometimes have to give a little push, if your fingertip is well-calloused.

Originally posted by yogaman:Maybe flicker can cause eye fatigue, although at 60hz or above, more likely causes include lack of blinking, long duration close focusing, and lack of scanning eye (and head and neck) movements.

IME, some are more sensitive to this than others.

quote:

Anyway, this invention doesn't suggest using poly-si's faster switching for improving flicker rate; it's for the extra time needed for capacitive sensing while using the same circuit elements as for display.

Originally posted by new2mac:Apple pixels are sure going to be working very hard. They're going to be touch sensors AND video cameras. Remember that behind-the-screen videochat camera patent? That was cool.

Originally posted by grimlog:I don't see how it's a major advance. How thick is the ITO layer anyway? A few nanometers? A few micrometers at worst? The reduction in thickness and transparency will be negligible.

The author of the article is a bit misleading---the reduction in brightness is mostly caused by the Fresnel loss due to the index mismatch between the ITO and substrate and ITO and air. You are correct in saying that the absorptive loss will be small. However, some of the light will be reflected off the ITO back into the device, and will probably be absorbed by the rear wall behind the screen, or some of the other components. This is where the majority of the reduction in light will come from.

Originally posted by foresmac108:The effect is (I'm told by Synaptics) due to the water content in your finger-tip, which is why the iPhone (and other capacitive touch screens) don't work with fingernails. It's also why you sometimes have to give a little push, if your fingertip is well-calloused.

I've found that my macbook will wake up from detecting my gloved hand on the trackpad, but to actually be able to use it, the fingertips need to be wet.

I recall that apple also applied for (and presumably got approved) a patent for integrating the camera into the display. This coincidentalley makes med think of a hybrid ccd + ips tft panel. THAT would be something. Throw in 3D and you'll get everything the rumors have been promising. All while Apple will be able to cut cost. Imho.

I remember once seeing a video of somebody that made "stereo" pictures that you could shift focus within afterwards, while editing it. If anyone have clue what i talking about, please post aome more info. thx