June 02, 2011

Apple working on a Sophisticated Infrared System for iOS Cameras

On June 2, 2011, the US Patent & Trademark Office published a patent application from Apple that revealed various concepts behind a newly advanced next generation camera system that could employ infrared technology. On one side, the new system would go a long way in assisting the music and movie industries by automatically disabling camera functions when trying to photograph or film a movie or concert. On the other hand, the new system could turn your iOS device into a kind of automated tour guide for museums or cityscapes as well as eventually being an auto retail clerk providing customers with price, availability and product information. The technology behind Apple's patent application holds a lot of potential.

The Problem to Solve

Many electronic devices include cameras designed to detect images. For example, a traditional cellular telephone or portable media player may include a camera. Such cameras can typically detect images based on visible light but do not receive any data communications through either visible or invisible light. Accordingly, the functionality of cameras in traditional electronic devices is limited.

Apple's Solution

Apple's patent describes systems and methods for receiving infrared data with a camera designed to detect images based on visible light. A system could include a camera and image processing circuitry electrically coupled to the camera. The image processing circuitry could determine whether each image detected by the camera includes an infrared signal with encoded data. If the image processing circuitry determines that an image includes an infrared signal with encoded data, the circuitry may route at least a portion of the image (e.g., the infrared signal) to circuitry operative to decode the encoded data. If the image processing circuitry determines that an image does not include an infrared signal with encoded data, the circuitry may route the image to a display or storage. Images routed to the display or storage could then be used as individual pictures or frames in a video because those images don't include any effects of infrared light communications.

Based on the decoded data, a device could display information to a user or modify an operation of the device. In some embodiments, a device could, based on received infrared data, display information to a user relating to an object near the user. For example, an infrared emitter could be located near an object and generate infrared signals with encoded data that includes information about that object. The electronic device could then receive the infrared signals, decode the data and display the information about the object to the user.

In some embodiments, a device could, based on received infrared data, disable a function of the device. For example, an infrared emitter could be located in areas where picture or video capture is prohibited, and the emitter could generate infrared signals with encoded data that includes commands to disable the recording functions of devices. An electronic device could then receive the infrared signals, decode the data and temporarily disable the device's recording function based on the command.

Future iOS Devices will incorporate an Infrared Data System

Apple's patent FIG. 2 is a schematic view of a system for communicating infrared data which could include a transmitter (290) and electronic device (210) which could be an iPhone (or iOS device with a camera). The transmitter could generate an infrared signal (299) with encoded data and the iOS device could detect one or more images that include an infrared signal. The iOS device could then decode the data in the infrared signal and provide information to a user and/or modify its operation based on the decoded data.

How it basically Works

An Infrared emitter (297) could include any component that could transmit infrared signals based on a control signal. For example, the infrared emitter could include an infrared light-emitting diode (LED). In some embodiments, the infrared emitter may emit a strobe of infrared light that cameras in the same general area of the transmitter could detect, regardless of the direction the cameras are facing. For example, the transmitter could function as a beacon generating an infrared signal that is easy for cameras to detect. In other embodiments, the infrared emitter may emit a directed beam of infrared light that only cameras in the path of the beam could detect. For example, the transmitter could function as a "spot light" generating an infrared signal that could only be received by cameras generally in front of the transmitter.

So why include this feature into Cameras?

As an example of why this would be incorporated into cameras is explained as follows. A transmitter could be located adjacent to a museum exhibit and its infrared signal could include encoded data that represents information about the exhibit. In some embodiments, the infrared signal could include encoded data that represents a command. For example, the transmitter could be located in an area where photography is prohibited and the infrared signal could include encoded data that represents a command to disable recording functions.

This example could easily apply to movie theatres trying to stop customers from filming a movie for illegal distribution or any kind of music concert to protect an artist's image from being photographed or videoed illegally, as shown below.

As shown in Apple's patent FIG. 5, transmitters (590) could be located adjacent to a stage (580) and when a camera is pointed at the stage to take a photograph or video, the infrared transmitter near the stage would communicate with your camera and instruct it to disable capturing capabilities.

Applying Watermarks Automatically to Images

In some embodiments, a device may apply a watermark to detected images as an alternative to completely disabling a recording function. For example, a device may receive infrared signals with encoded data that includes a command to apply a watermark to detected images. In such an example, the device may then apply the watermark to all detected images that are displayed or stored (e.g., single pictures or frames of a video). This feature could be convenient, depending on how this is applied.

For instance, let's say you're on vacation and visiting an aquarium, zoo or museum. That particular establishment could adjust their infrared system so as to automatically generate a watermark on the photos you take so that it identifies that particular establishment, such as: The Vancouver Zoo or The Monterey Aquarium or The New York Metropolitan Museum of Art. Apple could conveniently build that kind of feature into iOS cameras for an update to a feature like "Places."

Used as an Auto Tour Guide

As shown in Apple's patent FIG. 4 below, the transmitter could be located adjacent to an object. For example, the object 480 is an Aztec water jug that could be on exhibit at a museum and the transmitter could be located adjacent to the object. In some embodiments, the transmitter could include visible indicia that also convey information about an object. For example, the transmitter could be in the form of a plaque with writing that conveys information about an object.

In some embodiments, Apple's iOS device display could provide options for a user to obtain additional information or content about an object, such as the Aztec water jug as noted in the patent graphic. For example, the iOS device could include an audio option (shown as option 423) that a user could select to request a prerecorded audio segment and video option that a user could select to request a prerecorded video segment.

Apple makes it clear that their infrared system could obviously work in differing environments beyond tourist attractions. It could be applied to a retail environment so as to inform users of retail information on any given item in the store such as price, availability, discount status or any other relevant data that the retailer thinks will assist their customers in making a decision to make a purchase.

User Controls for the Infrared System

Apple's patent FIG. 6 is a perspective view of an illustrative screen for configuring an iOS device to receive infrared data.

A configuration screen could include options corresponding to one or more functions performed based on received infrared data. In some embodiments, provided options may include corresponding alerts when receiving infrared data. For example, a user may set option 623 to "OFF" so that the device will not provide any alerts when receiving infrared data. When set to "ON," the iOS device could provide an audio alert (e.g., a chime), a visual alert (e.g., an icon), a tactile alert (e.g., a vibration), or any combination thereof in response to receiving infrared data.

It is understood that, in embodiments where an infrared data includes commands to temporarily disable a device function, a user may not be able to set configuration options that override the disable commands.

Today's published patent application from Apple was originally filed in Q4 2009 by inventors Victor Tiscareno, Kevin Johnson and Cindy Lawrence. Update # 2, June 22, 2011: For those wishing to review Apple' full patent application, see Apple patent 20110128384.

Update # 1, June 3, 2011 4PM MST: one last factiod about this patent for the X-File crowd. In Apple's patent point # 48, the patent specifically adds "classified facilities" as being another application for this technology: "In some embodiments, a transmitter can be located in areas where capturing pictures and videos is prohibited (e.g., a concert or a classified facility) and the transmitters can generate infrared signals with encoded data that includes commands temporarily disabling recording functions." Hmm, I guess that means that area 51 will be better protected in the future. And no, you didn't see any little green men, right? Kidding aside, you could be rest assured that the request for such a feature must have come directly from the US government at some point in time.

Other Noteworthy Patent Applications Published Today

A 2008 continuation patent (20110131495) titled "Graphical Representation of Assets Stored on a Portable Media Device" surfaced today which also relates to a provisional patent dating back to 2007. The patent application relates to the integration of progress based GUI's displaying various information on your about playlist progress and/or breakdown of your playlist by genre. Although I've seen a progress bar displaying how much space was left on my iPod, I've never seen charts as those presented below. If you have, then send in a comment describing which iPod model it was on.

Another continuation patent application that surfaced today (2010130140) covers their 2006 patent application and one that Apple was granted a patent for in January 2011. In that report we stated the following: "It's an interesting read but you have to put it into proper context. Firstly, Apple isn't seen in this patent as the "MVNO" as some would hope for. Secondly, the "bidding" system goes out to a series of MVNO's which would confirm that Apple wasn't planning on being the MVNO. Thirdly, the patent was filed in 2006 - well before any iPhone announcement was made. So this patent may simply have been Apple's backup plan if their original talks with Cingular would have failed." Considering that it's already a granted patent, you have to wonder why they're back tinkering with the idea.

And lastly, Apple's patent application 20110130958 titled dynamic alerts for calendar events is, according to Apple's application, related to two older patents dating back to 2008 and 2009. Unless an older or continuation patent is worthy of reporting on, Patently Apple simply adds it to its current report as a notable mention as we've done here.

Notice:Patently Apple presents only a brief summary of patents with associated graphic(s) for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for further details. Patent applications shouldn't be digested as rumors or fast-tracked according to rumor time tables. Apple's patent applications represent true research that could lead to future products and should be understood in that light. About Comments:Patently Apple reserves the right to post, dismiss or edit comments.

Here are Some Great Community Sites that are covering our Original Report

Note:The sites that are linked to above offer Apple community members with an avenue to make comments about this report in many original languages. Additionally, many of these sites provide our guests with different takes on any given patent or concept that is presented in our reports to make it more fun, interesting and/or personal. If you have the time, join in!

Comments

Let me clarify one last point:

The fact is that this technology works on the basis of reading “encoded data.” Meaning that Apple could work with the music industry to encode specific tour codes that restrict filming. The encoding may be applied to only a few of the top tours and no more. Extrapolating that to not being able to film riots or police/army abuses is a stretch beyond reason. Apple’s iPhone would require a code that they approved and could be worked into an iOS update (theoretically). That’s not going to happen for anything outside of what Apple and specific venues agree to. It’s not a random ability to shut off the camera by any other parties.

The spin about the "cost" at the end of this report is playing into the hysteria that is being drummed up about the technology described in Apple's patent.

While I fully understand and appreciate legitimate feedback about cutting back on filming concerts, which Apple and the industry could tweak, the hysteria about repressive regimes and police activity in North America and around the world abusing this is just that, hysteria. Hypotheticals like this are never ending loops. You could put a negative spin to anything. It doesn't mean it'll come to pass.

I made my view point known earlier, before the post report was ever posted.

Yes there is a positive side to Apple's patent and only the concert angle is a little negative. What came up much later about police state activities from someone's imagination is what I see as a total over reaction to a simple "patent application."

If Apple would integrate an infrared camera into an iPhone, it would be possible to use the camera to detect heat gain / heat loss in homes and businesses. Owners would be able to see where insulation was missing or inadequate in their homes. One could see if a distribution panel or a light switch was overloaded. This would be of great benefit.

If the Apple IR camera patent does come to fruition, perhaps venues would at the very least have a tiered pay system to photograph. However, the negatives to this technology clearly outweighs the positives. What I worry about are IR receivers overstepping the bounds to where we are legally allowed to photograph and record.

My point is that the patent covers many angles and that the camera shut-off is just one. I doubt that Apple dreamed up that latter feature. I believe it was an industry-request. Other music-phone players may also be forced into following suite if they want to have phones with music. It's the music industry taht will hold the key on this feature, not Apple. If you notice in the patent, the emiter is on stage with the lighting system. That's not an Apple initiative, plain and simple.

@Fred. Every time there's a new technology, we find that it has a good side that could be exploited for evil doing, such as your Rodney King example. I'm sure that Apple and responsible industry leaders (HP, Msft, Dell, Google) are working on ideas to protect content and artists right to make money and protect their image etc. That's business. I don't expect younger people to get that, but reasonable adults will.

Apple is also emphasizing other aspects of this technology for guided tours and for use in retail environments. These are positives that are being overlooked because "teens" are getting all upset at not being able to record concerts. The reaction just proves that the industry is right about illegal recording being a problem. Somewhere along the lines however, there could be compromise. Perhaps the emitter will allow for a single song or 10 minutes of filming instead of it all. It's a patent application that hasn't been approved yet, so there's time for tweeking the technology.

Apple watches the reaction the community has to these patent applications so that the final technology could be accepted by all parties concerned.

Scary.
So basically reality now comes with a watermark, or even an "invisibility cloak" as far as digital recording goes.
No more Rodney King incidents then. Or at least, such incidents won't get recorded in future, as the police wil carry "no record" beacons as a matter of routine.

This wouldn't get backed by any company unless it were a standard across all smartphone platforms, since they just patented it, it would obviously be an iphone only thing, therefor no company would back it. Why would any company take the time to block only one specific device instead of them all?