Apple working on iPhone video feature that automatically switches cameras during a FaceTime calls

A newly published patent application from Apple outlines teleconferencing and video capture technology that would process video from an iPhone's front and rear cameras simultaneously, and automatically determine which feed to stream based on voice or visual cues.

Source: USPTO

Apple's "Automatic video stream selection" patent application, published by the U.S. Patent and Trademark Office on Thursday, describes a system that can intelligently determine whether to use the video feed from a smartphone's front or rear facing camera. The tech can be used in a video calls, such as those made via Apple's FaceTime, or for content stored locally on a device.

As noted in the document, many handhelds have two cameras for video and picture taking. The shooters are typically facing in opposite directions, with the rear-facing module used for higher-resolution shots, while the front-facing unit handles video call duties.

With advancements in mobile processing power, some smartphones are able to capture video streams from both cameras simultaneously, though the use cases for such a feature have been limited. The iPhone currently uses only one camera at a time, offering users the option to toggle between the two as desired.

Apple notes that current bandwidth restrictions would make it difficult to stream both video feeds at the same time, thus a method is needed to intuitively select one while keeping audio in sync.

What Apple proposes is a system that captures and monitors two video streams for voice or visual cues, then outputs one of the two depending on a set of predetermined criteria. One example would be a FaceTime call in which a rear-facing camera video stream is being used before the feed automatically switches to the front-facing camera when a person begins to talk.

In order to decide which camera to use, the technology detects speech by using data from microphones facing in the same direction as each camera, or facial expressions such as lip movements.

One embodiment describes an "Auto-Select" indicator that, when selected, will activate the system and begin outputting the multiplexed video.

Various implementations are described in the paper, including "report mode," which captures a a scene from the rear-facing camera and switches to the user when they start talking. When no speech or lip movement is detected, the video switches back to the scene. Both video streams can be interleaved throughout the process and output to local storage.

In "interview mode," the phone is located between a user, who is being captured by the front-facing camera, and another person facing the rear camera. The video switches between the two when one person is determined to be speaking.

Interview mode.

Finally, along with streaming the multiplexed video in a FaceTime call, the invention allows devices without cellular access to store the content, to be uploaded whenever a Wi-Fi connection is available.

While the invention relies heavily on software, there are hardware limitations, such as audio and video processors, that may restrict Apple from releasing such a feature in the near future. However, as the final specifications of the upcoming "iPhone 5S" have yet to be revealed, it is possible that the smartphone could have the necessary components baked in. As of now, iOS 7 beta does not carry such functionality.

Apple's automatic stream selection patent application was first filed for in May and credits Jae Han Lee and E-Cheng Chang as its inventors.

To summarize what this could do, iOS could listen to your audio stream and use voice or lip movements to figure out what the speaker is talking about and automatically switch between front and back cameras.

For example : Say the front camera's active at the moment. When the user says, "Hey Jim take a look at my new computer i got", iOS could auto switch to the rear camera without ever having to manually do it. Pretty Powerful. Can you guys think of other implementations ?

Shouldn't Apple be fixing things instead of mucking around with these fancy ideas that most likely won't see the light of day? The virtual keyboard is a joke. ICloud back ups that don't work. Home buttons that freeze up.
I would prefer perfection or close to in their existing products that are used extensively.

The virtual keyboard is deemed awesome by the rest of the world. Don't understand why you think its a joke (cause of ur if / of issues :P ).

iCloud backup works just fine if you haven't exceed ur backup quota on the cloud.

Home button that freezes up : Where did you get ur iPhone from, a gray market source ?

The few times I've used the rear camera during Facetime calls, I was speaking while recording, showing something off. Would the user have to be silent so it doesn't switch back to the front camera? Also, seeing you can switch back and forth with a simple tap of an icon, I don't see the real need for this. However, the "interviewer" mode seemed like an interesting use case.

The virtual keyboard is deemed awesome by the rest of the world. Don't understand why you think its a joke (cause of ur if / of issues :P ).
iCloud backup works just fine if you haven't exceed ur backup quota on the cloud.
Home button that freezes up : Where did you get ur iPhone from, a gray market source ?

You are so wrong on all counts.
The keyboard does not compare with android or the new Blackberry, this is fact.
Did not exceed limit, I think I know how to count, but it refused to back up on iCloud.
Updated to iOS 6 on a iPhone 4S, and it bricked the wi-fi, Apple admitted it was their fault, and a common one.
Returned two iPhone 5s where the home button would freeze up, again many complaints about this on the Internet.
No they are not purchased in the grey market, purchased through a Telco in Australia, I don't reside in China.
Happy now?
You are a funny guy, if of, the software is crap that auto corrects when it does NOT need to.
Plus starting to get lag in typing.
I only have Apple products at home, but I am getting annoyed that these common problems are not being fixed.