Apple wants Siri to be able to search your iPhone photos

Starting with iOS 7, Apple began automatically sorting users' iPhone photos based on time taken and location. But a new patent application reveals the company is interested in taking it one step further, and allowing Siri to sort through iOS photos based on voice search and tagging.

The company's interests were revealed in a new patent application published by the U.S. Patent and Trademark Office on Thursday, discovered by AppleInsider. The filing, entitled "Voice-Based Image Tagging and Searching," describes associating "natural language" text strings with photographs saved on a device, like an iPhone.

Such text strings would be associated with speech input, much like users can access data on their iPhone by speaking in plain conversation to Siri, the device's voice-driven personal assistant. Text strings associated with photos could cover an entity, an activity, or a location, according to the filing.

Apple's application notes that a growing volume of photos collected by users on devices like iPhones makes them increasingly hard to sort through. The company notes that tagging photos based on names of people or places makes it easier to find what users are looking for.Apple's system would allow users to use their voice to tag and search photos based on locations, people's names and more.

Apple's system would also allow users to tag photos with their voice. In one example provided by Apple, a user tells their device, "This is me at the beach," and the corresponding picture is tagged accordingly.

The proposed invention could even automatically tag corresponding photos, based on the time and location at which they were snapped, to make it easier for users to sort their pictures and not require them to individually tag each picture.

Apple's system could even recognize faces, buildings or landscapes to tag similar photos. For example, by a user telling Siri that they are captured in a photograph, the system could then intelligently tag other photos that capture the user's face.

With photos properly tagged, users could then use their voice in a similar manner to search for the pictures they are looking for. In another example, a user asks their device, "Show me photos of me at the beach," and related items are delivered.

Apple began automatically sorting pictures with a revamped Photos application that debuted in iOS 7 this year. Starting at a macro level, photos are presented in tiny thumbnails based on the year they were captured, and users can zoom in to find their photos sorted based on date and location.

In this manner of auto-tagging alone, Apple could utilize its proposed invention to allow Siri to sort through pictures. For example, saying "Show me photos from Hawaii taken in 2012" could present users with relevant photos, even without the need for tagging faces.

The application, first made public this week, was filed by Apple with the USPTO in March of this year. It's credited to Jan Erik Solem and Thijs Willem Stalenhoef.

But seriously, we are very close to having true artificial intelligence on our phones. Only about 5-6 years off according to Ray Kurzweil.

I like it Spam! :) I try to use Siri for all my searches. Not perfect, but very impressive. I'm really pleased with what Apple is doing with this tech and mobile tech in general. Can't wait to see what 2014 brings.

I'm one of those that think the iPad Mini, Air, and 5s are extraordinary devices along with the MacPro. Making the OSX and iWork free is just icing on the cake.

I like what Apple has done with iWork, making it more usable across all my devices including that once a year time I have to use someone else's Windows PC to do something on iCloud. Just brilliant.

Anyway, Best! :)

Edit: Didn't mean to forget the iMac, MBP and MBA lines. All which are best in class devices! :)

But seriously, we are very close to having true artificial intelligence on our phones. Only about 5-6 years off according to Ray Kurzweil.

What evidence do you have of this? Siri can hardly have a one diminutional instruction sent to her, let alone AI. Siri was able to map directions, then understand when I said I wanted to stop for coffee on my way, which is very surprising she did. But that is the exception, not the rule.

"Do I have availability January first" Siri: "Yes, you have a couple of appointments for Wednesday" {shows calendar} Me: "What's my best option to schedule a meeting" Siri: "I didn't find any appointments about 'best option to schedule a meeting' between today and March 26, 2014". For the life of me, no matter what combination I use, I could not get Siri to tell me when the best time fold me to schedule a meeting, when my last meeting was, when I was available.

"Can I move any of my meetings on January 1 {I said first} to another day" Siri: "I didn't find any meetings about 'Another day' for Wednesday". Me: "Can I move any of my appointments this coming Wednesday {Siri likes Wednesday not January 1st} to another day" Siri: " I didn't find any meetings about 'Another day' for Wednesday" The trick was, one appointment was the Jan 1st holiday and the other is a birthday. So no, neither can be moved.

"Schedule a lunch meeting with Jimmy Smith at 12 on the 31st" Siri hears " Schedule a lunch meeting with Jimmy Smith at" Siri: "Ok, when" Me: "on the 31st" Siri: "what time is your meeting" Me: "noon" Siri sets up the meeting Me: "did you send him an email" Siri: "You have no email bout 'you send him an'". No matter how I edited the question, Siri could not tell me if she sent him an email confirmation.

{added} Siri can't yet perform localized tasks that should not need her to phone home first.

As you see, Siri is really one diminutional in her conversation. Getting Siri to actually understand the words that are coming out of my mouth is very frustrating at times. Unless Apple is hiding some huge update, I don't see AI coming into its own for 10+ years. Siri, for me, is only good at very basic tasks such as reminders, getting directions, web searches, and those are not even 100%.

Not really a Siri problem. But when editing an event in calendars, I wish that when I put 'Ann's house' as the location, iOS links my contacts' addresses first. Tried it today and it linked it with a place called 'Ann's house' 300 miles away...

I'll look over his stuff, thanks for the links. I see he has a 2005 book, and a 2009 video on Google's top SERPs. How near was he talking? He says here, real AI will be 2029 and Singularity by 2045.

No, this is not just a complain, as I gave real world scenarios and results. It was to show, that although Siri was introduced two iPhone versions ago, it still is not intelligent.

"Since Kurzweil believes computational capacity will continue to grow exponentially long after Moore's Law ends, it will eventually rival the raw computing power of the human brain. Kurzweil looks at several different estimates of how much computational capacity is in the brain, and settles on 1016 calculations per second and 1013 bits of memory. He writes that $1,000 will buy computer power equal to a single brain "by around 2020""

"Since Kurzweil believes computational capacity will continue to grow exponentially long after Moore's Law ends, it will eventually rival the raw computing power of the human brain. Kurzweil looks at several different estimates of how much computational capacity is in the brain, and settles on 1016 calculations per second and 1013 bits of memory. He writes that $1,000 will buy computer power equal to a single brain "by around 2020""

He also said software is the gating issue. So by 2020 he predicts the computational power will be there and by 2029 the software should be capable and by 2045 we'll have his Singularity. 6 years to hardware, 15 years to software and 31 years to Singularity.

​My point was not to doubt his theory, but that we need an exponential jump from where Siri is now to any resemblance of true AI. I hope Apple is investing in this heavily and pioneers this new age. I would love to see working models and not simply projections. We all know Future prediction never hit the mark.

He also said software is the gating issue. So by 2020 he predicts the computational power will be there and by 2029 the software should be capable and by 2045 we'll have his Singularity. 6 years to hardware, 15 years to software and 31 years to Singularity.

​My point was not to doubt his theory, but that we need an exponential jump from where Siri is now to any resemblance of true AI. I hope Apple is investing in this heavily and pioneers this new age. I would love to see working models and not simply projections. We all know Future prediction never hit the mark.

Your point is taken, but both Moore's Law and Kurzweil's "evolving" timeline have been fairly good indicators. I can almost guarantee in 5-6 years that the Siri of today will be a quaint memory (although it will probably still be called Siri).

Your point is taken, but both Moore's Law and Kurzweil's "evolving" timeline have been fairly good indicators. I can almost guarantee in 5-6 years that the Siri of today will be a quaint memory (although it will probably still be called Siri).

yes, yes, don't get me wrong, as I agree Apple will have made great gains in Siri, just using their current increments, I'm not encouraged to real AI.

The amount of work involved in conceiving 'did you send a conformation email' is 1-2 years away I would imagine, but something like "Siri, pull together research on {topic} and report back with what is trending" seems very far off.

All I am saying is I can't get humans to follow simple instructions most of the time, how is a computer going to :)

yes, yes, don't get me wrong, as I agree Apple will have made great gains in Siri, just using their current increments, I'm not encouraged to real AI.

The amount of work involved in conceiving 'did you send a conformation email' is 1-2 years away I would imagine, but something like "Siri, pull together research on {topic} and report back with what is trending" seems very far off.

All I am saying is I can't get humans to follow simple instructions most of the time, how is a computer going to :)

In my opinion, the leaders in this area will be IBM and Google, especially IBM with their planned "expert" systems. It's entirely possible that Apple will demonstrate some blockbuster new features that better tie together their services, but IBM has a demonstrated leadership position that will be hard for anyone else to beat for the time being.

Your point is taken, but both Moore's Law and Kurzweil's "evolving" timeline have been fairly good indicators. I can almost guarantee in 5-6 years that the Siri of today will be a quaint memory (although it will probably still be called Siri).

What I find fascinating is that AI will need much much more than rules written for it in software, but the ability to create its own understanding. Just in simple expression of 'Dud, that's wicked' needs such a high level of understanding an extrapolation it is almost ming-blowing. Humans picks up on voice inflections, facial expressions and body language as well as determining meaning from context.

You take a beautiful woman and dress her up in some really hot sexy outfit and 'Dud, that's wicked' can be taken two ways from two different perspectives. Literally, and sensationally (perhaps colloquialism would be better).

There are so many veritable that you can't program them all, but the system would have to take concepts such as voice, facial, body, context, persons, places, and things, as well as what was said, and determine the meaning from that.

That is incredible. Just thinking on the complexity make me giddy. lol

In my opinion, the leaders in this area will be IBM and Google, especially IBM with their planned "expert" systems. It's entirely possible that Apple will demonstrate some blockbuster new features that better tie together their services, but IBM has a demonstrated leadership position that will be hard for anyone else to beat for the time being.

I only take into account Apple's money and people wanting to work for them, thus perhaps giving them any edge is recruitment.

Of course, if you whisper in Apple's ear that Big Blue is beating them to it, I'm sure they will 10 fold their efforts :)