With SDKs available right now, old Amazon app features are meeting a fantasy future.

At a Wednesday press conference in Seattle, Amazon announced a service that would go along with its newly debuted “Fire Phone.” Called Firefly, this new technology is packaged in an app that can identify up to 100 million objects. For the most part, this feature will integrate with the Amazon marketplace, allowing you to take photos of products and buy them from Amazon, but the technology used to make it run will also be available to developers in an SDK available now.

Amazon also introduced "Dynamic Perspective," which uses the tilt of the phone to change the image you're seeing on the screen. The Dynamic Perspective SDK is also available today.

Using Firefly, a button on the side of the Fire Phone will instruct the camera to recognize a phone number, a book, a DVD, a URL, a QR code, and more. Additionally, Firefly will be able to listen for music (like Shazam) and identify a song that's playing in the ambient noise around you. Amazon said that iHeartRadio, a popular app developed by Clear Channel, is already integrated with this function and will let you build a playlist based on an artist you hear and like.

Firefly can also identify TV shows down to the episode you're watching, as well as art pieces (identifying art based on an image was something this writer desperately desired while failing an art history class in high school).

Firefly strips down images and just sends the most important information to augment "machine vision."

"Machine vision problems are a tough nut to crack,” Amazon CEO Jeff Bezos told the crowd. “You have glare, wrinkles, curves, and other problems.” He noted that Amazon uses “semantic boosting” to keep Firefly's identification response time fast and accurate. For example, he said, if a picture is taken of a phone number with glare, Firefly can take what it knows about phone numbers in the area and use that to “improve the probability of character recognition.”

Additionally, when the camera takes a picture of an image, it will strip the image down to just “the parts that matter” before sending it back to Amazon's servers to respond. Other streamlining technologies that help Firefly's machine vision include underline removal, two-phase MSER scanning, orientation normalization, multiframe combination, multiple OCR engines, page-layout analysis, and truncation filtering, according to a slide Amazon presented.

With Dynamic Perspective, Amazon uses machine vision to assess the world around the phone and change what the phone user sees on her screen. Using natural movement, like tilting the phone side to side, the screen will resolve adjusted images of maps to give you a 3D view of landmarks or auto-scroll a piece of text as you tilt the phone downward. Dynamic Perspective can also be integrated into games, so as you tilt the phone you can view the level around you.

This creepy photo came about as Amazon taught its Dynamic Perspective technology how to identify human faces.

To create this effect, Amazon put four, ultra low-power, face-tracking cameras on the front of its Fire Phone, with wider-than-normal, 120-degree fields of view and an infrared sensor in each to "see" in darkness. When Dynamic Perspective is enabled, the phone will use the two cameras with the best view of the phone user to accurately adjust the perspective of the images displayed on the screen.

Amazon also said that it worked for years to "teach" its Dynamic Perspective technology to identify all kinds of faces in all kinds of lighting situations. "We collected millions of images with the actual camera hardware from all over the world," Bezos told the audience. "I am compressing years of work by some of the world's best computer scientists into two minutes. I apologize to that team."

Bezos also added that Amazon had "found heuristics and workarounds" for many of the problems involved in face-tracking, like distinguishing the user's face in a crowd or rejecting images of faces that are in photos near the user.

Apparently, you find it difficult to make a profit. Here's a suggestion: stop pumping vast amounts of money into vain attempts to create an Apple-style walled-garden, and instead do the three things that a huge number of your customers are getting increasingly irate about:

* Get over yourself, and release your apps for Android devices. I and many others will never buy into an Amazon hardware or software ecosystem. (edit: and I am a huge Amazon customer and Prime member)

* Instead of these other distractions, try getting product discovery and search to work properly on your site. I find myself increasingly going to Newegg for electronics, simply because I'm tired of digging through 50,000 tenuously relevant and uselessly sorted products no matter what I put into the search bar. And the next time you demand that I pick a department before I can sort, when I don't know which department has the most results in it, I think I will scream.

* Stop trying to turn Amazon Prime into a walled garden. I buy Amazon Prime to get fast delivery, period. If you keep trying to force me to pay more and swallow products I don't want or can't use (example: Instant Video, which you won't let me view on my devices), I will cancel Prime. If you do indeed raise the price on me to force me to pay for these other services, it will happen.

Brick and mortar stores will need to set up signal jammers to prevent becoming even more of a showroom than then already are. I can definitely see pushback against this.

Cell and radio jammers are so illegal (at least in the USA) that even prisons aren't allowed to install them. Any store manager caught setting one up is going to be facing serious federal felony charges.

Brick and mortar stores will need to set up signal jammers to prevent becoming even more of a showroom than then already are. I can definitely see pushback against this.

Cell and radio jammers are so illegal (at least in the USA) that even prisons aren't allowed to install them. Any store manager caught setting one up is going to be facing serious federal felony charges.

This is true of active jammers, but passive jammers such as Faraday cages are perfectly legal. (I've been wondering for years why more movie theaters don't have Faraday cages built into the auditoriums, actually.)

Apparently, you find it difficult to make a profit. Here's a suggestion: stop pumping vast amounts of money into vain attempts to create an Apple-style walled-garden, and instead do the three things that a huge number of your customers are getting increasingly irate about:

* Get over yourself, and release your apps for Android devices. I and many others will never buy into an Amazon hardware or software ecosystem.

This is true of active jammers, but passive jammers such as Faraday cages are perfectly legal. (I've been wondering for years why more movie theaters don't have Faraday cages built into the auditoriums, actually.)

This is true of active jammers, but passive jammers such as Faraday cages are perfectly legal. (I've been wondering for years why more movie theaters don't have Faraday cages built into the auditoriums, actually.)

Emergency phone calls, perhaps?

More likely the cost of materials for a Faraday cage that seats 500 people.

Additionally, when the camera takes a picture of an image, it will strip the image down to just “the parts that matter” before sending it back to Amazon's servers to respond. Other streamlining technologies that help Firefly's machine vision include underline removal, two-phase MSER scanning, orientation normalization, multiframe combination, multiple OCR engines, page-layout analysis, and truncation filtering, according to a slide Amazon presented.

The interesting thing here is between Amazon's horsepower, and Google's breadth of data, smartphones are going to be doing a whole lot more in the future.

That's cool, a huge improvement on what Google has been doing over the years. I've always wondered why Google didn't just put all that shit into a single interface. Looks like they may be pushed into doing so by the new competition.

This is true of active jammers, but passive jammers such as Faraday cages are perfectly legal. (I've been wondering for years why more movie theaters don't have Faraday cages built into the auditoriums, actually.)

Emergency phone calls, perhaps?

More likely the cost of materials for a Faraday cage that seats 500 people.

Yeah, this. Nicely conductive metals aren't cheap. It also turns out that modern cell phones have incredibly sensitive radios. You would have to be quite meticulous in building such a thing. To give you a sense of how hard that is, the US DOD operates some facilities that handle large volumes on classified information, SCIFs, that are supposed to include faraday cages. It's always embarrassing when someone's cellphone (which they're not supposed to have in there to begin with) rings while they're inside. The point is that failures happen, in a setting where getting the construction right is a lot more critical and worth a lot more than a movie theatre.

Apparently, you find it difficult to make a profit. Here's a suggestion: stop pumping vast amounts of money into vain attempts to create an Apple-style walled-garden, and instead do the three things that a huge number of your customers are getting increasingly irate about:

* Get over yourself, and release your apps for Android devices. I and many others will never buy into an Amazon hardware or software ecosystem.

Yes, you are missing something. That video streaming app only works with Google TV and is "incompatible" with my the Android devices. Amazon has intentionally and deliberately crippled their offerings on Android.

It is completely insane.

I would have bought all my everything in terms of media from Amazon if they had simply let me. It never would have even occurred to me to use Google's offerings except for the fact that Amazon drove me into Google's arms.

Amazon's entire Android strategy is nuts. They could have stolen the show a dozen times over by offering proper Android devices with custom skins and their media market baked in while keeping it all compatible with the rest of Android. Whatever paltry sales they missed by directly competing with the Android store would have been nothing to all the sales they would have made when people like me simply pick Amazon over Google even when given the option. Hell, I might even have gotten a Kindle tablet if it had been a normal Android tablet. The entire strategy is just nuts when you realize that all they get is a few extra app sales that have to be pocket change when compared to all the other media they are busy not selling.

A pretty misleading chart, since the bar on the right only shows 50% of the tablets shown on the left. And even more misleading because of the basic misrepresentation that Kindle Fire is an Android device. It is not. You can argue all you want about sideloading blah blah, but Android is a legal trademark and a set of standards that the Kindle Fire does not follow and cannot use.

I'm not sure what your point is. Are you trying to say that 50% of all the "Android-derived" tablets in the world (which is what the share of actual Android devices in that graph add up to) does not constitute "many"????? I never said most. And even if it was most, the minority still exists, and is very large.

If that chart is true, and we assume that most people in the United States are Amazon customers, then tons of Amazon's customers don't have Amazon devices. Amazon is being stupid by repeatedly shafting those people while lying that they want their content to be available on every device (except, er, the dominant mobile operating system).

edit: Oh, and I just noticed the killer: that chart doesn't include smartphones. The number of tablets is insignificant next to smartphones. If you include them, the market share of the "android-derived" devices that Amazon has is completely insignificant.

My point is that a lot of people are already invested in Amazon to some degree or another. Because you aren't, doesn't mean a vast majority of people aren't. I live in New York city and on every subway ride I see at least 3-4 people with kindles on every ride, not including myself.

If Amazon is smart, they'll release a version of Firefly for Android. I already buy a ton of stuff through Amazon Prime, but I'm not going to buy a crippled FireOS phone. That said, using Firefly on my Note 3 might compel me to buy a ton-and-a-half of stuff from Amazon Prime.

Amazon also said that it worked for years to "teach" its Dynamic Perspective technology to identify all kinds of faces in all kinds of lighting situations. "We collected millions of images with the actual camera hardware from all over the world," Bezos told the audience.

So I'm actually a bit surprised at how interested I am in this phone. Firefly looks like it could actually pretty exciting, but more info is needed.

How much control are developers going to have over Firefly's computer vision functions? I'm assuming the SDK isn't going to be an alternative to something like OpenCV4Android, but are we potentially going to see apps that go beyond recognizing barcodes and doing fun things with those barcodes?

Four front-facing infrared cameras? Yes please. Tell me that developers are going to be able to use those for more things than just Dynamic Perspective. Head tracking (done well) is superbly cool, but there is so much potential for other, equally as cool things using computer vision with those cameras.

Brick and mortar stores will need to set up signal jammers to prevent becoming even more of a showroom than then already are. I can definitely see pushback against this.

Cell and radio jammers are so illegal (at least in the USA) that even prisons aren't allowed to install them. Any store manager caught setting one up is going to be facing serious federal felony charges.

This is true of active jammers, but passive jammers such as Faraday cages are perfectly legal. (I've been wondering for years why more movie theaters don't have Faraday cages built into the auditoriums, actually.)

It possibly wouldn't always work that well due to constraints; then there's legacy - most theaters probably weren't constructed with phones in mind simply due to age. It's possible you'd annoy your customers too; while people dislike the (very rare) mid-movie ring, they might actually stay away if unreachable for emergencies (anyone with kids, say, and some jobs). Since you can't actually close the Faraday cage, you might not even catch all the calls, and if people get in the habit of not bothing to turn phones down, this is more likely to be an issue than now. Then there's smartphone apps that beep for other reasons (like calendars), which you wouldn't prevent. You might annoy your customers by draining the battery very quickly (as constant searching for wifi+cell is wont to do). And since current phones sometimes only last a day, and movies are seen at the end of the day, you're more likely than usually to actually empty the battery into the (beeping) warning zone, quite contrary to the silence you're trying to achieve. You're also limiting your own ability to use any kind of radio transmission in the building (like plain old wifi), and that's a real cost because wiring everything is much more expensive - and likely impossible, because you can't predict 10 years down the line what devices you might want to install where.

All in all, why take all those risks and costs just to prevent the fairly rare mid-movie ring?