Sony invented augmented reality 15 years ago – and had no idea what to do with it

A logo of Sony Corp is pictured at an electronic store in Tokyo May 14, 2014. Sony Corp said on Wednesday it expects to log a 50 billion yen ($489 million) net loss this financial year, its sixth in seven years, and plans additional restructuring measures that include its loss-making PC operations. REUTERS/Toru Hanai (JAPAN - Tags: BUSINESS LOGO) - RTR3P19R
(REUTERS/Toru Hanai)

Most major technology companies have major R&D arms for “blue-sky” research: deep theoretical science and wild new creations. Microsoft Research – 1,100 strong in labs all around the globe – built a mood-sensing “smart bra” and a holographic desk and is working on Windows 9. Amazon’s R&D arm is called Lab126. They recently came up with the Fire Phone, and a wacky bar-code scanner called the Dash. And reporters have written tens of thousands of articles about Google X, the company’s blue-sky, moon-shot research arm that’s dabbling in anything and everything from self-driving cars to robot moon landers to a cure for death itself.

But Sony? I didn’t even know Sony had a research arm until last week.

It’s a much smaller operation, with approximately 30 researchers and a very different goal, explained Hiroai Kitano, Director of Research, President, and CEO of Sony Computer Science Laboratories (CSL).

“We want to go for big ideas, we want to contribute to society,” Kitano, 53, told Digital Trends. “There’s a LOT of blue-sky stuff.” It’s a very different mandate than Microsoft or Intel Research, and it’s not always a simple or fast process. One technology for near-field communication called FEEL took 13 years to make it into a Sony product. “Sometimes it goes very quickly, sometimes it takes a decade or so.”

Indeed, some of the division’s work isn’t taken up Sony at all. One of those inventions was an early network awareness service for augmented reality and location awareness.

“This was what, 10 or 15 years ago or something? And Sony had absolutely no clue what to do with it,” he said. CSL called the technology PlaceEngine and spun it off into a separate company called Koozyt, which Kitano said is one of the major providers of such service today in Japan. You can find the technology in the Tokyo National Museum as of April, among other places.

New frontiers: Open power systems, augmented humans

Sony has a second R&D lab, Kitano explained point out, a corporate lab more focused on products. CSL is unique among these tech labs for its focus on pure research, not just computer science fluid linguistics and system biology and statistics and something called synecoculture, which involves applying chaos theory to agriculture in the hopes of improving crop yields.

What, you didn’t know Sony was into agro-business? It’s not, at least not yet. And despite the enormous investment in battery technology and development, the company isn’t really in energy, at least not on the same level as Keyspan (formerly Consolidated Edison) or California Power and Light (CPL). But with the energy business set to boom …

“I think IAEA’s prediction is that we’re going to have $2.3 trillion U.S. dollars of new electricity market in the next 20 to 30 years. So if we can capture even a fraction of that it would be huge — way bigger than the current Sony business!” He joked.

Kitano said with a laugh. To that end, Sony has been at work on what it calls an Open Energy System, a way to create a new type of grid system for renewable energy. The big test is currently on faculty roofs at the Okinawa Institute of Science and Technology; nine houses there have been outfitted with solar panels and battery packs. They can generate electricity and share surplus to a neighbor’s battery pack, creating a DC load-balancing micro-grid. Sony plans to expand it to 20 of the 22 houses there shortly.

“What we’re trying to achieve is a new kind of grid system for renewable energy.”

Meanwhile the group has been testing the adaptability of its solar panel tech in other ways; they just returned from a trip to Cote D’Ivoire in West Africa, where they used roll-up sheets of solar panels to power a massive screen and let an entire village watch a 2014 World Cup match (Sony is a FIFA sponsor). The group conducted a similar test in Ghana in 2010, unexpectedly dropping in on a village after a 13-hour journey to show a game on a 200-inch screen.

“We just went there, without any appointment or anything, because there was no way to make an appointment,” Kitano told Digital Trends. He’s right: Whom do you call in a village that doesn’t have power? Word spread around nonetheless – more than 3,000 people came to watch the match. It was remarkable.

It’s one of those long-term projects at CSL, and still very much in the early stages. But Kitano – envisioning a new market where people can buy, sell, and trade renewable power with one another – believes it to be viable, with or without Sony.

“We are determined to go … even if Sony Corporation is not going to do this we actually are going to do it with a partner. Because energy, I don’t think we can do alone anyway.”

From Aibo and eBooks to ficas and food? CSL on Agriculture

Chemicals. Current crops are filthy with ‘em. And while they transformed our ability to grow crops, over reliance on them is a widespread concern. Can mathematics and chaos theory help? Perhaps, Kitano explained.

“Conventional agriculture, they plant one kind of species … one kind of crop. The problem with that is you’re going to have quite a bit of agrichemicals, and also it’s very fragile against the climate,” he said. By using what he deems open systems science – a way of looking at larger problems like climate change or financial crisis, things that are unrepeatable and hard to study – Kitano hopes his team can reduce reliance on pesticides and chemicals.

The concept: Plant a few different crops in the same place, for a “controlled micro agricultural system,” for a better yield that’s far more sustainable, especially in places where soil is poor or unsuitable for certain crops. What are you talking about? This guy seems to be crazy!” Kitano said. “Okay, let’s take him.”

The idea was spearheaded by Masatoshi Funabashi, who approached Sony CSL and said “I want to be a farmer.”

Sony has been working with local farmers, and some crops developed with the concept are being sold in the market. That’s right, Sony’s selling vegetables in Japan.

To help build its presence, Sony Computer Science Laboratories (CSL) will hold its first-ever symposium in the United States in September, an event at the Museum of Modern Art in New York to show off its work in human augmentation, among other things.

They’ll show off some of these concepts, as well as a prosthetic foot developed by Jun Rekimoto, a professor at the University of Tokyo and Deputy Director of Research with the CSL. His area of research spreads through augmented reality, including games inspired by quidditch from the Harry Potter series of books.

“We created a ball that can fly,” Rekimoto told Digital Trends. “There’s a huge gap between people that are good at sports and those that aren’t. But in the computer game area, there are a lot of imaginary things … “for the 21st century we can merge normal sports with virtual sports.”

He envisions a ball that can flee from the pursuer, just like in quidditch, help an infant develop eye-hand coordination, or train a young Jedi (we presume). Baby’s first gadget?

On a more serious side, Rekimoto is developing prosthetics for ordinary people (which can be printed out using 3D printers) and athletes. He plans to show off a working version at the symposium, a “smarter” version of the blades worn by such athletes as Oscar Pistorius.

“We think the future of sports is very exciting,” he said.

“Frontiers change,” Kitano told us. “We want to actually explore the frontiers. How can we contribute to society?”