Stories

Behind the boom in machine learning

Photo illustration: Axios Visuals

Machine learning is a core technology driving advances in artificial intelligence. This week, some of its earliest practitioners and many of the world's top AI researchers are in Long Beach, CA, for the field's big annual gathering—the Neural Information Processing Systems (NIPS) conference. In all, some 7,700 people are to attend AI's version of high tech's glitzy South by Southwest conference, and the electronic device industry's even bigger annual CES conference.

It's NIPS' 31st year in what originally drew just a few hundred participants — computer scientists, physicists, mathematicians and neuroscientists all interested in AI. Terrence Sejnowski, a computational neuroscientist at the Salk Institute for Biological Studies and president of the NIPS Foundation, spoke with Axios about growth in the field and what's next.

How machine learning has grown since NIPS' start in the '80s: "Over that period what happened was a convergence of a number of different factors, one of them being the fact that computers got a million times faster. Back then we could only study little toy networks with a few hundred units. But now we can study networks with millions of units. The other thing was the training sets — you need to have examples of what it is you're trying to learn. The internet made it possible for us to get millions of training examples relatively easily, because there's so many images, abundant speech examples, and so forth, that you can download from the internet. Finally, there were breakthroughs along the way in the algorithms that we used to make them more efficient. We understood them a lot better in terms of something called regularization, which is how to keep the network from memorizing — you want it to generalize."

The role of hardware: "By far, right now, the most exciting part of the hardware development is special purpose digital chips that speed up and are able to enhance the learning. That is to say, the bottleneck right now for learning is the fact that you have to give it many examples. Basically it's applying the same simple operation over and over again.

"What's happened is that Nvidia and Intel, and a dozen other startup companies, are designing special purpose learning chips. Actually, Google already has one that's called TPU, tensor processing unit, which they're using in the cloud because of the fact that it's much more efficient in terms of the energy use and the speed. Without it they wouldn't have been able to roll out services using deep learning — things like language translation. There's literally billions of dollars that are being invested right now in digital hardware.

"The problem though is that, of course, if you want to put it into a cell phone you have to make it very low power. The next generation will be even lower power chips using analog VLSI [very-large-scale-integration]. That's being driven by the applications and the technology. The cell phone market is huge, we're talking about billions of chips out there that can be put into cell phones."

The coming challenge: "We can now put a million units with a billion connections and train it to do something. If you look in the brain, that's about five square millimeters of the cortex. What will eventually happen — and it is happening — is that we know that each part of the cortex is specialized for a different function. Each little patch, very tiny patch, has been dedicated to all these different functions, which we know are separate networks and are doing separate tasks, which is kind of a modular approach.

"If you look into the way the cortex works it's really interesting because all these areas are interconnected with each other. It's not like they're isolated from each other. Right? There are long range, and there are short range connections. The big challenge is the global organization of all of these, right now, modular networks that have been designed for one task each network. It'll happen. It's beginning to happen but it will require theoretical advances for how to organize all the information that is distributed over the entire cortex.

"This is a very exciting area in neuroscience right now because we have tools and techniques, like brain imaging, that we can actually see that happening, both during learning and also during memory consolidation during sleep. As we learn more about how the cortex organizes information globally, it should be possible to translate that into a global workplace built out of all of these chips that are being designed for all these special applications."

"There are a lot of other problems too, but it seems to me that the integration problem may be the key to general intelligence. That's something people like to talk about. They say, 'Oh, you solved these little applications but you haven't figured out how to get much more flexible behavior that involves integrating all that information.' My guess is that if we could figure out how the brain solves the global integration problem, we'll be on our way to understanding a little bit more about general intelligence."

Early humans innovated tools earlier than thought

Unpredictable climate and natural disasters like earthquakes may have spurred early humans to create innovative tools and ways to communicate earlier than previously thought, according to 3 studies published Thursday in Science.

What they found: Evidence that around 320,000 years ago — near the start of the Middle Stone Age (MSA) and tens of thousands of years earlier than previous evidence has shown — early humans in East Africa may have created projectile hunting tools, developed ways to communicate using colors for mapping or identification purposes, and traveled longer distances to trade, hunt or obtain valuable materials.

"It's not just humans changing but really the entire ecosystem. It's a picture that's bigger than just the human ancestors themselves."

Yonatan Sahle from Tubingen University told The Atlantic that different parts of Africa may have had varied timing when MSA first appeared, how much it overlapped with the older Acheulean tech, and whether it occurred together with Homo Sapiens fossils.

Why it matters:

"This is surely a landmark study. The work at Olorgesailie is most welcome as plaeoanthropologists have very little information about the habitats and behaviors at 320,000 years ago, a critical time period in human evolution... the rigorous work at Olorgesailie fills a a gap in our knowledge about environments and human behaviors at this critical time. It is very rare to have well-dated stone tools in association with animal remains and environmental information."

— Michael Petraglia, from the Max Planck Institute for the Science of Human History, Germany, who was not part of the studies

Environmental triggers: The scientists believe dramatic changes in the environment like periods of intense rain or drought, earthquakes, and altered animal communities pushed early humans to travel greater distances for food, find new super-sharp obsidian rocks to use for tools and pigments to use for communication, and locate other communities of early humans with whom to trade.

"[T]he behavioral hallmarks of the MSA they observe — refined tool manufacture, wider mobility and foraging, pigment use, implied social networks — occurred during a time of enhanced environmental variability. As with modern hunter gatherer populations, these are anticipated behavioral and cultural responses to greater environmental variability and food insecurity."

— Columbia University's Peter de Menocal, who was not part of the study

Innovative tools and weapons: Prior to this time period, early humans had been using rudimentary hand axes and cleavers to hunt animals and as weapons.

The teams discovered large amounts of non-native obsidian rock in the basin. The obsidian, which is extremely sharp when fractured, was traced back to their origin at locations up to 55 miles away through very rugged terrain.

Prior to this, 99% of their rocks were obtained within 5 km — this is a "radical change," Potts says.

What it means: "The transfer of obsidian from long distances is essentially the first evidence of trade," Potts says.

The smaller shapes and modification at the base suggests the pointed tips were hafted to wood or bone — essentially the original projective weapon.

"What can you say — the world hasn't been the same since [the discovery of projective weapons]," Potts says.

Pigments: The teams found red, green and black pigments used in various locations. "Pigments are evidence of symbolic communication," Potts says. It can be used to stain the hair or skin to show if you are "friend or foe." It could also be used to mark the best way to get somewhere on a map — or warn people off a particular territory.

"The possible production of red pigments is especially exciting, as this may imply that the humans at Olorgesailie were cognitively advanced. This is some of the best, and earliest evidence for pigment use in the archaeological record, reaffirming that at the outset, Homo sapiens was a symbolic species," Petraglia says.

What's next: Research is lacking for a chronological gap — the time period between 500,000 years ago and 320,000 years ago. Potts says his team is currently working to discover more about that period.

Yejin Choi: Trying to give AI some common sense

Photo illustration: Axios Visuals

Artificial intelligence researchers have tried unsuccessfully for decades to give machines the common sense needed to converse with humans and seamlessly navigate our always-changing world. Last month, Paul Allen announced he is investing another $125 million into his Allen Institute for Artificial Intelligence (AI2) in a renewed effort to solve one of the field's grand challenges.

Axios spoke with Yejin Choi, an AI researcher from the University of Washington and AI2 who studies how machines process and generate language. She talked about how they're defining common sense, their approach to the problem and how it's connected to bias.

How do you define common sense?

"Common sense is fairly trivial everyday knowledge that we have about people and about the world. It's knowledge about how the world works — how people think, what motivates them, how they act, and why they do what they do."

"Imagine there's a robot in your household in the future, and you want to store leftover pie in a container. The robot should pick a container that's large enough to store that pie, and today that spatial reasoning relative to physical properties of different objects in the world and how you interact with them are not quite well represented in these system models."

Why common sense poses a challenge to machines:

"We have a world model in our mind when we do daily operations. AI systems today, despite tremendous advancement in recent years, they are not very good at generalizing out of pure example, so they tend to be very, very task specific, and very domain specific."

"A machine translation system may seem like it understands some language enough to translate into another language, but actually there's not that much understanding happening, per se, because that syntax knowledge cannot be reused for making very trivial small talk with a human, for example."

Their approach:

"We have this commonsense knowledge without our parents or teachers having to enumerate all of it one by one. Nobody told us that elephants are usually bigger than butterflies, however we can reason about it. You ask me that question, I can think about it, and I can answer that question even though I've never seen that statement explicitly written anywhere."

"We're taking a similar approach. It may be possible that we can learn to answer these sort of questions, even including those that we've never seen before. That's fundamentally the ability that AI systems needs to have — dealing with unknowns and previously unseen situations."

What data is needed to make common sense models?

"There is a paper called Verb Physics, and in that work the dataset is basically a combination of a lot of natural language documents — a huge corpus of how people use language and from that we look for patterns. For example, what kind of things do I throw? What kind of things do I enter into? I enter my house. I exit my house. And, that sort of implies that my house must be bigger than me for me to enter into and exit from."

"So, we can infer different action dynamics, preconditions, and post conditions — all those different physical objects, for me to do some action involving them."

"The short term goal is to develop a common sense benchmark dataset. Then, the ultimate goal is to acquire knowledge that's good enough to do well in that benchmark dataset. That's step one."

The issue of AI and human bias:

"We showed [in a study of movie scripts last year] how women in movies carry much less power compared to men. It's the kind of actions that they do and the kind of language they use when they speak. Men usually fight and they do stuff, they save the world. Women, on the other hand, they tend to wait, they are being watched, and they look pretty. What they do tends to be pretty passive."

"It's one of my passions to develop AI technology that can detect all these biases in humans and also, ideally, be able to correct them in the future."

How is bias connected to common sense?

"These are connected in that the way bias is coming across, often times can be inclusive or implied. Current models are much better at understanding what's explicitly stated, but less good at anticipating what's not said. It's good to be catching some of the explicit biases, but it's important to also detect all of the implied ones because that still influences us. The ability to read between the lines ultimately is what requires common sense, so that's the connection between the two."