Cooking with I.B.M.: The Synthetic Gastronomist

One sunny afternoon a few weeks ago, during a visit to the Institute of Culinary Education in Manhattan, I had a taste of Czech pork-belly moussaka. It was neither the best nor the worst meal I’ve ever had, but in some ways it was the most remarkable. For it was, almost literally, a taste of the future: a collaborative product between man and machine.

The man was James Briscione, a dashing young chef and instructor at the Institute, who specializes in modern cooking techniques like sous-vide and elBulli-style hypercolloids. The machine, as yet unnamed, came from I.B.M. It wasn’t the “Jeopardy”-winning Watson (as erroneously reported recently by the New York Times), or even closely related to Watson in its underlying algorithms. But it was quite original in its own way.

From the botched Times story, I had envisioned a version of I.B.M.’s Watson retooled to find recipes for pastries—a sort of glorified version of Epicurious.com. What I found instead was nearly the opposite, an effort at “computational creativity” that relies as much on software models of the psychology of taste perception as it does on a database of recipes. The goal of this computer isn’t to retrieve what is already known but to discover something new.

That goal in itself is unusual. Most work in computational creativity aims to replicate the styles of earlier masters; only rarely do machines reach for genuinely new territory. (David Cope’s famous computer program Emily, for example, tried to make ersatz Bach and faux Chopin, internalizing the greats but not daring to stray far from their well-trodden paths.) The goal here was different. “We’re not trying to solve the Turing test for cooking,” said Lav Varshney, one of the system’s designers. “We’re trying to invent new kinds of recipes.”

For now, the Gastronomist (as I shall call it) is desperately crude; it can spit out lists of ingredients, but it doesn’t say what should be chopped and what should be stirred. It doesn’t even tell you how much you’ll need. Instead, for now, it takes selections about genres and key ingredients and feeds them into a complex system that combines a repository of recipes with databases of “psycho- and chemo-informatics”—computer models of how the human palate might respond to different combinations of flavors. There are literally trillions of possibilities. The program’s mission is to sort them based on its predictions of how pleasant and surprising those combinations might be.

The Gastronomist frequently produces head-scratching provocations. When I got a chance to try out the program, with Chef Briscione and two programmers from I.B.M. looking on, I requested a surprising Indian ceviche with fruit and tequila, as if playing some sort of deranged culinary Mad Libs. The program responded in kind by proposing the inclusion of red cabbage, based on its “hedonic flavor prediction” modules.

Briscione lives for these challenges, and he pointed out that you could blanch the cabbage and make it into rolls with which to wrap the ceviche. On the day I visited, Briscione was developing three computer-inspired dishes, starting from the program’s list of ingredients, filling in his own guesses about quantities and cooking techniques, and then testing out those ideas in the well-stocked Institute of Culinary Education kitchen. On the whiteboard for the day were a Spanish butterless pastry (designed inadvertently because the program overgeneralized the Spanish love of olive oil), a Midwestern mushroom Stroganoff, and the Czech moussaka, the product of another round of culinary Mad Libs in which one person shouted out “Czech,” another “moussaka,” and a third “pork belly.” On the day I visited, the chef was busy transforming the computer-generated list of moussaka ingredients into a three-dimensional meal.

Czech moussaka is such a rare dish that a Google search yields little. The Gastronomist itself didn’t have a single Czech moussaka in its own database; in the twenty thousand “seed recipes” it culled for initial inspiration, there were only two moussaka recipes with pork (both found on the Internet, a sometimes dubious dispenser of culinary wisdom), and not a one under the heading of Czech moussaka. The Gastronomist combined the meagre inspirations it could find with its broader database of guesses about how human taste buds might respond to various possible combinations. In the end, it spat out a very particular list of ingredients: pulled pork, peas, celery root, parsley root, red bell peppers, Cheddar, Swiss, dill, butter, cottage cheese, eggs, flour, and milk. Eggplant was notably absent.

From there, Briscione came in and figured how to turn those ingredients, including the unlikely soul mates of celery, dill, peas, and pork, into an edible concoction that could fairly be called moussaka. Briscione quickly rejected his own first thought—to use pork in place of moussaka’s traditional mincemeat—as too easy and too obvious. Briscione insisted instead on reconstruing the vegetables as the substitute for the usual mincemeat and the pork as substitute for the eggplant, ultimately employing half a dozen of the ingredients in a celery-root béchamel that was shockingly tasty.

For now, humans are essential. No robot has hands as dextrous as Briscione’s, and no bit of software can see the whole culinary picture as he can. The computer can make some educated guesses about what ingredients might work well together (it turns out that chocolate works well with caviar), but it doesn’t yet have an internal representation of what it means to stir, chop, fillet, or boil something, nor an understanding of the spatial placement of the molecules that make up a dish. It has no idea, for instance, of the role of yeast in the structural coherence of bread (even if it might notice that yeast is a common ingredient). It also lacks a model of the human visual system, so it has no understanding of what looks good. In consequence, the Gastronomist knows nothing about plating, texture, or the miracles of sous-vide. Big Data might help somewhat—one might imagine looking for correlations between reviews and techniques on Epicurious.com, but there’s no reason to think such data dredging will, by itself, lead to the kind of intuitive understanding of the physics and chemistry and visual aesthetics of cooking that skilled chefs develop.

Not long after the béchamel was ready, Briscione had completed the dish and served out small plates of the moussaka; criticisms were swapped. The flavor was good, but the pork didn’t stand out enough; the peas, too, needed to pop more. The texture wasn’t quite right. Next time, Briscione would sear the pork and cut it into tiny strips. Sous-vide seemed to be on the agenda as well. For now, Briscione was only willing to give the dish, borne of man collaborating with machine, a grade of a strong B, nothing more. Then he helped himself to seconds.

Illustration by Richard McGuire

Gary Marcus is a professor of cognitive science at N.Y.U. and the author of “Guitar Zero.”