AI suggests recipe for a dish just by studying a photo of it

Ever eaten a dish you didn’t know the name of and wished you had the recipe so you could recreate it at home? Soon you might only need a picture of it.

Researchers have devised a machine learning algorithm that looks at photos of food and predicts the recipe that created the dish.

Nick Hynes at Massachusetts Institute of Technology and his colleagues trained the algorithm on one million recipes, each with an illustration of the finished result, from dozens of cooking websites. Given a fresh photo of a dish, the system picked the right recipe 65 per cent of the time.

Advertisement

The AI can also work out from a photo what ingredients went into a food: presented with an image of a plate of biscuits, for example, it knows that they are likely to include flour, eggs and butter. But at the moment it can’t necessarily tell how the ingredients were prepared – it can’t work out whether onions were stewed or fried, for example, an ability Hynes hopes it will gain in future.

And although it struggles to recognise the hidden ingredients in a sushi roll, the system is particularly good at finding recipes for cookies and muffins, Hynes says, because they are relatively popular online. The research will be presented later this month at the Computer Vision and Pattern Recognition conference in Honolulu, Hawaii.

Hamed Haddadi at Queen Mary University of London is impressed with the idea. Eventually people could use an improved version of the algorithm to help them track their diet throughout the day. “The bigger goal is to accurately tell how many calories there are in a specific dish,” he says.

App such as MyFitnessPal already let people track calorie intake, but they have to manually input what they eat. “It’s pretty tedious,” says Haddadi.

Hynes’s AI is also not too good at recognising the subtleties of dishes. Given a photo of an aubergine lasagne, for example, it’s more likely to dish up a generic lasagne recipe rather than a specific one. This could be improved, Hynes says, if users specified a couple of hard-to-see ingredients as well as providing a photo.

The dream of complete recipe recreation from a single snap is still a while away, says Haddadi, but app makers are already working hard on the problem.

In May, Pinterest added dish recognition to its image-searching app. Now if you take a photo of a meal using the app, it will recognise certain ingredients and offer recipes related to them. The company plans to use the technology to help food brands advertise to Pinterest users.

But image recognition algorithms can only go so far, says Christoph Trattner at MODUL University Vienna in Austria. He thinks two very similar-looking foods, such as a glass of cola and a black coffee, could easily fox a machine. “I doubt that image technology on a mobile device will ever be able capture accurately the difference between the two without human intervention,” he says.