A diet rich in meat, eggs, milk and cheese could be as harmful to health as smoking, according to a controversial study into the impact of protein consumption on longevity.

High levels of dietary animal protein in people under 65 years of age was linked to a fourfold increase in their risk of death from cancer or diabetes, and almost double the risk of dying from any cause over an 18-year period, researchers found. However, nutrition experts have cautioned that it's too early to draw firm conclusions from the research.

The overall harmful effects seen in the study were almost completely wiped out when the protein came from plant sources, such as beans and legumes, though cancer risk was still three times as high in middle-aged people who ate a protein-rich diet, compared with those on a low-protein diet.

But whereas middle-aged people who consumed a lot of animal protein tended to die younger from cancer, diabetes and other diseases, the same diet seemed to protect people's health in old age.

The findings emerged from a study of 6,381 people aged 50 and over who took part in the National Health and Nutrition Examination Survey (NHANES) which tracks a representative group of adults and children in the US.

The study throws doubt on the long-term health effects of the popular Atkins and Paleo diets that are rich in protein. Instead, it suggests people should eat a low-protein diet until old age when they start to lose weight and become frail, and then boost the body's protein intake to stay healthy. In the over-65s, a high-protein diet cut the risk of death from any cause by 28%, and reduced cancer deaths by 60%, according to details of the study published in the journal Cell Metabolism.

Valter Longo, director of the Longevity Institute at the University of Southern California, said that on the basis of the study and previous work, people should restrict themselves to no more than 0.8g of protein a day for every kilogram of body weight, equivalent to 48g for a 60kg person, and 64g for an 80kg person.

"People need to switch to a diet where only around nine or ten percent of their calories come from protein, and the ideal sources are plant-based," Longo told the Guardian. "We are not saying go and do some crazy diet we came up with. If we are wrong, there is no harm done, but if we are right you are looking at an incredible effect that in general is about as bad as smoking."

"Spend a couple of months looking at the labels on your food. There is a little bit of protein everywhere. If you eat breakfast, you might get 4g protein, but a piece of chicken for lunch may have 50g protein," said Longo, who skips lunch to control his calorie and protein intake.

People who took part in the study consumed an average of 1,823 calories a day, with 51% coming from carbohydrates, 33% from fat, and 16% from protein, of which two thirds was animal protein. Longo divided them into three groups. The high-protein group got 20% or more of their calories from protein, the moderate group got 10 to 19% of their calories from protein, and the low group got less than 10% of calories from protein.

Teasing out the health effects of individual nutrients is notoriously difficult. The apparently harmful effects of a high-protein diet might be down to one or more other substances in meat, or driven by lifestyle factors that are more common in regular red meat eaters versus vegetarians. Other factors can skew results too: a person on the study who got ill might have gone off their food, and seen a proportional rise in the amount of calories they get from protein. In that case, it would be the illness driving the diet, not the other way round.

"I would urge general caution over observational studies, and particularly when looking at diet, given the difficulties of disentangling one nutrient or dietary component from another. You can get an association that might have some causal linkage or might not," said Peter Emery, head of nutrition and dietetics at King's College London.

Gunter Kuhnle, a food nutrition scientist at Reading University, said it was wrong "and potentially even dangerous" to compare the effects of smoking with the effect of meat and cheese as the study does.

"Sending out [press] statements such as this can damage the effectiveness of important public health messages. They can help to prevent sound health advice from getting through to the general public. The smoker thinks: 'why bother quitting smoking if my cheese and ham sandwich is just as bad for me?'"

Heather Ohly at the European Centre for Environment and Human Health in Exeter said: "Smoking has been proven to be entirely bad for us, whereas meat and cheese can be consumed in moderation as part of a healthy diet, contributing to recommended intakes of many important nutrients."

Most people in Britain eat more protein than they need. The British Dietetic Association recommends a daily intake of 45g and 55g of protein for the average woman and man respectively. But according to the British Nutrition Foundation the average protein intake per day is 88g and 64g for men and women.

In a series of follow-up experiments, Longo looked at what might lie behind the apparently damaging effects of a high-protein diet on health in middle age. Blood tests on people in the study showed that levels of a growth hormone called IGF-1 rose and fell in line with protein intake. For those on a high protein diet, rises in IGF-1 steadily increased their cancer risk. Further tests on mice found that a high-protein diet led to more cancer and larger tumours than a low-protein diet.

Zoe Williams: This week a new report said that a diet high in animal protein could increase oour chances of dying from cancer or diabetes fourfold. So if protein has joined sugar, fat and carbohydrate in being bad for us, what's left for dinner?