TUESDAY, April 5, 2016 (HealthDay News) -- People who consume a lot of processed carbohydrates -- think snack foods and sweets -- and sugary drinks may face heightened risks of breast and prostate cancers, a new study suggests.

Researchers said the study, reported Tuesday at the American Society for Nutrition annual meeting in San Diego, does not prove that "bad" carbs cause cancer.

But given that breast and prostate cancers are two of the most common cancers in the United States, the connection gives more reason for people to cut processed foods from their diets, said lead researcher Nour Makarem.

"The carbohydrate quality of your diet matters for a number of reasons," said Makarem, a Ph.D. candidate in nutrition at New York University.

So the new findings -- considered preliminary until published in a peer-reviewed medical journal -- add more weight to that advice, Makarem said.

She pointed, in particular, to the link her team found between sugar-sweetened drinks (both soda and fruit juice) and prostate cancer risk. Compared with men who never drank sugary beverages, those who had them a few times a week showed more than triple the risk of developing prostate cancer.

And that was with other factors -- including obesity, smoking and other diet habits -- taken into account, Makarem said.

Still, it is difficult to weed out the effects of particular diet habits on cancer risk, said Marji McCullough, strategic director of nutritional epidemiology for the American Cancer Society.

When it came to prostate cancer risk, men who regularly drank sugary juices or soda were more than three times as likely to develop disease versus men who steered clear of those drinks, the findings showed.

That does not prove sweet drinks directly contribute to prostate cancer, Makarem acknowledged. Still, she said, many studies have implicated the beverages in the risks of obesity and type 2 diabetes -- so there are other reasons to cut back.

"Plus," Makarem said, "it's an easy change to make in your diet."

The American Beverage Association took issue with the findings.

"The authors of this study abstract acknowledge their findings do not show that beverages cause any disease," the group said in a statement. "Moreover, the study was limited to one demographic group that is not reflective of the population of the United States." (Most study participants were white.)

The beverage association also said that the American Cancer Society cites multiple potential risk factors for breast, prostate and colon cancer, so singling out diet is difficult. The group also said that because the study hasn't been published in a peer-reviewed journal, "very few study details are available" and it's therefore tough to draw firm conclusions.

Sugary drinks weren't the only diet factor that mattered, though, according to the researchers. Prostate cancer risk was also heightened among men whose diets were generally high in "glycemic load" -- which, Makarem said, basically means they ate a lot of refined carbs.

The study also implicated "processed lunch foods," including pizza, deli meats and burgers. Men who ate those foods four or more times a week were twice as likely to develop prostate cancer, compared to men who had them no more than once a week, the researchers found.

According to McCullough, it's hard to know whether certain foods, per se, contribute to breast or prostate cancers -- or whether, for example, it's overall calorie intake and weight gain that are the true culprits.

But the bottom line, Makarem said, is that whole, "high-quality" foods are a generally healthier choice than processed ones.