Bayer AG is stepping up the legal defense of its flagship weed killer, after a verdict in a recent case alleging the chemical causes cancer sent shares down sharply and raised the prospect of costly plaintiff payouts.

Bayer on Tuesday said it wants a California state court judge to overturn the jury's verdict, order a new trial or reduce damages, according to a court filing. The company aims to defeat a $289 million award in August in one of the first of thousands of cases filed by gardeners, farmers and others claiming Bayer's Roundup herbicide gave them cancer.

The jury in that case ruled unanimously in favor of a former groundskeeper who sought to hold the maker of Roundup liable for his non-Hodgkin lymphoma. The verdict came only two months after the German pharmaceutical and chemical conglomerate sealed its takeover of Monsanto, the U.S. agriculture giant that invented the herbicide.

Bayer shares have dropped about 22% since the verdict to five-year lows. Investors fear a lengthy legal battle and more damage awards could cost the company billions of dollars. Some have questioned whether Bayer Chief Executive Werner Baumann properly evaluated the risks of taking over Monsanto in a 2016 deal valued at more than $60 billion, the biggest ever by a German company.

Bayer now faces 8,700 plaintiffs in the U.S., up from a few hundred in the spring of 2016. Bayer has said it expects that number to grow.

"Werner Baumann must ask himself if Bayer took too lightly the lawsuits against Monsanto," said Winfried Mathes, a corporate expert from Bayer shareholder Deka Investment. The share-price drop has also rattled employees inside the Leverkusen, Germany-based company, according to people familiar with their thinking.

Mr. Baumann has told investors that Bayer's arguments about science proving the weed killer's safety will prevail. "We stand behind the product and the science backing it up," he said on a conference call in August.

In its filing on Tuesday with the California state court, Bayer argued that the plaintiff's lawyers relied on flimsy scientific evidence that doesn't support a link to cancer, and that jurors were swayed by overly emotional and speculative arguments from the plaintiff's lawyers.

Pedram Esfandiary, an attorney for Baum Hedlund Aristei and Goldman PC, which is representing the plaintiff in the case, said the jury verdict demonstrated that Monsanto's scientific case wasn't convincing. "I think the odds are very slim of them prevailing," he said.

Judge Suzanne Ramos Bolanos is expected to rule on Bayer's requests by late October or early November.

Bayer entered deal talks with Monsanto in 2016 aware of the problems facing Roundup, people familiar with the negotiations said. Still, there were legal limits to how much information Monsanto could share before antitrust officials approved the merger, Bayer has said.

Representatives for the European Union's competition authority and the U.S. Department of Justice had no immediate comment.

Bayer first learned of dozens of internal Monsanto emails discussing glyphosate's safety and strategies to publicly defend it as part of U.S. court proceedings. Those emails include what plaintiffs' lawyers say is evidence of Monsanto ghostwriting articles for outside scientists to defend the chemical's safety.

After assuming control of Monsanto this summer, Bayer found no "smoking gun" in Monsanto's internal communications, Mr. Baumann told investors last month. Lawyers representing cancer victims used those internal communications "out of context on purpose," he said. Monsanto's scientists and the article authors denied ghostwriting allegations, Bayer said.

The deal made Bayer the world's biggest supplier of pesticides and seeds for farmers, which now generate nearly half of group sales. Profit margins are slim for Roundup, its top-selling crop spray, but the bulk of the nearly $11 billion in crop seeds Monsanto sells annually are genetically engineered to withstand glyphosate, the potent weed-killing chemical in Roundup.

The International Agency for Research on Cancer, part of the World Health Organization, in 2015 classified glyphosate as likely having the potential to cause cancer. Monsanto fought back, pointing to studies by academics and agencies like the U.S. National Institutes of Health and Environmental Protection Agency that showed no cancer risk.

Legal academics said successful challenges to jury verdicts aren't uncommon. Last October, another California state judge overturned a $417 million judgment against Johnson & Johnson after a woman alleged that the company's baby powder contributed to her developing ovarian cancer.

In January, a judge in Pennsylvania state court overturned a jury's verdict and $28 million in damages and ruled in Bayer's favor in a lawsuit alleging the company and Johnson & Johnson didn't properly warn about internal bleeding risks from the drug Xarelto.

"For Bayer, the most important thing is to have a judge say the science doesn't hold up," said Alexandra Lahav, law professor at the University of Connecticut.

Bayer has sometimes settled product lawsuits. In 2005, Bayer paid $1.15 billion to settle some 3,000 death and injury claims over its withdrawn cholesterol-lowering drug Baycol.

The company also spent over $2 billion to settle thousands of cases claiming it didn't adequately inform women of the risk of thrombosis and other side effects from its hormone-based contraceptive pills Yaz, Yasmin and Yasminelle.