WASHINGTON — The Reagan Administration said Monday that it believes a key part of the landmark Gramm-Rudman budget-slashing measure recently signed into law by the President is unconstitutional and declared that it will not defend the provision in court.

The development, which came in response to a lawsuit filed by Rep. Mike Synar (D-Okla.) challenging sections of the bill, increases the possibility that the mandatory, wide-ranging spending cuts called for in the legislation may never take place.

In a complex legal brief filed in U.S. District Court, the Justice Department argued that a special three-judge panel should throw out Synar's suit on a technicality when it hears arguments Jan. 10.

But, in a letter sent to Congress, Atty. Gen. Edwin Meese III said his department agrees with Synar that a mechanism established by the bill to trigger across-the-board spending cuts could not pass constitutional muster.

Meese's position signaled the Administration's unwillingness to defend a significant element of the Gramm-Rudman law if the court allows the Synar suit to proceed--or in other legal attacks that are sure to follow in the coming months.

It also put the Justice Department in the curious position of asking the court to uphold the legislation on technical grounds while acknowledging the merits of the substantive case against the bill.

California Rep. Leon E. Panetta (D-Monterey), one of the negotiators who helped fine-tune the controversial legislation, warned that a court order striking down parts of the legislation requiring compulsory cuts "knocks out the guts of the Gramm-Rudman process."

And Panetta said the Justice Department stand raises questions about President Reagan's sincerity in signing the politically popular legislation, which critics of the bill within the Administration had warned could harm the President's military buildup.

"If Justice can shoot a torpedo into the trigger mechanism, that's as good a way as any to torpedo the bill," Panetta said in a telephone interview.

The legislation, named for its principal Senate sponsors, Phil Gramm (R-Tex.) and Warren B. Rudman (R-N.H.), mandates an end to annual federal deficits--at $212 billion last year--by 1991.

Series of Targets

To accomplish that goal, the bill outlines a series of descending annual deficit targets that must be met either through congressional action or, failing that, through automatic spending cuts.

Synar contends that the procedure outlined for estimating the size of the mandatory cuts violates the separation of powers between the executive and legislative branches because it binds the President to implement trims based on deficit figures provided by the comptroller general of the General Accounting Office, an arm of Congress.

"In the event that the case is not dismissed, you should be advised of our view that the role prescribed for the comptroller general . . . is not constitutional," Meese wrote in his letter.

Backup Procedure

He contended that invalidating the provision would not seriously undercut the overall goal of the legislation because Congress, anticipating such a legal challenge, built in a backup procedure that eliminates the automatic nature of the spending cuts should the GAO provision be struck down.

Under that provision, the GAO would still determine the size of trims needed to meet the deficit targets, but Congress would have to vote to let the cuts go into effect.

However, during the months-long debate over the legislation in Congress, advocates repeatedly argued that it was the mandatory nature of the program that gave it teeth.

Could Use OMB Figures

Panetta acknowledged that the mandatory procedures could be saved and constitutional objections satisfied if the law were amended to rely solely on economic and spending projections prepared by Reagan's Office of Management and Budget instead of a congressional agency.

But, he cautioned, Democrats in Congress strongly resisted such a proposal during negotiations because they thought it would be tantamount to a wholesale transfer of budget-making power from Capitol Hill to the White House.

"If you just went to OMB, you might cure the constitutional problems, but you'd never get it through Congress," he said.

Paul Jacobson, a spokesman for Rudman, said the senator thinks the law still would place significant new pressure on Congress to slash spending, even if mandatory provisions were eliminated.

Joint Resolution

"The fallback position probably doesn't carry as much pull, but it does require a joint resolution by Congress (to countermand cuts), and there will be an awful lot of pressure," he explained.

White House spokesman Larry Speakes, explaining the Justice Department action to reporters in Palm Springs, Calif., said the Administration supports the Gramm-Rudman legislation.

". . . But there are certain parts of it that need clarifying so that you don't get down the road somewhere and look back and have a court case that would negate something," Speakes said.

Despite that position, the Justice Department brief sought a delay in getting such a clarification. It asked the court not to consider the Synar suit because the lawmaker had yet to demonstrate that he had been injured by the law and therefore had no legal standing to file suit against it.