ATLANTA — The outcome of the Alabama Democratic primary runoff election for governor, which gave conservative Atty. Gen. Charles Graddick a razor-thin victory over liberal Lt. Gov. Bill Baxley, was invalidated Friday by a three-judge panel.

The judges, ruling in a case that has kept Democratic politics in the state in turmoil since the fiercely contested runoff, declared that Graddick violated federal election laws in his quest to succeed Alabama's long-time political boss, Gov. George C. Wallace.

The judges ordered the Democratic Party either to certify Baxley as the primary victor or else hold a new runoff election to determine Wallace's successor. Baxley is a frequent Wallace ally and the favorite of top Democratic Party regulars in the contest against Graddick, a former Republican.

"Interesting banana-republic politics," a Wallace aide, who requested anonymity, said of the decision.

State Democratic Party Chairman John Baker said party officials would consider naming Baxley the winner.

'Not Legal Votes'

"The court seems to say very clearly these are not legal votes. We must now go through the process to determine whether Mr. Baxley got the majority of the legal votes cast," he said.

Graddick won the June 24 runoff against Baxley by 8,756 votes out of the more than 930,000 ballots cast and claimed victory as the Democratic nominee to succeed Wallace.

Guy Hunt was the winner of the June 3 Republican primary, in which about 33,000 people voted. No Republican has been elected governor of Alabama since 1874, however, and Hunt is given little chance of victory in the November general election.

Baxley contended that many Republicans who voted in the June 3 GOP primary crossed over and voted in the Democratic runoff on June 24 as well, supplying Graddick with his slender victory margin.

Chris Grimshawe, a spokesman for Graddick, said the attorney general would not appeal the ruling. "We are not going to be sore losers," Grimshawe told the Associated Press, adding that Graddick would welcome a new runoff.

The judges, sitting in Alabama, ruled in a lawsuit filed by a black Pike County commissioner, Jerry Henderson, and his wife. The plaintiffs argued that the Republican crossover voters, who were overwhelmingly white, were able to vote twice and thus diluted the voting strength of blacks.

Blacks voted predominantly for Baxley, who had the endorsement of the state's two leading black political organizations, the Alabama Democratic Conference and the Alabama New South Coalition. Graddick, who campaigned on a "law and order" theme, largely ignored black voters.

The lawsuit charged that Graddick, in his capacity as attorney general, sent a letter on the eve of the runoff threatening elections officials with prosecution if they attempted to enforce the Democratic Party's rule against crossover voting.

The judges, ruling in the suit, dismissed Graddick's argument that the crossover rule has never been enforced and it would have resulted in illegal disenfranchisement of part of the electorate if the rule were enforced this time.

The rule was approved by the state party in 1979 and cleared by the Justice Department.

The special judicial panel--made up of U.S. Circuit Judge Frank Johnson and two U.S. district judges--found that Graddick, as the state's chief legal officer, violated the 1965 Voting Rights Act by using his office to block Democratic Party officials from fully enforcing a party rule against Republican crossover votes being cast in the Democratic runoff.

David Johnson, an attorney for Baxley, said: "What he (Graddick) did is totally illegal. If he can get out of this without being indicted, I'll lose faith in the justice system."

The judges said that Democratic Party officials advertised the anti-crossover rule after its passage and again in recent election years. But the panel said the Graddick letter, and a suit filed by a Republican at the request of Graddick campaign officials, had the effect of blocking party efforts to stop crossover voting when Graddick and Baxley met in the runoff.

The panel cited evidence from a pollster showing that about 15,800 GOP voters crossed over to vote in the Democratic race and that at least 84% supported Graddick. They represented the winning margin, the pollster contended. Graddick disputed the figures.

The three judges said the evidence made it clear that, without the crossover votes, Baxley would have been the winner, even though it was unclear how many people crossed over or how they voted.