Goldberg was born and raised on the West Side of Chicago, the youngest of eight children of Rebecca Perlstein and Joseph Goldberg, Jewish immigrants from the Russian Empire. The maternal side of the family originally came from a shtetl called Zenkhov in Ukraine. The paternal side of the family, Goldberg-Flaumen, came from the Polish shtetl of Oświęcim (Auschwitz), later infamous as the site of the Auschwitz concentration camp. Goldberg's father, a produce peddler, died in 1916, forcing Goldberg's siblings to quit school and go to work to support the family. As the youngest child, Goldberg was allowed to continue school, graduating from high school at the age of 16.

Goldberg's interest in the law was sparked by the noted murder trial in 1923 of Leopold and Loeb, two wealthy young Chicagoans who were spared the death penalty with the help of their high-powered defense attorney, Clarence Darrow. Goldberg later pointed to this case as inspiration for his opposition to the death penalty on the bench, since he had seen how inequality of social status could lead to unfair application of the death penalty.

In 1931, Goldberg married Dorothy Kargans. They had one daughter, Barbara Goldberg Cramer, and one son, Robert M. Goldberg (an attorney in Anchorage, Alaska).[5] He was the uncle of Barry Goldberg.[6]

During World War II, Goldberg served in an espionage group operated by the Office of Strategic Services, the precursor to the CIA.[7] The Jewish Telegraphic Agency stated that: "Goldberg's file notes that as both a civilian and a member of the Army, he supervised a section in the Secret Intelligence Branch of OSS to maintain contact with labor groups and organizations regarded as potential resistance elements in enemy-occupied and enemy countries. He organized anti-Nazi European transportation workers into an extensive intelligence network."[8]

Goldberg became a prominent labor lawyer, representing striking Chicago newspaper workers on behalf of the Congress of Industrial Organizations (CIO) in 1938. He served in the Office of Strategic Services as chief of the Labor Desk, an autonomous division of the American intelligence agency that was charged with the task of cultivating contacts and networks within the European underground labor movement during World War II. Appointed general counsel to the CIO in 1948, Goldberg served as a negotiator and chief legal adviser in the merger of the American Federation of Labor and CIO in 1955. Goldberg also served as general counsel of the United Steelworkers of America.

Perhaps Goldberg's most influential move on the Court involved the death penalty. Goldberg argued in a 1963 internal Supreme Court memorandum that imposition of the death penalty was condemned by the international community and should be regarded as "cruel and unusual punishment," in contravention of the Eighth Amendment. Goldberg was the first to argue this position: prior to Goldberg's memo, no Supreme Court case had addressed the question of whether the death penalty violated the Eighth Amendment. Finding support in this position from two other justices (William J. Brennan and William O. Douglas), Goldberg published an opinion dissenting from the Court's denial of certiorari in a case, Rudolph v. Alabama, involving the imposition of the death penalty for rape, in which Goldberg cited the fact that only five nations responding to a UN survey indicated that they allowed imposition of the death penalty for rape, including the U.S., and that 33 states in the U.S. had outlawed the practice.

Goldberg's dissent sent a signal to lawyers across the nation to challenge the constitutionality of capital punishment in appeals. As a result of the influx of appeals, the death penalty effectively ceased to exist in the United States for the remainder of the 1960s and 1970s, and the Supreme Court considered the issue in the 1972 case of Furman v. Georgia, where the Justices, in a 5 to 4 decision, effectively suspended the death penalty laws of states across the country on the ground of the capricious imposition of the penalty. That decision would be revisited in 1976's Gregg v. Georgia, where the justices voted to allow the death penalty under some circumstances; the death penalty for rape of an adult female victim, however, would be struck down in 1977's Coker v. Georgia. In 2008 the death penalty for rape of children was ruled unconstitutional by a 5 to 4 decision (Kennedy v. Louisiana).

During his tenure on the Supreme Court, one of his law clerks was future associate justice Stephen Breyer, who holds the exact seat Goldberg once occupied. Another was prominent criminal law professor Alan Dershowitz. Goldberg resigned from the Supreme Court to become the U.S. ambassador to the U.N, in what has been described as a calculated move by Lyndon Johnson in order for Johnson to appoint his longtime friend Abe Fortas to Goldberg's seat (which some at that time called the "Jewish seat" on the Court).[9][10]

In 1965, Goldberg was persuaded by President Johnson to resign his seat on the court to replace the recently deceased Adlai Stevenson as the Ambassador to the United Nations. Johnson wanted to appoint his friend Abe Fortas to the court,[10] in case any of his Great Society reforms were going to be deemed unconstitutional by the Court; he thought Fortas would notify him in advance.[11] Goldberg had declined an earlier offer to leave his position to be Secretary of Health, Education and Welfare.[11] He did take Johnson's offer of the UN ambassadorship when Johnson discussed it with him on Air Force One to Illinois for the burial of Stevenson, however.[11]

Goldberg wrote in his memoirs that he resigned in order to have influence in keeping the peace in Vietnam and that after the crisis had passed he expected he would be reappointed to the Supreme Court by Johnson. "I had an exaggerated opinion of my capacities. I thought I could persuade Johnson that we were fighting the wrong war in the wrong place [and] to get out."[12]

David Stebenne, Goldberg's biographer, adds "Many observers, then and later, found this answer hard to accept." He suggests that "Johnson must have had some influence over Goldberg that induced him [to resign from the Supreme Court]." Time reported in 1962 that Johnson knew that for a party thrown in Johnson's honor that year, a Goldberg aide, Jerry Holleman, solicited contributions from wealthy supporters of Johnson, including Billy Sol Estes. Holleman accepted responsibility and there was no public awareness of Goldberg and Johnson's involvement.[13]

Johnson said of the Goldberg decision in his later-released audio tapes:

Goldberg would be able to answer the Russians... very effectively... He's got a bulldog face on him, and I think this Jew thing would take The New York Times-- all this crowd that gives me hell all the time-- and disarm them. And still have a Johnson man. I've always thought that Goldberg was the ablest man in Kennedy's Cabinet, and he was the best man to us.... Goldberg sold bananas, you know.... He's kind of like I am... He's shined some shoes in his day and he's sold newspapers, and he's had to slug it out...[11]

In 1984, while teaching a seminar at Boston University School of Law, Goldberg addressed the question of his resignation, stating that he was persuaded to leave the Court for the United Nations based on certain promises made by President Johnson. Goldberg stated that Johnson promised to give Goldberg full discretion in negotiating an end of the war in Vietnam. In addition, Johnson told Goldberg that if he successfully negotiated an end to the Vietnam War, Johnson would support Goldberg for the presidency and Goldberg would be the first Jewish President of the United States. Once Goldberg resigned, however, he failed to get the support from Johnson that he had been promised.

In 1967, Goldberg was a key drafter of Resolution 242, which followed the 1967 Six-Day War between Israel and the Arab states. While interpretation of that resolution has subsequently become controversial, Goldberg was very clear that the resolution does not obligate Israel to withdraw from all of the captured territories. He stated that:

The notable omissions in language used to refer to withdrawal are the words the, all, and the June 5, 1967, lines. I refer to the English text of the resolution. The French and Soviet texts differ from the English in this respect, but the English text was voted on by the Security Council, and thus it is determinative. In other words, there is lacking a declaration requiring Israel to withdraw from the (or all the) territories occupied by it on and after June 5, 1967. Instead, the resolution stipulates withdrawal from occupied territories without defining the extent of withdrawal. And it can be inferred from the incorporation of the words secure and recognized boundaries that the territorial adjustments to be made by the parties in their peace settlements could encompass less than a complete withdrawal of Israeli forces from occupied 23 territories [italics by Goldberg].[14]

Frustrated with the war in Vietnam, Goldberg resigned from the ambassadorship in 1968 and accepted a senior partnership with the New York law firm Paul, Weiss, Rifkind, Wharton & Garrison. Longing to return to the bench, Goldberg later claimed that he was Earl Warren's preference to succeed him when the chief justice announced his retirement in 1968, but President Johnson selected Abe Fortas instead.[16] After Fortas's nomination was withdrawn in the face of Senate opposition, Johnson briefly considered naming Goldberg chief justice as a recess appointment before rejecting the idea.[17]

With the prospect of a return to the Supreme Court closed to him by the election of Richard Nixon, Goldberg contemplated a run for elected office. Initially considering a challenge to Charles Goodell's reelection to the United States Senate, he decided to run against New York GovernorNelson Rockefeller in 1970. Though the former justice initially polled well, a contested primary and Goldberg's own poor skills as a campaigner, coupled with Rockefeller's formidable advantages, resulted in a 700,000 vote margin of victory for the incumbent Republican.[18]

After his defeat Goldberg returned to law practice in Washington, D.C., and served as President of the American Jewish Committee.[19] In 1972, Goldberg returned to the Supreme Court as a lawyer, representing Curt Flood in Flood v. Kuhn. His oral argument was referred to by one observer as "one of the worst arguments I'd ever heard – by one of the smartest men I've ever known..."[20] Under President Jimmy Carter, Goldberg served as United States Ambassador to the Belgrade Conference on Human Rights in 1977, and was awarded the Presidential Medal of Freedom in 1978.

^Editors of Time (18 May 1962). "Tauter & Tauter". Time. Retrieved May 28, 2008. Just two days before Holleman confirmed that he had asked Estes and other Texans to pick up the tab for a January dinner Labor Secretary Arthur Goldberg gave for Lyndon Johnson, but said he backed off when he learned that Goldberg's policy was to pay for all such dinners himself. Goldberg promptly offered to produce canceled checks to prove he had paid for the dinner. Said Holleman of Billie Sol, in words reminiscent of a previous Democratic Administration: "I have not and I never will deny him as a friend."

^Along with Goldberg, Abe Fortas was the only other Warren era justice to later argue a case before that body. Dan Levitt, quoted in Brad Snyder, Curt Flood's Fight for Free Agency in Professional Sports, p. 281