The president of the United States was both a racist and a very difficult man to get along with.

He routinely called blacks inferior. He bluntly stated that no matter how much progress they made, they must remain so. He openly called critics disloyal, even treasonous. He liberally threw insults like candy during public speeches. He rudely ignored answers he didn’t like. He regularly put other people into positions they didn’t want to be in, then blamed them when things went sour. His own bodyguard later called him “destined to conflict,” a man who “found it impossible to conciliate or temporize.”

But the nation’s politicians simply had to interact with Andrew Johnson, for he had become the legitimate, constitutionally ordained chief executive upon Abraham Lincoln’s death by assassination.

Their path for managing this choleric man reveals that a president need not be kicked out of office to be removed from holding a firm grip on the reins of power. It also shows that people around the president, from Congress to the Cabinet, have many more tools at their disposal than, say, writing an anonymous New York Times op-ed to stop a leader they consider reckless or dangerous.

This is true even thoughJohnson’s vice presidency remains historically unique. For his 1864 reelection bid, Lincoln had dumped his first-term vice president, Hannibal Hamlin. To appeal to non-Republicans and show he wasn’t just a Northern leader in the middle of the Civil War, the president instead ran on a new “National Union” ticket. He picked Johnson, a lifelong Democrat from Tennessee who had been the only senator out of 11 Southern states to remain with the Union in 1861 instead of walking out of the Senate and leaving a vacant seat in protest.

Johnson did the noble thing and kept himself mostly out of the public eye for the next 10 days to let the scandal subside.

But Johnson turned out to be a poor choice, and the new vice president couldn’t have started his term much worse. Feeling ill, Johnson threw down three glasses of whiskey right before his swearing-in ceremony and inaugural speech. “I need all the strength I can get,” he told Hamlin,who was there to hand off the office Johnson would soon assume.

The audience noticed, and not just because Johnson’s face had turned bright red and his planned five-minute address stretched to three times longer. Shouting, gesticulating wildly, stumbling over his words and shaking his fists, he went into stump-speech mode, declaring violently that he was a man of the people and that Tennessee had never left the Union. Hamlin tried to shut him up and pull him away but failed in both. Johnson stammered and had to ask assembled officials nearby who the secretary of the Navy was. During the spectacle, the attorney general leaned over and called it “a wretched mess” to Gideon Welles — who happened to be the secretary of the Navy — who in turn said, “Johnson is either drunk or crazy.”

Johnson finally stopped his meandering and allowed Hamlin to administer the oath of office. Unfortunately, he bungled that, too, stumbling through the words and adding his own befuddled commentary along the way. After putting his lips to the Bible he’d just sworn on and yelling, “I kiss this Book in the face of my nation of the United States!” officials moved him on and asked someone else, on his behalf, to perform the new vice president’s traditional duty and administer oaths to the new senators. Lincoln took it all in stride, denying that his new vice president was a drunkard while acknowledging his “bad slip.” Johnson did the noble thing and kept himself mostly out of the public eye for the next 10 days to let the scandal subside.

That, combined with the tradition of the time that presidents didn’t consider their vice presidents part of the inner circle, meant Johnson and Lincoln didn’t interact much in the six weeks between inauguration and Lincoln’s assassination on April 14. Johnson thus started his time in office without a strong sense of exactly how Lincoln planned to ensure the final surrender of all Confederate forces in the Civil War and rebuild the war-torn nation.

Although Congress had already put in place some features of the post-war period that would serve as flash points with the new administration — like the Freedmen’s Bureau, charged primarily with feeding and caring for former slaves — Johnson came to the top job with a very different conception of post-war reunification. He questioned the federal government’s right to do much of anything in the formerly rebellious states until they had representatives back in Congress, even if those officials were former Confederates.His vision, of course, clashed with the so-called Radical Republicans in Congress, intent on reconstructing the South from Washington in order to guarantee the freedoms of those who had been enslaved for so long.

Johnson rankled most legislators, and the vast majority of Northerners, almost immediately. He released leading members of the Confederate Cabinet from government custody, up to and including the former Confederate vice president, Alexander H. Stephens.He appointed governors in Southern states and allowed their legislatures to meet. Dominated by secessionists, these governments passed “black codes,” allowing slavery in all but name to continue in many areas.

Johnson also made his racist views clear in statements like this one to the federal commissioner of the Public Buildings Service: “Everyone would, and must admit, that the white race was superior to the black, and that while we ought to do our best to bring them ... up to our present level, that, in doing so, we should, at the same time raise our own intellectual status so that the relative position of the two races would be the same.”

“Is there no way,” declared leading radical senator Thaddeus Stevens of Pennsylvania just months after Johnson’s inauguration, “to arrest the insane course of the president in Washington?” He even mused that by taking actions more properly lying with Congress — like treating Southern states as legitimate entities, even as most legislators considered them still occupied territory — thenew president was setting the stage to be “crowned king.” Leading legislators urged the president to call Congress into a special session, or at least delay controversial moves until it was scheduled to convene in December. Johnson obstinately ignored them. By the winter of 1865–1866, the president had proved himself “already more disposed to be the political partisan of the Southerners than the ally of those who had elected him,” according to Adam Badeau, a confidant of Army General Ulysses Grant.

This stubbornness and refusal to cooperate with even moderate Republicans escalated once Congress came back into session in December of 1865,still without Southern representation and still dominated by Republicans steadfastly opposed to the leniency the new president was offering to the former Confederate states. Johnson vetoed both a civil rights bill designed to fight back the dreaded black codes and another measure to expand the functions of the Freedmen’s Bureau. His message to Congress about the latter veto included condescending language, like urging legislators to take “more mature considerations.” The vetoes enraged Capitol Hill, especially the author of the bills, to whom Johnson had raised no objections when he’d sought the president’s opinions during the drafting process.

The legislative branch, as a consequence, did something that was then unprecedented in American history on a major piece of legislation: They overturned a presidential veto. Then they did it again. Ultimately, they turned back the president’s rejections of bills a stunning 15 times — a record to this day, even though Johnson served a shorter term than most presidents. The Civil Rights Act’s veto override in the House prompted a spontaneous outburst of applause among both representatives and spectators; the speaker found it impossible to restore order for several minutes.

From summer 1866 on, both General Grant and War Secretary Edwin Stanton were resisting — often all but openly defying — Johnson’s orders from within the executive branch.

Also in early 1866, a congressional joint committee on reconstruction developed a constitutional amendment, which presidents have no power to either approve or deny. It sought to prohibit states from depriving citizens of fundamental rights or equal protection under the law and to rescind the constitutional formula by which states had gained the benefit of additional representation in Congress for slaves within their borders, without letting those slaves vote. Both Houses of Congress passed it in June, but behind the scenes Johnson obstructed its ratification. The measure would ultimately become the Fourteenth Amendment in 1868.

The president also saw his judicial appointment powers curtailed. When a Supreme Court vacancy came up, Congress eliminated the seat rather than confirm Johnson’s nominee. As a hedge against a potential future Johnson appointment, they went ahead and legislated in advance that the next high court vacancy, too, would not require filling.

In August and September of 1866, Johnson tried to rally public support around him in a multistate 19-day road trip, during which he gave more than 100 speeches. Typically frosty audiences greeted the president, often drawing him into unseemly shouting matches or forcing him to cut short his visit. In Bloomington, Illinois, one heckler yelled that traitors weren’t welcome in the land of Lincoln; the ensuing uproar made it impossible for Johnson to complete his planned speech.

“For the first time in the history of our country,” wrote the New York Independent, “the people have been witness to the mortifying spectacle of the president going from town to town, accompanied by the prominent members of the Cabinet, on an electioneering raid, denouncing his opponents, bandying epithets with men in the crowd, and praising himself and his policies. Such a humiliating exhibition has never before been seen, nor anything even approaching to it.”

Pushback wasn’t just coming from Congress and rally crowds. From summer 1866 on, both General Grant and War Secretary Edwin Stanton were resisting — often all but openly defying — Johnson’s orders from within the executive branch.

The president had declared the Southern rebellion over in 1866, seemingly ending the army’s primacy over local law enforcement there. ButGrant later sent confidential instructions that commanders should continue martial law as needed and resist any presidential attempt to curtail the Freedmen’s Bureau’s actions. In October, Grant twice refused Johnson’s request that he join a diplomatic delegation to Mexico — the president’s ploy to get the war hero out of the way. Then, in a full Cabinet meeting, Johnson pretended that Grant had never objected to going to Mexico by having the secretary of state read to the general detailed instructions for the diplomatic mission. When Johnson condescendingly asked the attorney general to lecture Grant on the duty to obey presidential orders, the general stood and declared, “I am an American citizen, and eligible to any office to which any American is eligible. I am an officer of the Army, and bound to obey your military orders. But this is a civil office, a purely diplomatic duty that you offer me, and I cannot be compelled to undertake it. … No power on Earth can compel me to do it.”

Grant feared being away from Washington if, as he thought increasingly likely, the beleaguered president were to decide that disbanding Congress and using force to take total control of the government offered him the best way through the impasse. Badeau recorded that the general, while looking to the general public like a faithful follower of Johnson and his policies, “was in reality doing more than all the country besides to thwart Johnson’s designs.”

In this Harper's Weekly cartoon, Johnson is depicted as Mercutio, with War Secretary Edwin Stanton as Romeo | Wikimedia Commons

Stanton, despite his steadfast opposition to the president’s approach, stubbornly remained in Johnson’s Cabinet because he felt he could delay or even prevent some of Johnson’s worst actions. The secretary favored Reconstruction legislation in early 1867 that provided for military government in the South, which was passed over the president’s veto. He watched as successive bills from Capitol Hill received presidential vetoes, but became law anyway over Johnson’s objections. “The situation was unprecedented in the history of the country,” wrote Badeau. “A Cabinet Minister and the General of the Army were doing their utmost to thwart the President. . . . They then more than once discussed the means by which they too could apparently obey the directions of a superior and yet neutralize his intent and purpose.”

Particularly distressing to Grant, Stanton and many others around them was the increasing violence in the South between emboldened former Confederates and former slaves asserting their rights. Already by the end of 1866, the president “became, if not treasonable in intent, yet unpatriotic in action,” Badeau noted, probably representing Grant’s views. “He fostered a spirit that engendered massacre, and afterward protected the evil-doers. He spoke, both with Grant in private and openly to the public, as if the Congress elected by the faithful States was an illegal body. He suggested to men’s minds that he might be plotting to allow the Southerners to return to their places in spite of the North.”

Johnson faced a dilemma. He couldn’t easily get rid of Grant, the most popular living American, who had won respect even from Southerners for his gracious treatment of Robert E. Lee after the Confederate general’s surrender at Appomattox Court House in 1865. Plus, Congress wrote language into the military appropriations bill for 1867-1868 that both denied the president’s right to directly control the military — all orders had to go through the general of the army (Grant) — and prevented Grant’s demotion without the Senate’s consent. Stanton also gained protection from Congress in the form of the Tenure of Office Act of February 1867, a constitutionally unsound measure that nevertheless prohibited presidential removal of certain executive branch officers, including Stanton, unless the Senate agreed.

Johnson’s veto of the Tenure of Office Act got the usual treatment from Congress: a prompt override. “He is of no account,” one senator said bluntly. “We pay no attention anymore to what he says.” Virginia’s Republican governor wrote to a prominent congressman, “I fear there will be no peace in the country as long as Johnson is in the Presidential Chair.” Secretary of the Navy Gideon Welles reported in June 1867 that the president was “nervous and apprehensive,” all but trapped in the White House “in constant dread of impeachment.”

But Johnson just couldn’t leave things well enough alone.

After suspending Stanton until Congress reconvened, the president ended up firing his war secretary outright in February 1868. Stanton refused to leave his office — literally moving in and hunkering down, day and night, for the duration of the crisis — giving representatives the excuse they’d been hoping for to try to kick the president out of office. Johnson that same month became the first president to be impeached by the House.

The failure of the Radical Republicans to convict him in the Senate (by one vote) and thus remove him from office didn’t stop Congress from keeping Johnson boxed in. He remained something short of a full chief executive during his final 10 months in office, with effective restrictions on his power, like the Tenure of Office Act, locked in. General Grant, by this time a candidate for the presidential election that November, believed that “Johnson had been taught a lesson which he would not forget.” Johnson’s leading biographer Hans Trefousse calls him, for the remainder of his time as chief executive, a “president in limbo.”

Why wait until the end of a term to remove a president?

Before the trial, Congress had passed a new Reconstruction bill, which became law when Johnson didn’t even bother to veto it. He did, however, issue a veto against a bill denying appeal rights in some cases, but Congress overrode that. Legislators in June also rejected new presidential vetoes, on the very days they were issued, against bills readmitting several Southern states with new constitutions. The following month it happened again on bills excluding electoral votes of states that had not yet been reorganized and extending the Freedmen’s Bureau another year.

“Somehow I expected that there would be a change in Mr. Johnson’s position after his victory over the Radicals,” wrote the president’s bodyguard, William Crook, who had an inside view of the whole situation. “If I had thought of it, I might have realized that the two-thirds majority was still against him. The only difference was that when they passed measures over the President’s veto it was without debate. There was no longer the need for discussion.”

Senators also turned down Johnson’s nominees for lesser diplomatic posts so often that, exasperated, he announced he would only put forward a prospective nominee able to prove to him in advance that confirmation would come.

One power he still held was that of the federal pardon. And pardon he did, issuing many more of them than all other presidents to that point, combined — overwhelmingly for those who participated in the rebellion against the Union. Johnson even pardoned a few of the men convicted in the conspiracy to kill President Lincoln. He anticipated his old party would show some appreciation for the pain he had caused to Republicans by giving him the presidential spot on the Democratic ticket for the coming election against Grant. The party convention dashed those hopes by instead choosing Horatio Seymour, a man who didn’t even want the nomination.

Why wait until the end of a term to remove a president? Methods ranging from the deft to the downright unsavory have undermined presidents’ authority, seen so clearly in Andrew Johnson’s case. Most of the same mechanisms used to undermine him remain in others’ toolkits today, which means it’s equally true now as it was under Johnson: You don’t have to formally eject an unpopular or unfit president from the White House if you can use various other means to limit the damage he is causing to the country.

David Priess is an author and speaker. He is the author of "The President's Book of Secrets: The Untold Story of Intelligence Briefings to America's Presidents," and "How to Get Rid of a President," from which this is adapted.