Baldwin considered resigning in 1831. In a letter to President Jackson, he complained about the Court’s extension of its powers. Some historians believe that Baldwin suffered from mental illness during this period. However, he continued to serve on the court until his death in Philadelphia, Pennsylvania.

Justice Baldwin was personally involved in cases deciding the issue of slavery. In the case of Johnson v. Tompkins, 13 F. Cas. 840 (C.C.E.D. Pa. 1833), he instructed the jury that although slavery's existence "is abhorrent to all our ideas of natural right and justice," the jury must respect the legal status of slavery. He was the sole dissenter in the case United States v. The Amistad, in which Associate Justice Joseph Story delivered the Court's decision to free the 36 kidnapped African adults and children who were on board the schooner, La Amistad. In Groves v. Slaughter, 40 U.S. (15 Pet.) 449 (1841), Justice Baldwin emphatically expressed his opinion that, as a matter of constitutional law, slaves are property, not persons.

In another federal case, Justice Baldwin interpreted the Privileges and Immunities Clause of the Constitution. That case was Magill v. Brown, 16 Fed. Cas. 408 (C.C.E.D. Pa. 1833), in which Justice Baldwin stated: "We must take it therefore as a grant by the people of the state in convention, to the citizens of all the other states of the Union, of the privileges and immunities of the citizens of this state." This eventually became the view accepted by the Supreme Court, and remains so. He also interpreted the Clause that way, in dictum, when speaking for the Court in Rhode Island v. Massachusetts, 37 U.S. (12 Pet.) 657, 751 (1838) (each State "by the Constitution has agreed that those of any other state shall enjoy rights, privileges, and immunities in each, as its own do").

Justice Baldwin was a friend and admirer of Chief Justice John Marshall, and wrote of Marshall that "no commentator ever followed the text more faithfully, or ever made a commentary more accordant with its strict intention and language." Baldwin was at Marshall's bedside when the old Chief Justice died in 1835.

Justice Baldwin suffered from paralysis in later years and died a pauper, aged 64. Historian William J. Novak of the University of Chicago has written that, "Baldwin’s jurisprudence has been treated rather shabbily by historians."[1]

Baldwin, Henry (2011). A General View Of The Origin And Nature Of The Constitution And Government Of The United States, Deduced From The Political History And Condition Of The Colonies And States, From 1774 Until 1788. ISBN9781446061398.