Baker has also borne the names New Nantucket and Phoebe. Whaling vessels prequently visited the island, in the early 19th century, using it as a mail drop-off and provisioning station. In 1832, Michael Baker of New Bedford, Massachusetts, named it "Baker."

Baker was exploited for commercial guano harvesting during the 19th century. "Guano" is bird droppings, which are used as fertilizer. The American Guano Company began mining guano there in 1858, and mining peaked in the 1860s and 1870s. An estimated 200,000 to 240,000 tons of guano were removed. Large basins from mining excavation and mounds of low-grade guano still mark the island landscape. After 17 years of guano mining, Baker's guano deposits were exhausted, and the islnad appears to have been uninhabited until the 1930s.

The establishment of trans-Pacific air routes, territorial disputes between the United States and the United Kingdom, and the threat of a second world war led to colonizing efforts by the United States in the 1930s. In 1935, several military personnel and graduates of Kamehameha Schools, Hawaii, established a colony on Baker. The colonists erected Meyerton as a permanent settlement, with structures for water, food storage, radio equipment, and walls around the main settlement. During this era, a beacon was constructed from the stone slabs from the guano mining houses.

The colony continued undisturbed until December 1941, when Japanese air and naval attacks destroyed all the buildings except the shed and beacon. The colonists were not injured, and they were removed in January 1942.

In July 1943, U.S. forces reoccupied Baker and built an Army air base. About 2,000 to 3,000 men were stationed on the island during peak operations. The Army Air Force abandoned the island was evacuated in March 1944, but in June, the U.S. Coast Guard began construction of a LORAN station. The Coast Guard administered the island until it was abandoned in May 1946. Today, the World War II runway is completely covered with vegetation and unusable.

Baker Island National Wildlife Refuge was established on June 27, 1974. Recent issues include abandoned World War II military debris and trespass by commercial fishing boats.