Whitfield's accomplishments are polar opposites of Seb Coe's. While Coe set amazing records but almost never won when it was truly important, Whitfield almost never lost but never even came within a second of the then-current World Record (1:46.0 by Germany's Rudolph Harbig in 1939). Whitfield won two Olympic gold medals, with five #1 rankings over a six-year period and 85 wins in 88 finals in his best streak.

Whitfield's off-track accomplishments are little-known but highly important. Kip Keino has called him the "father of African athletics". Almost immediately after his running days were over, he accepted short-term jobs developing athletics in Liberia and then Nigeria, and then spent the remainder of his career in the United States Information Agency where he worked to develop sports in 132 countries, mostly in Africa.

His daughter Fredricka is a CNN anchor. He was the first black athlete to win the Sullivan Award, some seven years after Jackie Robinson broke baseball's color barrier.