The town was originally named Mortlach (Gaelic:Mòrthlach). In the 19th century, James Duff, 4th Earl Fife, established it as housing for soldiers returning home from war and renamed it Dufftown.

In the Middle Ages, Mortlach (in Latin Murthlacum) was an episcopal see.

Dufftown hosts a highland games event every July which attracts many tourists.

Dufftown's notable buildings include Balvenie Castle, which was partly built with stones from the ruins of nearby Auchindoun Castle. The street plan of Dufftown is mainly four main roads meeting at the clock tower; previously a prison but now a tourist information centre.

Dufftown has an oceanic climate typical of Northern Scotland, characterised by relatively mild, damp winters and cool cloudy summers. The area averages 76 air frosts a year. Like much of Europe, Dufftown's climate is insulated somewhat by the Gulf Stream, a warm ocean current originating near the gulf of Mexico. This warm ocean current makes Dufftown's climate significantly milder during the winter than expected for its latitude.