The word is an old one. The OED2 attests to nightmare as early as c. 1290. Night is recorded as early as c. 825, but it is such a basic word that it is likely far older than surviving manuscripts. Mare is dated as early as c. 700. Mare also has cognates in many languages, including Dutch, German, French, Polish, and Czech, and ultimately derives from the Indo-European root *mer-, meaning to rub away or to harm. *mer- is also the root of murder and mortal.

Mare is simply an Old English term for a demon. So a nightmare is a demon that visits you at night--a scary dream. A mare was a demon, known as an incubus (male) or succubus (female) that descended on a sleeper, paralyzing and suffocating them, and had sexual relations with the sleeper. Over the centuries the meaning has become generalized to any frightening dream.