From hideous hags feeding on babies' blood to silky seductresses weaving spells to bewilder menfolk, artists' fantasies of witches have produced some of the most extreme and blackly fantastical images in art history.

This summer, witches will be the subject of a major museum exhibition for the first time. The Scottish National Gallery of Modern Art in Edinburgh is bringing together over 80 drawings, prints and paintings of witches – from Albrecht Dürer's 1501 engraving of a shrieking hag flying backwards on a lustful goat to Paula Rego, Cindy Sherman and Kiki Smith's deft reappropriations of the idea of the witch for a feminist age.

The exhibition will be arranged thematically, examining the tropes in witch-depiction as they cross the centuries: one room, for example, will examine the idea of the witches' sabbath, including Salvator Rosa's 17th-century painting from the National Gallery in London, Witches at their Incantations. The work depicts a complex scene in which one witch squeezes the blood from a human organ, another clips the toenails from a hanged corpse and a hag presents a swaddled baby as a sacrificial victim.

Another section will look at the notion of the airborne witch – such as the 1762 engraving by Paul Sandby, showing a witch carrying two tatterdemalion men on an extremely phallic broomstick, satirising the subject of emigration from Scotland to England.

Some of the images are less crudely salacious and cruel, but still carry more than a whiff of danger: the pre-Raphaelite Frederick Sandys, for example, frequently depicted his Gypsy mistress and model Keomi Gray as a witch. Looking back to Euripides and other classical models, he painted her as the beautiful but terrifying Medea, the Colchian princess who helps Jason to steal the golden fleece but then concocts a magical poison to destroy her love rival, Glauce.

An engraving from 1762 by Paul Sandby, satirising Scottish emigration to England. Photograph: National Galleries of Scotland

"There is a sense that there are two kinds of witches," says co-curator Patricia Allerston. "The hag who is the opposite to the perfect Venus or Madonna, and who has her badness almost written on her body. Then there is the beautiful but dangerous witch. We are, of course, for most of the period of the show, talking about the male gaze."

The exhibition also shows how the iconography has developed from the mid-15th century, when a growing spirit of church reform was defining the boundaries of acceptable religious belief and practice, and when fear of, and intellectual interest in, witches burgeoned.

An important text was the 1486 Malleus Maleficarum (Hammer of Witches); such treatises had wide circulation thanks to the cutting-edge technology of the printing press, and, said Allerston, had a strong impact on the way artists imagined witches.

"It's important to remember that people absolutely did believe in witches," said Allerston. "They were considered a real danger" – so much so that James VI of Scotland, soon to become James I of England, produced his own treatise, Daemonologie, warning against the "fearefull aboundinge at this time in this countrie, of these detestable slaues of the Deuill".

The birth of the gothic and romantic tendencies in literary culture, however, saw a less gruesome attitude adopted. The three witches of Macbeth became a popular theme for artists, and the exhibition will include an intriguing 1775 pastel portrait by Daniel Gardner of three politically significant aristocratic women – Georgiana, Duchess of Devonshire, Elizabeth Lamb, Viscountess Melbourne, and Anne Seymour Damer – in the guise of Shakespeare's weird sisters. Impeccably dressed and elegant, they hurl flowers, rather than eye of newt and toe of frog, into their cauldron; and it is probable that the scene was painted at Lamb's behest, perhaps to illustrate some in-joke.

Another late 18th- or early 19th-century depiction of the witches from Macbeth has proved something of an art-historical mystery. The crude but striking work was made by scratching into thick paint – creating a graphic and expressionist effect that looks oddly modern. It was spotted by a curator on the Public Catalogue Foundation website , which is amassing a list of all works of art in the public domain. Its artist is unknown, and the work which usually hangs in a fellow's study at St John's College, Cambridge; is thought to have been brought there by the artist George Romney's son, John, making a tantalising link with one of the 18th century's great artists.

It is only in the later 20th century that the visual idea of the witch was thoroughly clawed back by women artists. A photograph by Cindy Sherman shows her appropriating and pastiching all the most characteristic visual tropes of witches, such as warty chin and huge hooked nose. The most recent image in the exhibition is by Markéta Luskacová – a compellingly odd photograph of a woman dressed in a carnival costume, taken in 2000 in the artist's native Bohemia. Dressed in a cat mask and clutching a bird head to her midriff, the enigmatic image suggests that the trappings of the occult and the magical still have the power to unnerve.

Witches and Wicked Bodies is at the Scottish National Gallery of Modern Art 2 from 27 July – 3 November