Freud's strange essay of 1919, translated into English as "The Uncanny", casts a shadow longer than the length, or coherence, of the piece seems to promise. It opens with one of his disingenuous disclaimers: "Only rarely does the psychoanalyst feel impelled to engage in aesthetic investigations." Freud's cautioning reservation has been observed too infrequently by his followers; Freud himself, however, is a different matter. He was, or could be, a brilliant stylist, and his fascination with art, and with artists, was as powerful as his preoccupation with sex (he would claim that they are one and the same subject); certainly, it was a good deal more ambivalent.

The dichotomy of the famous statement "poets and philosophers before me discovered the unconscious - what I discovered was the scientific method by which the unconscious can be studied" is characteristically misleading. The pursuit of a scientific status for psychoanalysis was more a bid for academic respectability than a true representation of its radical character, and in adopting the camouflage of a scientific dispassion, Freud muzzled the artist in himself. Some of his most delightful and generative papers - for example, the essays on Michelangelo's Moses and on Leonardo - which proceed under the cover of mere "aesthetic investigations" are flights of prodigiously inventive fancy, whose value lies not in any scientific analysis (by that standard they are quite deranged) but in the brilliant apprehension, and scintillating expression, of art's essential ambiguity.

"The Uncanny" explores this ambiguity. In English, to be canny is to be knowing, and thus "uncanny" has the sense of that which is outside our everyday knowledge or "ken". The word in German, unheimlich, means un-homely; as Freud goes on to explain, heimlich has a more fundamental meaning of hidden, secret or concealed. Thus the uncanny has something to do both with unknowing, and with the other side of knowing, or what is known; what the known conceals. That "un" is meat and drink to Freud because it indicates what is for him the nature of the unconscious: its Janus-like capacity to point in opposite directions. That which is most unsettling and perturbing is the other face of what is best known and closest to home.

Freud's essay recapitulates, and then debates, a 1906 paper by Ernst Jentsch (according to Freud the "unrivalled master of the uncanny in literature") which cites, as an example of uncanny effects, a story by the German writer ETA Hoffman called "The Sandman". In this tale a child becomes terrified at his nurse's description of the sandman, who will steal his eyes and carry him off to the moon. As a young man, he falls under the spell of a beautiful doll whose eyes have been put in place by a terrifying figure whom he takes - or mistakes - for the sand-man. The use of this story as one of Freud's case studies inspired the editors of The New Uncanny to offer a number of modern writers the chance to read Freud's essay and produce their own uncanny stories.

A part of the expressed purpose of this "experiment" was to see how far Jentsch's list of attributes, quoted by Freud, would be matched in the efforts of the contemporary authors. One problem with this idea is that the list itself inevitably imposes a mould. The subjects covered by the stories in this collection do reflect some of the preoccupations quoted by Freud - dolls and automata, sleepwalking, doubles, eyes and blindness - but the trouble is we cannot know if this is a consequence of reading the essay or a confirmation of Jentsch and Freud's theories.

This would not matter if the effect, in many of the stories, were not of something too self-consciously predetermined and pinned down, an absence of the sudden hole in the middle of the story through which we fall, or through which the unexpected startlingly creeps or blasts. What characterises the most successful uncanny stories (Kipling's "They", for example, or Penelope Fitzgerald's "Desideratus") is not the subject matter but the sensibility of the writer which perceives, usually rather quietly, another order (or disorder) within the quotidian world, which often seems to emerge as a revelation as much to the writer as to the reader.

Many of these stories are accomplished; some, such as AS Byatt's horribly creepy doll story, or Frank Cottrell Boyce's unnerving moral tale about parental dereliction and the consequent attempts at material compensation, highly so. Adam Marek's sad little fable about the contagious effects of autism is touching. In each of these cases, the writers illustrate Freud's thesis that toys (in our age more technologically developed systems than dolls) act as the recipients of projections which are so powerful as to drain their subjects of life and animate the seemingly inanimate. This is the antithesis of the theme of the double. Either all things have their shadow opposite, or there is only so much vitality available to us and we squander it at our peril.

The other stories were waylaid by the lines laid down by the exercise and tended to confuse the scary or the spooky with the capacity to suggest fissures in the known universe which reveal undisclosed, and thus unnerving but not necessarily negative, aspects of nature - our own included. Only Hanif Kureishi's story of a man meeting his dead father had the deadpan, existentially elusive quality of the truly uncanny. This collection is a bold idea, but the uncanny should not be understood too easily, or grasped too firmly; and it is unlikely that it can be reproduced to commission.

• Salley Vickers's Where Three Roads Meet is published by Canongate. To order The New Uncanny for £7.95 with free UK p&p call Guardian book service on 0870 836 0875 or go to guardian.co.uk/bookshop