The late Susan Sontag summed up all the suspicions that surrounded Diane Arbus in her short lifetime, and continue to echo throughout her work. Arbus photographed, said Sontag, 'people who are pathetic, pitiable, as well as repulsive', and her point of view was 'based on distance, on privilege, on a feeling that what the viewer is asked to look at is really other'.

Arbus, too, seemed sometimes aware of the predatory nature of her photographs, but was unable to resist the call of the transgressive and the taboo, the whole gamut of otherness that she was repeatedly drawn to, and that gives her work its still unsettling power. 'There's some thrill in going to a sideshow,' she once confessed of her obsession with photographing the human oddities who were paraded nightly in the stalls and circus tents of Coney Island in the Fifties and Sixties. 'I felt a mixture of shame and awe.'

Those same mixed feelings attended her nocturnal wanderings among the transvestites, rent boys, bag ladies and bums, as well as her hours spent in hospitals for the mentally disturbed. In all of this, her awe always outweighed her shame until, in her work at least, she became utterly shameless. There is at least one photograph in this big, often disturbing retrospective - of two mentally ill people hurrying through a park at night, both caught unawares, and rendered grotesque, by Arbus's unforgiving lens - that seems like the visual equivalent of a mugging. 'I don't press the shutter,' she once wrote to one of her mentors, the art director Marvin Israel. 'The image does. And it's like being gently clobbered.'

There was nothing, of course, gentle about it. Germaine Greer, whose gypsy portrait appears in this show under the almost dismissive title, Feminist in her hotel room, NYC, 1971, described recently how Arbus bade her lie on a bed and then straddled her, pinning her arms down as she shot frame after frame of her discomfort. The end result, to be fair, is both intimate and beautiful, though the vulnerability it captures may not have fitted Greer's feminist image.

Arbus, too, was vulnerable for all her sublime cruelty with a camera. She was small, shy and unobtrusive, and, when she first started taking pictures, they were too. The first surprise in this big, constantly surprising retrospective, her first in this country for more than 30 years, is the hazy, unfocused nature of her early work. The prints are small, people are shrouded in semi-darkness, places seem indistinct and shadowy; you have to squint to see the fire-eater at work in the sideshow menagerie. This visual coyness did not last long and, at some point in the mid-Fifties, there is a sudden quantum leap in both the clarity of her images and her vision. Her artistic epiphany was inspired by her teacher, the photographer Lisette Model, who, she would later say, 'finally made it clear to me that the more specific you are, the more general it'll be'.

Having separated from her husband, Allan, with whom she worked as a fashion photographer in the early Fifties, Arbus's work, under Model's tutelage, suddenly took flight into new, and strange, territory. Another leap occurred in 1962, when she swapped her Nikon for a larger format camera that produced big, square negatives and finished prints that, in their unsparing detail, made all but the most beautiful subjects look stark and ugly. This is the moment when she finds her signature style, in portrait after portrait, each one edging beyond the intimate towards the invasive, each one compelling in its stark scrutiny of human otherness.

At this point, too, Arbus's life and art converged in their mutual obsessiveness. She haunted the Bowery and the parks of Brooklyn, the fleshpots and fleapits of Times Square, in search of the out-of-the-ordinary, the marginalised, whom she befriended, and in some cases, according to Patricia Bosworth's controversial biography, bedded. Often, her subjects mistook her for one of them, an oddball with a squeaky voice, a strange story to tell and ... a camera. What they thought of their portraits, if indeed they ever saw them, was anyone's guess, but the sheer intimacy of her photographs from this time suggest a level of trust that mitigates the reductive notion of Diane Arbus, photographer of freaks.

Her portraits of human oddities and carnival 'freaks', which constitute only a small portion of her work, never bothered me that much. She worshipped them for their difference, referring to them as 'aristocrats', while they responded in kind. Having been gawked at all their lives by people in search of cheap thrills, Arbus's posed sessions must have seemed like an acknowledgement of their difference, rather than a denigration of it.

On magazine assignments, she often turned her democratically cruel gaze on celebrities, though she was never invited back for a second session. Here, freakishness is conferred by her camera, which is so close-up as to be obtrusive, the ensuing portraits intimate to the point of disturbing. Mae West looks like she has been baked to a human crisp by the sun's relentless glare; James Brown is transformed into a haughty tranny in a kimono; Norman Mailer is caught, splayed across an armchair, in all his cocky, narcissistic maleness. He later said: 'Giving a camera to Diane Arbus is like giving a hand grenade to a baby.'

One of Arbus's most arresting images is of a young boy in Central Park clutching a toy hand grenade, his face and hands contorted in mock outrage. Arbus photographed children a lot, and usually made them seem adult in their expressions of worry, woe and sadness. Babies cry in monstrous close-up. Often, her children look like ungrown adults, just as many of her adults look like malformed children, grotesquely foreshortened by her lens. 'If I stand in front of something,' she said, 'instead of arranging it, I arrange myself.'

Because of her premature death by suicide, Arbus's body of work was relatively small, and what is most intriguing in this show are the images we have not seen before. Some, like Young Man in a trench coat, NYC, 1971, are typically Arbus, illuminated by that otherworldly beauty that many of her portraits of teenagers give off - and nearly all of her portraits of adults do not. Others, like Girl sitting in bed with her boyfriend, NYC, 1966, seem prescient in their intimate everydayness, a template for the kind of low-key, ultra-realist fashion photography of today.

In keeping with the myth of Arbus the doomed seeker of truth, the show proper is punctuated by a series of small, dimly lit ante-chambers, where the faithful can peruse her contact sheets, cameras, diary entries, journals. 'This is just to absolve Esquire of any liability or responsibility for my safety and well-being when I am photographing ...' reads one scrawled note. Photography, it seems, was, for a while at least, Arbus's salvation and, contrary to the received myth, she was often happy to the point of ecstatic when she was engaged in her work. Clinical depression was her undoing. The myth endures, though, made inviolate by the violence of her suicide - barbiturates and a razor blade. So, too, do the suspicions that so exercised Sontag, whose moral rigour was assailed by Arbus's perceived lack of the same. It is difficult still to know how to respond to her late images of the insane and those with learning disabilities, to the people whose otherness obsessed her, but whose trust she could not have won. They did not ask for, nor deserve a clobbering with her camera, however gentle, and their otherness paraded tells us nothing about them, but much about her.

Arbus's willingness to follow where her obsessions led her, unquestioning of the cost to her or her subjects, changed photography forever. Had she not pushed the boundaries of morality and taste, it would be hard to imagine the likes of Nan Goldin or Joel-Peter Witkin or Nobuyoshi Araki following in her footsteps, each in their inimitable ways. She often referred to her body of work as her 'butterfly collection', and no one, before or since, has produced work of such sustained and exquisite cruelty.