My abiding image of the photographer Pieter Hugo comes from the Arles Photography Festival which finished last week. It was around two in the morning and the hotel bar had just shut but Hugo was in full flow, holding forth - intense, animated, provocative - about the imminent death of photography.

'It's over, it's finished,' he declared, gesticulating with his beer bottle. 'If you really want to know about anything - a war, a place, a person - you go read a book, right? You don't look at a photograph. It's just a moment, a glimpse. It's just a bloody photograph.'

Two nights later, Hugo bounds down the steps of the packed Roman amphitheatre in Arles to grab the prestigious Discovery award for his extraordinary images of the Hyena Men of Nigeria and the wild honey collectors of Ghana. The death of photography, it seems, has been postponed. When I rang him in Cape Town to congratulate him, I couldn't help mentioning his monologue. Does he really believe photography is, as he put it, 'incapable of telling us anything truly meaningful'? He laughs.

'I get carried away, don't I?' he says. 'But, you know I do think that a lot of the time. I have a deep suspicion of photography, to the point where I do sometimes think it cannot accurately portray anything, really. And, I particularly distrust portrait photography. I mean, do you honestly think a portrait can tell you anything about the subject? And, even if it did, would you trust what it had to say?'

This might seem an extraordinary thing for a photographer to say, especially one whose portraits have caused such a sensation, but Hugo is part of a generation whose work is informed by an understanding of the limitations of their vocation. 'It sounds extreme, but for me to work at all as a photographer, I have to be conscious always of the problems inherent in what I do. I have to be conscious, if you like, of the impossibility of photography.'

Perhaps it is this rigorous interrogation of his vocation that makes Pieter Hugo's images so powerful. In 2006, he won first prize in the Portraits section of the World Press Photo. At Arles, even before he won the Discovery award, Hugo's arresting and unsettling photographs of the Hyena Men were the talk of the town.

'It was a public vote and I don't think it was a big surprise to anyone that Pieter won hands down,' says Elisabeth Biondi, visuals editor of the New Yorker magazine and one of the most influential taste-makers in modern photography, who chose him for the shortlist. 'The images just stop people in their tracks. You can't look away.'

She is right. In the interests of journalistic research, I spent some time lurking in Hugo's exhibition space at Arles, gauging the reaction of the visitors. It would have made a great fly-on-the-wall documentary: the open mouths, the intent stares, the heads shaking in disbelief, the occasional spasm of horror. It was verification that photography still has the power to disturb even in this image-saturated era.

Hugo is one of a new generation of savvy young photographers who have emerged from post-apartheid South Africa with work that challenges our preconceptions about their country. Alongside the likes of Guy Tillim and the young Magnum photographer Mikhael Subotsky, Hugo represents what might be called a new photographic consciousness as regards the representation of Africa to the West.

He considers himself 'a political-with-a-small-p photographer... it's hard not to be as soon as you pick up a camera in South Africa'. He cites the work of veteran South African photojournalist David Goldblatt as an influence, but also Boris Mikhailov's edgy, provocative images of Russia's alcoholics and down-and-outs. Hugo's work spans, sometimes uneasily, these two aesthetics, the one campaigning, the other shocking in its graphic depiction of transgressive subject matter.

'I matriculated at the end of apartheid,' he says, 'and the photographs I grew up looking at were directly political in that they attempted to reveal, or change, what was happening. Back then, the lines were clear. You tried to tell the world what was going on with your photographs. It's much more complex now. I am of a generation that approaches photography with a keen awareness of the problems inherent in pointing a camera at anything.'

Besides his best-known work, Hugo has turned his camera on Aids victims in their coffins in his native land and sites of mass slaughter in Rwanda, as well as South African football supporters in animal masks and faith healers in trances. He is drawn, he says, to 'that which we do not want to look at, be it the old or the terminally ill or the marginalised'.

His images of South Africans with albinism, whose ultra-paleness sets them apart in a country where skin colour is inextricably linked with identity and belonging, are stark and, some might say, pitiless. Likewise, his portraits of poor, white South African families. He is currently working on a series about 'Nollywood', the burgeoning Nigerian film industry, which, even at its most graphic comes as a kind of light relief, and suggests that he may be giving himself, as well as the viewer, a break from the confrontational.

Hugo grew up in a bohemian family in Cape Town, and is self-taught, having picked up a camera aged 10. He remembers the first image he printed, which was of a homeless person. You sense that his work is always somehow informed by his problems of belonging.

'My homeland is Africa, but I'm white,' he says. 'I feel African, whatever that means, but if you ask anyone in South Africa if I'm African, they will almost certainly say no. I don't fit into the social topography of my country and that certainly fuelled why I became a photographer.'

When Hugo's book The Hyena & Other Men was published last year, it became that rare thing, a photography book that flew of the shelves on word-of-mouth reputation. It sold out in a matter of weeks and is already into its third print run. He had first heard of the Hyena Men in 2005 when, while working on a photographic project in South Africa, he received an email from a friend. It was a snapshot, taken through a car window in Lagos, of a group of men walking down a street led by a huge hyena on a chain. The image stayed in his head, and, as he puts it, 'kept niggling at my curiosity'.

A few days later, Hugo was surprised to find the same photograph in a South African newspaper where it was used to illustrate the reported lawlessness that pervaded Nigerian cities. The Hyena Men, or 'Gadawan Kuru', were described in lurid tones as criminals, drug dealers and debt enforcers. The hyena, a creature both feared and mythologised in urban as well as rural areas of Nigeria, was supposedly used to inspire fear in their victims, and to give the men themselves an aura of mystery and almost supernatural power.

'Many people in Nigeria still believe hyenas are witches,' says Hugo, 'or that they are humans who have been reincarnated as wild beasts to cause mischief and havoc in the night. There's also the fact that the African hyena just looks kind of otherworldly.'

A few days after receiving that email, Hugo set out for Nigeria to track down the Hyena Men of the Hausa people. It was not an easy task. Like his native South Africa, Nigeria is a violent, anarchic place. The troupe was constantly on the move, surviving on the fringes of society, and continually harassed by the police. Hugo was a towering white man with a South African accent, an outsider and a perceived authority figure. Having tracked them down to Benin, he arrived to find that they had moved on in the night. Then, driving around the edge of the capital, Abuja, at dusk, he caught sight of them for the first time.

'We found them on the periphery of the city in a shantytown - a group of men, a little girl, three hyenas, four monkeys and a few rock pythons,' he writes in his illuminating introduction to the book, 'a group of itinerant minstrels, performers who used animals to entertain crowds and sell traditional medicines. The animal handlers were all related to each other and were practising a tradition passed down from generation to generation.'

Hugo initially spent two weeks travelling with the Hyena Men but went back to Cape Town 'dissatisfied' with the photographs he had taken. He returned to Nigeria two years later and spent another few weeks with them. This time the relationship between photographer and subject deepened into a kind of trust.

'By then I had gotten to know the men, and they were eager to pose, but I was always wary of the animals,' he says. 'Up close, a hyena is an almost overpowering presence. It is a wild animal however much it has been subdued. I never felt comfortable being close to them, and their effect on local people was so powerful that you could easily start to believe some of the mythology.'

At times his task took a surreal turn. Once, he found himself hiding in some bushes by the roadside with another man, a couple of baboons, a hyena and a pair of pythons, while another handler haggled with a passing taxi driver. When the fare was agreed, the strange crew jumped out of the bushes and piled into the taxi. 'The taxi driver was horrified', Hugo writes in his introduction. 'I sat up front with a monkey and the driver. He drove like an absolute maniac. At one stage the monkey was terrified by his driving. It grabbed hold of my leg and stared into my eyes. I could see its fear.'

Most of the time, though, the Hyena Men simply strolled into a new town, led by a hyena on a long leash, and created their own carnival. Adetokunbo Abiola, the Nigerian journalist who acted as Hugo's intermediary in Nigeria, later described their impact on the local populace. 'Commercial buses and private cars stop, causing a traffic jam, while passengers gape at the animals as they perform their tricks. Within seconds people start to gather and a crowd forms, everyone staring in wonder.'

For Pieter Hugo, the Hyena Men were a 'complex spectacle' that had to be recorded, but he soon realised that, photographically at least, the crowds they attracted were a distraction. His real interest was the relationship between the men and their captive animals. He began photographing the Hyena Men before and after their public performances, often on relatively deserted streets at dawn or dusk, and sometimes in big urban spaces - beneath vaulting flyovers, in wastegrounds full of discarded vehicles - where their surreally formal poses seem even more strange. In the process, he created a body of work that sits somewhere between documentary photography, art-photography and portraiture.

'Hugo eschews wide-angle and close-up shots,' writes South African art critic Bronwyn Law-Viljoen in an essay for Hugo's book. 'By giving the men and their animals space and setting them against the desaturated sky and white dust of the landscape he directs an understated drama that poses more questions than it answers.'

One of the questions that crops up, particularly in the West, concerns the welfare of the hyenas and the baboons. 'They were not treated well and it started to get to me by the end of my time with the handlers,' says Hugo. 'There were a lot of big sticks and they were used when the animals did not do what they were supposed to. The hyenas are so big they seemed to absorb even the strongest blows to the head but the monkeys seemed terrified at times. When I asked people about the cruel treatment of the animals in Nigeria, they just looked at me as if I was mad. It simply isn't an issue there.'

And what about the charge that, by taking photographs that are set up to a great degree, he is exploiting his subjects for their 'exotic' otherness?

'I reject that view utterly,' he says, suddenly angry. 'There's always an element of condescension in it, the notion that the people I photograph are somehow not capable of making their minds up about being photographed. And, you know, it always comes from white, liberal, European people, which suggests to me that there is something essentially colonial about the question itself.'

He takes a deep breath. 'Look, there is always permission when I take a photograph, and there is always an exchange, emotional or financial. I paid these guys because I was taking up time when they could have been working or travelling.' So, what did his subjects make of the photographs? 'They were happy. They're performers. They loved the camera.'

What is it exactly that makes these images so arresting? Perhaps it has to do with their contradictions. On one level, they are studies in extreme masculinity; on another, they show once-wild animals made vulnerable through their subjugation. The presence of monkeys, often posing as if human, surely stirs something in our subconscious. Likewise, in a different way, the presence of the little girl who rides, or, in one instance, hugs, an unmuzzled hyena as if it were a pet pony. So many issues reverberate here that, as Elisabeth Biondi says, 'the urge is to not look away however much we want to'.

For all his talk about the death of portaiture, Hugo's photographs are meticulously set up. In divorcing the Hyena Men and their beasts from their performance context, he accentuates rather than reduces their otherness. He also makes the viewer consider the deeper questions around the images - ideas about man and nature, about the wild and the tamed and about what he describes as 'the liminal spaces' that exists on the vast edges of cities where outsider communities survive however they can.

'What's really interesting to me, as a white South African who doesn't really fit into the social topography of his own county, is the psychological state these guys operate in,' says Hugo. 'They exist and survive in a place that is neither here nor there. They are transgressive but what they do is either allowed or overlooked by the state. Their way of life is almost medieval, and they survive like this in the most oil-rich country in Africa. Sure, the hyenas and the baboons are dramatic but it's all the big questions that hover over the images that I'm interested in.'

Like Britain's Richard Billingham, another self-taught photographer who has struggled to make work as powerful as his early images of his dysfunctional family, Pieter Hugo may find the Hyena Men series a hard act to follow.

'Some people have said to me that Pieter's subject is so dramatic that it would be hard to take a bad picture,' says Biondi, 'but, you know, a photographer chooses his subjects, and that, too, is an important part of having a great eye. Photographers go where their instinct leads them and then try and work out their fascination for the subject through the photographs they take. That's what Pieter's doing but in a kind of extreme way.' She pauses for a moment. 'He has a vision and he pursues it relentlessly. He has what it takes.'