In fact, he conveys that precise blend of exhaustion, distraction and authority instantly recognisable from any hospital ward in the world. This should come as no surprise, for he is a senior oncologist – assistant professor of medicine at Columbia University, and staff cancer physician at Columbia University Medical Centre. And yet, until we met it had seemed scarcely possible that the author of The Emperor of All Maladies could really be an actual doctor and not a writer, so exquisitely is his book crafted and paced.

Published a year ago, the Emperor of All Maladies has won the Pulitzer prize for non-fiction, been shortlisted for the National Book Critics Circle award, and named one of the Top 10 Books of the Year by the New York Times, Time magazine and Oprah Winfrey; the sort of success that soars beyond the wildest heights of literary ambition into the stratosphere of fantasy. Yet when Siddhartha Mukherjee talks about his book, it is with a striking air of disinterested detachment. At first I put it down to jet lag. Then I think, no, of course, the poor man must just be so accustomed by now to the carousel of plaudits and prizes and media demands, he has reached the glaze of autopilot. Soon, though, I realise that is not it either. Mukherjee's impression of reluctant ownership of his own success is, I suspect, down to a profound sense of personal insignificance in the face of his subject's enormity.

Mukherjee decided to write a history of cancer when a terminally ill patient asked him a simple question: could he explain exactly "what it is I'm battling?" But as Mukherjee immersed himself in research, the disease quickly began to assume the characteristics of a personality, and so cancer's historian became its biographer. He takes us from the earliest records of cancer in 2,500BC, through medieval theories of black bile and bloodletting, on to the surgical butchery of 19th-century mastectomies, performed with no anaesthetic or penicillin but reckless confidence, before reaching the rollercoaster of 20th-century medical politics, which swung between indifference, euphoria and despair, each wild lurch owing more to socio-economic fashion than to anything resembling solid science. Mukherjee brings every new medical plot twist alive by populating the pages with a vivid cast – patients, physicians, politicians and, at times, himself – whose humanity and fallibility elevate what might have been a dry medical text book into a thriller, thus presenting the publishing world with quite a category challenge.

"I couldn't write a book proposal," Mukherjee explains, "because it was impossible to explain. How do you explain that there'll be a thread of memoir in it, which is small, and then the backdrop, which is much, much larger? One of the things about the book is that the scale shifts very dramatically; you're in a very narrow pinhole" – an individual's story – "and then in something much larger, and you go back and forth. The book lives in its seams, it lives in the connections in the shifts between scale, so it was really like writing a jigsaw puzzle. How do you fit all these moving parts together? You can't explain that in a book proposal." So the doctor had another idea. "I thought: you know what, I'm just going to write the book."

After 250 pages, he showed it to publishers. "Their response was very bipolar, very two-sided. Either publishers said: 'No one's going to read about cancer' – or they said: 'My God, why hasn't this book been written before?'" Some were worried cancer would scare readers off. "To me that was the wrong response, because if people are scared, then that's all the more reason to talk about it."

I confess that when I first heard about the book, it struck me as a marketing stroke of commercial genius – for what reader exists in the world who is unaffected by cancer? "Yes," Mukherjee concedes. "But it's not a feelgood memoir. It's not your plucky, feelgood cancer memoir."

Instead, he wanted to write an intelligent examination of a complex and highly technical subject, and yet accessible enough for a total novice to find it readable. But Mukherjee knew next to nothing about the discipline of writing.

"So I invented rules, such as you won't go through two chapters without meeting a real human character. How does one write the history of the epidemiology of cigarette smoking, for example – which is so abstract, and a story we all know superficially – how can one write that as if it's a discovery, so that you feel it's a discovery? It was very important to me to write this book not as an expert. Because writing anything as an expert is really poisonous to the writing process, because you lose the quality of discovery. So every time I felt I knew something particularly well I tried to unlearn it, and learn new things."

Siddhartha Mukherjee and his wife, the artist Sarah Sze, attend the MoMA Party in New York, May 2011. Photograph: Mehdi Taamallah/Abaca USA/Empics Entertainment

But how did a literary novice, with a full-time job and a young family, teach himself to write so beautifully? "I think the cardinal rule of learning to write is learning to read first. I learned to write by learning to read." He read everyone from Susan Sontag to Primo Levi and Mary Shelley, but wrote the book in 15-minute bursts, snatched from scrag ends of his working day. "But the book was also a conversation going on in my head," he quickly adds, "so I'd write after thinking for five hours."

One of its most arresting observations was inspired by a conversation between Mukherjee and a friend many years earlier "about the nature of interior and exterior", which returned to him as he was working on the book. "Every era," it suddenly struck him, "casts cancer in its own image." The US in the 70s was haunted by cold war fears of the enemy within – and so the "big bomb" was replaced by "the big C". HIV overshadowed the following decade, and then the search for cancer-causing viruses became oncology's new obsession. Now that we're obsessed with genetics, the focus of research has moved on to hereditary causes. "When a disease insinuates itself so potently into the imagination of an era," he writes, "it is often because it impinges on an anxiety latent within that imagination."

If the book has one pre-eminent message, it is simple: "Cancer is not one disease, but many diseases." A silver bullet that could cure all forms is a fallacy, he argues, because every cancer is a different disease, demanding different treatment. We have to stop talking about cancer, and think about cancers.

Other central themes are more nuanced. The "war on cancer" declared by US president Richard Nixon in 1971, a convert to the populist campaign of pressure conducted by a socialite lobbyist called Mary Lasker, was supposed to represent a victory. In fact, Mukherjee argues, it became responsible for propagating the scientifically baseless delusion that government money could cure cancer as easily as it had landed men on the moon. The simplistic hubris of Lasker's slogan – "Let's find a cure for America for its 200th birthday. What a great gift that would be" – draws a wry smile from Mukherjee. "But was it all useless hubris?" he reflects. "Absolutely not. This kind of rhetoric also swept away the cobweb of nihilism that had so deeply surrounded cancer."

He is similarly ambivalent about the impact of Nixon's war on cancer on research methods and funding. Until then, scientists had been largely free to pursue their own theories, however outlandish – but the war on cancer would now be won by bureaucrats demanding goal-orientated research defined by strict parameters. Nixon's war had no place for the colourful, constitutionally ungovernable and haphazardly qualified experimentalists who had dominated the field of cancer research until then, and Mukherjee's respect for those earlier pioneers is palpable throughout his book.

Does a part of him wish he had been born earlier, into the early 20th-century generation working in the wild west of research? "Yes, absolutely. The National Cancer Institute was literally called the wild west of cancer, because they did all these things that would have been impermissible today." The professionalisation and specialisation of medical training today has come at the cost of inspired idiosyncrasy, he suspects. "We have lost something."

Mukherjee himself belongs to an altogether different era – less flamboyantly anarchic, more cautiously measured. Born to middle-class Indian parents in New Delhi in 1970, he studied at Stanford before winning a Rhodes scholarship to Oxford, and fell almost by accident into oncology while at Harvard. He has that impenetrable sheen of the Ivy League star – effortlessly sophisticated and erudite, but ultimately rather unknowable – but his aversion to medical dogma is clear. Still, I can't help asking for a ruling on some of the questions most of us wonder about today. Can a positive mental attitude, for example, really cure cancer?

"I think it does a nasty disservice to patients. A woman with breast cancer already has her plate full, and you want to go and tell her that the reason you're not getting better is because you're not thinking positively? Put yourself in that woman's position and think what it feels like to be told your attitude is to blame for why you're not getting better. I think it's nasty."

But is it true? "No, I think it's not true. It's not true. In a spiritual sense, a positive attitude may help you get through chemotherapy and surgery and radiation and what have you. But a positive mental attitude does not cure cancer – any more than a negative mental attitude causes cancer."

A lot of my friends worry that stress is going to give them cancer. "I don't think so. I don't think it's true. There's a role of the immune system in cancer, but it's not as simple as people make out. It's not as if you get stressed, your immune system gets depressed, and all of a sudden you get cancer. Some cancers are more affected by it, such as lymphomas. But others – for example breast cancer – have very little to do with the immune system. There's no evidence that stress gives you breast cancer."

And yet we – particularly women – have been encouraged to blame ourselves for cancer. Mukherjee cites a study which found that women with breast cancer recalled eating a high-fat diet, whereas women without cancer did not. But the very same study had asked both sets of women about their diets long before any of them developed cancer, and the diet of those who now had breast cancer had been no more fatty than the rest. "In other words, women with breast cancer recalled – I suspect in an attempt to essentially blame themselves – having diets high in fat. It tells you how biased recollection is – but also how stigmatised the idea is, even today, because women think I must be to blame for something, I must have done something to myself."

When people ask Mukherjee to name the five things they should do to prevent cancer, he tells them: "Give up smoking, give up smoking, give up smoking, give up smoking, give up smoking." Like most of us, I've often been told that oncologists smoke more than anyone else – but when I ask how many of his colleagues smoke, he looks surprised. "Now? None. Zero. It used to be true. But not now."

What does he make of that other popular claim – that people have cured themselves of cancer with a diet of fruit juice and wheatgrass? "More power to them," he shrugs, reaching for his coffee. How does he explain their claims?

"We know there are spontaneous remissions in cancer, it's very well documented. Many cancers are chronic remitting relapsing diseases – that's their very nature. And human beings are pattern-recognising apes. It's the secret of our success; we recognise patterns. So we induce patterns; we have an unbelievably inductive imagination, and we say to ourselves, if the sun rose in the east for the last 365 days it must rise in the east tomorrow. So we typically indulge in inductive rather than deductive reasoning. It's very successful. But the problem with pattern recognition in this context is that it can become flawed. You might have a chronic remitting relapsing cancer and imagine it's remitting because you're drinking apple juice. But I don't think it's true. I think you're having a chronic remitting relapsing cancer – and that's the nature of your cancer.

"Maybe there are miracle substances out there that change the behaviour of particular cancers," he adds diplomatically. "But history suggests to us that we have to be sceptics here. If it was so simple then it would have been solved a long time ago."

Mukherjee could scarcely be less like the medical zealots he writes about in the early years of cancer research, who come across as frankly raving egomaniacs. The question that elicits by far his most unqualified response concerns the other books on the Guardian prize shortlist. "I read many of them. Amy Waldman is a very good friend, and I'd have given the prize to Amy – it's an amazing book, a transformative book, it's riveting and smart, and it's contemporary without being like a schtick. I loved it. In fact," he mentions, chuckling, "my wife [the artist Sarah Sze] and I are both in the credits. I think Amy's in my credits too. How many times has there been a book prize in which two shortlisted books have actually been critiqued and edited by other shortlisted authors?"

The self effacement makes me wonder how he has coped with his transformation from jobbing oncologist to international literary star.

"I've tried to take it in my stride," he says. "And the research grounds you because of the uncertainty – 99% of what we do in the laboratory is going to fail. So you deal with failure in a very fundamental way. And I was on call last weekend, the Thanksgiving weekend – and that grounds you. All of a sudden you come into hospital, and it grounds you in a way that's essential."