Barack Obama takes a break from his election campaign to visit his grandmother in Hawaii (Reuters)

By the time he was 10, Barack Obama was back in Hawaii. His mother had decided he needed to go to an American school, and she sent him back to live with his grandparents. She would follow the next year. These days, Michelle Obama says that the adolescence her husband experienced in Honolulu holds the key to his personality.

Some speculate that it's Hawaii that explains the languid cool that is so striking in Obama.

Visit what was his favourite eating spot, the beachside Hau Tree Lanai, where diners sit on a terrace beneath a canopy of low trees soaking up the evening sun — and where surfers and canoeists paddle past, the bright lights of Waikiki flickering a mile away — and it's easy to see how anyone raised in such a holiday paradise would grow up permanently laid-back.

Others have read much into the young Obama's talent for bodysurfing. Surely it was on the beaches of Hawaii that he learned to anticipate, judge and eventually ride the wave, giving him the knack for seeing early the shifts in the political tide that have now taken him to the presidency.

Yet his own account describes an upbringing that was not all placid serenity. His grandfather, outwardly easygoing, was frustrated with a luckless career as an insurance salesman: the young Obama would hear him making cold calls that hit a dead end. His mother was absent, eventually returning to Indonesia — even though her second marriage was now falling apart — to do field work on her doctoral dissertation in anthropology: a 1,045-page tome entitled "Peasant blacksmithing in Indonesia".

It fell to Obama's grandmother, Madelyn Dunham, to provide some constancy. The woman he called Toot and who died on Monday — on the eve of her grandson's triumph — was a model of Kansas solidity. While Stanley's career sputtered, she held down a senior job in banking, then all but unknown for a woman, which paid for Obama to be educated, privately and expensively.

Barack Obama in his 1978 senior yearbook class photo from the Punahou School Yearbook in Honolulu. Photograph: Punahou Schools/AP

He went to Punahou school, a 17-acre oasis of calm, order and privilege tucked behind a low stone wall topped with cacti. Thirty years after he graduated, you can still see teenagers at the end of the school day goofing around a basketball court as Obama would have done, playing a furious game of water polo or lounging around on the lawns.

But in Obama's own telling, this was a period of unease, a time when the inner wrestling with race and his own identity began in earnest, as he sought to understand his place as a black youth raised by white parents in a far-flung corner of the United States where there were few African-Americans. He has described his longing to feel like a black American, watching the grainy TV in his grandparents' 10th- floor, two-bedroom apartment so that he could copy the dance steps on Soul Train or lap up Richard Pryor's stand-up act. Bedtime reading was Malcolm X and James Baldwin.

Obama and the rest of his homeroom class pose for a portrait for the 1979 school yearbook.

A photograph in the yearbook for 1979, Obama's final year at Punahou, shows a young man who has shed the puppy fat of seven years earlier. Around him his classmates vamp it up for the camera, dressed in their finest finery, lounging on sofas in the palatial home of one of the students.

Obama is at the back. Dressed in a white suit over a wide-lapel shirt, his hand poised on his breast, he stands a head above his classmates, and not just because of his resplendent afro.

Barack Obama posing with his 1979 state basketball championship team for Punahou School. Photograph: The Oahuan, yearbook of Punahou School/AP

A few pages on, his personal entry has a photo of him playing basketball. Where other students used their entry to praise their teachers or to write about the challenges ahead, Obama left the school with the motto: "We go play hoop". The teenage Obama also left a self-composed photograph titled Still Life, featuring a turntable, a beer bottle, a basketball figure, a matchbook and a packet of Zig Zag cigarette papers. Alongside the photograph he gives thanks to "Tut, Gramps, Choom Gang, and Ray for all the good times". Tut for his grandmother, Ray is Keith Kakugawa, another mixed-race child, and Choom Gang is Hawaiian slang for marijuana.

Was Obama consciously trying to play up to a stereotype of the young black man? Eric Kusunoki, Obama's class teacher during his final four years at school, didn't see any of the alienation Obama discusses in his memoir. "It was a surprise," says Kusunoki, who still teaches at the school. "He didn't seem to be an outsider. He was well-known, popular, active in student activities." Other contemporaries also recall a relaxed, happy young man. Perhaps, they wonder, he was keeping his turmoil inside. Perhaps it was no more than "teen angst". But Obama himself has described these years — drinking and using marijuana and cocaine while at high school — as his greatest moral failure.

Once high school was over, Barack Obama — like his parents and grandparents before him — began his own journey. But where they had headed west, he now headed east from Hawaii, first to Occidental College in Los Angeles, then to Columbia University in New York, and finally to the very place his mothers's parents had escaped, the American midwest. And where his parents had yearned to slip free of roots and start anew, the young Obama seemed to be on a different quest. Having bounced from Hawaii to Indonesia and back before he was 10, he apparently longed for the sense of being anchored that comes with roots and memories that predate your own life. "I made a chain between my life and the faces I saw, borrowing other people's memories," he wrote. According to the New Yorker magazine, "he wanted to be bound".

The quest for steadiness seems to have begun immediately, whatever Obama would say later about his "dissolute" youth. His fellow students at Occidental have described young Barry Obama as a man who even then showed some of the discipline that has been on display during the last, long, two years. He jogged in the mornings and studied hard, even if he did allow himself the odd joint here and there. "Not even close to being a party animal," said one friend. Later, at Columbia, his mother confessed herself worried by the barrenness of his quarters: she told him he was living like a "monk".

Once he had graduated with a degree in political science, specialising in international relations, he stayed in New York, casting around for a career. He wrote to various civil rights organisations and black politicians: none wrote back.

In 1985 Barack Obama was unemployed. Eventually he answered an ad for a community organiser in Chicago, working with those on the south side of the city laid off by the shuttering of the steel industry. The Developing Communities project, inspired by the writings of social thinker Saul Alinsky, was looking for a black organiser who could work with the black unemployed. When Obama's application arrived, his future employer worried that this Obama of Hawaii might actually be Japanese. Once reassured, he offered him the job — on $10,000 a year, with $2,000 to buy a car.

He began work amid what he later described in Dreams from My Father as "the boarded-up homes, the decaying storefronts, the ageing church rolls, kids from unknown families who swaggered down the streets".

But Obama encountered scepticism among the neighbourhood's entrenched powers: church ministers and city bigwigs who clung to their turf, wary of the college-educated outsider.

Still, he was a good listener, often spending hours with individuals at a time to hear the full story of their lives. It was as if he wanted to learn from them as much as to help them, to learn how it was to live as a black American in a black American community.

Over three years, Obama led countless meetings on street corners and school gymnasiums, pulling together a coalition of churches, struggling middle-class residents and public-housing tenants.

He was efficient. He once arranged for 600 residents to talk with officials about contaminated water. He stood at the back, clipboard in hand, with a diagram setting out the names of all those who would speak and what points they would make. He had arranged substitute speakers in case anyone became nervous.

He was learning the centrality of preparation — and organisation — to making political change.

He achieved some modest victories, pressuring the city to open a jobs centre in the neighbourhood, demanding action to remove asbestos from a housing estate. And he was introduced to a minister who, he was told, was keen to help Obama's organising efforts. His name was Reverend Jeremiah Wright. It was in this period, in 1988, that Obama made his first journey to Africa, visiting the relatives and half-siblings he had never known, taking several weeks to travel around Kenya with his half-sister Auma. "It was like going through a secret garden," Auma told the Guardian. "Discovering a new plant here, a new plant there and watching it bloom."

Obama listened for hours to Mama Sarah tell her stories, then stopped at the graves of his father and grandfather. "I saw that my life in America … all of it was connected with this small plot of earth an ocean away," he wrote later. He sat at the double graveside and wept for a long time, before feeling a "calmness wash over me". Back in the US, and after three years of pounding the streets of Chicago — writing short stories on the people he met that were, according to one who read them, "beautifully crafted" — Obama wanted to send himself back to college. He won a place at the very top: Harvard law school.

It was there that the first outlines of what would become Obama's distinctive political style began to form, or at least become apparent.

President of Harvard Law Review, Barack Obama. Photograph: Steve Liss/Time Life Pictures/Getty Images

During his first year he became an editor on the Harvard Law Review — a publication whose prestige has no equivalent outside the US. The foremost jurists in the land submit their writings to a panel of students for acceptance or rejection — and these students then edit the work of their elders. Obama excelled at it. In his second year, he stood for election and became the first African-American President of the Law Review.

He won thanks, in part, to the votes of conservatives on the review. They did not agree with him on the issues, but they were impressed that he truly listened to them, that he seemed to take them seriously.

On one occasion, he made a speech defending affirmative action that effectively articulated the objections to it. Rightwingers believed Obama had shown them deep understanding and respect. It was a mode of discourse that Obama would employ again and again (most notably in May 2008, when he sought to stem the bleeding caused by his association with the fire-breathing Rev Wright, with a speech that demonstrated empathy for white as well as black resentments). A former teacher at Harvard, Martha Minow, has said that "he spoke with a kind of ability to rise above the conversation and summarise it and reframe it".

In the breaks from Harvard, Obama would return to Chicago, increasingly regarding the city as his true home. It was after his first year, in 1989, that he took a summer job at a prominent law firm now called Sidley Austin. There he met Michelle Robinson, a young attorney at the firm assigned as his mentor.

She was wary at first, a fact she likes to retell: "He grew up in Hawaii! Who grows up in Hawaii? Who names their child Barack Obama?" He seems to have had many fewer doubts about her. She had everything he did not: a solid, stable family rooted in a black American community. Obama soon decided he did not want a career in corporate law, and that he wanted to make a life with Michelle.

When he returned to Chicago after graduating from Harvard Law School, he found his reputation had preceded him. His election to the presidency of the Law Review had prompted a clutch of glowing profiles in the national press. Many of the city's top lawyers wanted to recruit him.

One civil rights attorney, Judson Miner, had read about Obama and called Harvard to speak with him.

"They said, 'Is this a recruiting call?' and I said 'I guess so,'" Miner told the Guardian in his office in a converted townhouse in downtown Chicago. "And this young lady said to me — jokingly — 'well, you're going to be 643rd on the list'."

As the Obama campaign often liked to mention, he could have had his pick of top, high-paying jobs, on Wall Street or beyond. But first he responded to a call which had come following those magazine profiles — from a literary agent suggesting he write a book. Initially conceived as a treatise on race relations, it became the deeply personal memoir Dreams from My Father.

So Obama passed up Wall Street and returned to Chicago in 1991. He did not become a bigshot lawyer but returned to community organising with Project Vote, a registration drive aimed at increasing participation among African-Americans. In less than a year, Obama had hired 10 staff, recruited 700 volunteers and registered 150,000 new voters. It was experience that stood him in remarkably good stead. Not only was he now making contacts with Chicago's political establishment, he had learned from the ground up the mechanics — and importance — of grassroots organisation. It was his mastery of this often neglected area of politics that would, come 2008, give Obama the edge over Hillary Clinton — and eventually help propel him to the White House.

In October 1992, as the project wound up, Obama and Michelle married at Jeremiah Wright's Trinity United Church of Christ. By now Obama had abandoned the agnosticism of his father and the eclectic, pick-and-mix anthropologist's view of religion of his mother.

His search for stability had, in part, led him to become a Christian. In a neat piece of symbolism, the wedding party was at the South Shore Cultural Centre — a rundown, formerly all-white country club.

Soon, Obama was building a portfolio career. He worked at a law firm renowned for its civil rights advocacy and its efforts to improve deprived neighbourhoods. In one case, he represented community organisation Acorn in a successful suit accusing the state of Illinois of failing to help the poor to register to vote.

In another, Obama represented a non-profit economic development group working with landlord Tony Rezko to provide better housing for working-class Chicago residents.

Both those associations would hurt Obama in the 2008 campaign. Republicans accused Acorn of involvement in voter fraud, while Rezko, who later donated money for Obama's first run for elected office, was this year convicted on corruption charges.

When he was not at the firm, Obama was teaching at the University of Chicago. Douglas Baird, the professor who recruited him, recalls Obama as a popular lecturer. In addition to Obama's charisma, Baird said students were drawn to his ability to present ideologically charged subjects like race, voting rights and constitutional law without thrusting his own beliefs on them.

"He always listens, and he might not agree with you, but you never felt he was brushing you off," Baird said in his office in the University of Chicago law library.

The Obamas were now living in Hyde Park, an uber-liberal neighbourhood of Chicago dominated by the university. Politically it was idiosyncratic, a middle-class island within the less affluent and largely African-American south side, racially mixed in a city that is still heavily segregated. It tended towards the liberal and intellectual, always keeping a distance from the legendary Chicago Democratic party machine.

Obama's thoughts were increasingly focused on elected politics. Ever since Harvard, if not before, he had been thinking about how best to effect change. His years as a community organiser had persuaded him that he couldn't do enough at grassroots level: the big decisions were taken by elected politicians.

In 1995 he saw his chance. Alice Palmer, a popular African-American member of the Illinois state senate, was vacating her seat, which included Hyde Park, so that she could run for the US Congress. Obama would aim to succeed her.

Baird was horrified. "Our basic view was that he was a very talented man who was squandering his future," he said. Baird urged him not to waste his intellect on petty state politics but to join the University of Chicago law faculty as a full professor. Obama said no: he was running.

As events unfolded, Obama would show the steel that has been revealed repeatedly in the 2008 campaign — and which confounded those who believed he would be too professorial for the hardball of politics. Palmer backed Obama to succeed her but when she lost her congressional race she changed her mind: she wanted to keep her state senate seat. Having built a campaign apparatus, attending countless get-togethers, some tiny, in people's homes — including one where he met the former Weather Underground militant William Ayers — Obama refused.

Palmer rushed to obtain enough petition signatures to win a spot on the ballot. Obama supporters mounted a legal challenge to the validity of her signatures and had some of them thrown out by a judge. While he was at it, the Obama team spotted irregularities on the rest of his opponents' forms too. By the time he was done, Obama's was the only name left on the ballot.