A hammer and sickle logo at the Chinese Communist party's school in Shanghai. Photograph: Carlos Barria/Reuters

To understand the long journey of the People's Republic of China and its rulers you might start at a modest two-storey grey brick building in Shanghai's former French Concession. It lies a short stroll from the Harry Winston store, with its blazing diamonds, past over-priced bars and glassy towers.

It was in 1921 that 13 young Chinese men gathered in this newly built home, then located at the edge of the city, overlooking a vegetable field. Though most were lodging at a nearby girls' school as the Beijing University Summer Vacation Tourist Group, sightseeing was not on the agenda. In strictest secrecy, with the aid of two Comintern representatives, they were hammering out the programme for the newly formed Communist party of China.

Threatened with discovery by the police, they fled to the nearby town of Jiaxing, where the Communist party's first national congress concluded on board a pleasure boat on South Lake.

Six weeks from now, their descendants will gather in Beijing for the 18th congress and will hand over power to a new generation of leaders, with Xi Jinping at the helm. The Communist party is now the world's largest and most powerful political movement, with more than 80 million members; it controls a fifth of the globe's population and the second largest economy.

The congress is expected to be its shot at returning to business as usual, after a tumultuous year culminating in Friday's announcement that the disgraced politician Bo Xilai, who was once tipped for promotion in this transition, faces criminal charges. He is accused of abusing power, corruption and sexual impropriety; he is said to bear responsibility for his wife's murder of the British businessman Neil Heywood.

Though most of November's meeting will take place behind closed doors, the party no longer needs to cherish obscurity; these days the congress is a carefully mounted display of power and unity. More than 2,200 delegates will meet at Beijing's Great Hall of the People in considerable pomp and some splendour. Police will have silenced any hint of discord; activists and dissidents will be detained or put under surveillance.

"The 18th party congress is coming, so we thought we should learn about the first one," said Wang Yao as he left the site of the first meeting, now a museum. "The first generation had such difficult conditions when they started, but now China is more and more prosperous, and it's getting better and better."

Wang and his family are among the winners. He works in sales and marketing; his son Tommy has just graduated from Newcastle University. "The new government will lead China at a new speed, but with the same original spirit from here," he predicted.

Others find it harder to see the continuity. "The internal procedures of the congress have not changed much over the decades since Stalin institutionalised the congress as a showpiece of party unity," said Jeremy Paltiel, an expert on the party at Carleton University in Canada. But the party itself is a long way from its roots, he noted: "Arguably it remained something of an organisation of revolutionaries until the death of Mao Zedong in 1976. Since then it has become a party of functionaries and officials. The party is a party of 'cadres' and so is the congress, with a few 'model workers' as window dressing."

Over nine decades, the party has proved extraordinarily adaptable as well as relentless in its pursuit and preservation of power. It has endured violent suppression and war; the devastating purges, famine and political turmoil its own leader created; the alienation of its people through the bloody crackdown on the 1989 pro-democracy protests that began in Tiananmen Square. It has survived by jettisoning the ideology once at its core, in favour of "socialism with Chinese characteristics" – today's hybrid of rampant capitalism and the heavy hand of the state.

Under its reign, life expectancy and literacy have soared; two years ago Shanghai topped the OECD's global rankings of schools. Women's rights have improved markedly. Hundreds of millions have climbed out of poverty – an achievement described by the World Bank as a miracle.

Yet tens of millions died in the manmade disaster of the Great Famine and tens of millions were persecuted in the chaos of the Cultural Revolution. In-equality is soaring. Human rights abuses and corruption are rife; the Bo scandal has highlighted the unbridled power of senior leaders and the often lavish lifestyles of their families. Still, facing it down may be the least of the new leadership's challenges; they face slowing growth in an imbalanced economy, environmental devastation, rising public expectations and an increasingly complicated foreign policy environment.

To trace the "glorious course" of the national congresses, as Shanghai's museum does, is to trace the evolution of the party and the country. Guards say visitor numbers have climbed as the 18th congress approaches; around 1,000 arrive each day, with varying degrees of enthusiasm and earnestness. In an upper gallery, one man is schmoozing a business contact via his mobile as he glances at exhibits. At the souvenir counter, official histories of the party jostle for space with Mao watches, kitschy Red Army mobile phone accessories and a pencil case reading: Study hard, earn money.

The group that gathered in July 1921, "unlike the party of a later time ... had a great deal of idealism but little organisational discipline", said Wen-hsin Yeh, an expert on the birth of Chinese communism at the University of California Berkeley. The early years were volatile; only two of the 13 remained in the party by the time it took power in 1949. Others had died, chosen to leave or were purged. Yet by 1927 it had 58,000 members and the Nationalist leader Chiang Kai-shek was alarmed enough to launch a brutal suppression of the party; the following year it would meet in exile, in Moscow. The seventh congress would not take place until 1945, at the Communist rebel base in Yan'an, just before the end of the Japanese occupation and the resumption of war with Chiang's forces.

"It confirmed Mao's absolute authority in the Communist party," said Gao Wenqian, previously a researcher at the Chinese Institute of Central Documents and now a senior policy adviser at New York-based Human Rights in China.

Eleven years later, the eighth congress was a celebration of power; the party now ruled China. More than 1,000 delegates, representing 10 million members, gathered in Beijing. "Mao hoped to become the leader of the world revolution," said Gao. "Then he started the Great Leap Forward."

The disastrous attempt to transform the economy and society overnight resulted in tens of millions of deaths in the Great Famine – and a new willingness to challenge the chairman. "Mao worried that his position was not stable, so the ninth national congress was not held until 1969," said Gao.

That year a giant portrait of Chairman Mao hung above the meeting. He had already unleashed the Cultural Revolution to eliminate his rivals; Lin Biao – who would die in a mysterious plane crash two years later – was confirmed as his new successor. "The ideological, political and organisational guidelines of the ninth national congress were wrong," a caption at the museum states flatly.

Eight years later, after Mao's death, the 11th congress saw the return of many of his victims; notably Deng Xiaoping, who would do more than anyone to turn China into today's global powerhouse. That year, 1977, saw the establishment of the pattern that still prevails, with a congress every five years. The party is an increasingly institutionalised body that has regulated the turnover of leaders, leaving behind the days when one man could control it.

Many within its top ranks appear to have feared Bo, who sought to harness mass support in pursuit of his ambitions, as a destabilising figure. These days "the national congress is like a rubber stamp. Before the 18th, all the significant decisions have already been made, such as the personnel arrangements and the policy direction afterwards," said Gao.

Gao, like others, believes three decades of economic reform without political change have led to contradictions and conflicts which are reaching a critical, explosive point. As growth slows, it can no longer disguise the problems.

The Bo Xilai incident gave the party the opportunity to turn over a new leaf, said Gao. "In my opinion, they missed it. From the trial of Wang Lijun to dealing with the Bo case, it has all been black operations ... It has been about the political struggles, trading by different groups in the power transition. The Communist party is 92 years old. It is senile and lethargic. They only want to maintain the present situation, not to make any changes."

Others hope a new generation of leaders could yet grasp the nettle. They are "collectively more diverse in terms of their professional and political backgrounds, more weathered and adaptable from their formative experiences during the Cultural Revolution and more cosmopolitan in their worldviews and policy choices than the preceding generations. They may contribute, in a profound way, to political institutionalisation and democratic governance of the country," argued analyst Cheng Li of the Brookings Institution in a recent paper on the handover.

Yet, he noted, the increasing pluralism of Chinese society and diversity among political elites make reaching consensus far harder. "Ideological disputes within the leadership are real and they may become too divisive to reconcile," he warned.

And while many in China still feel the party has brought stability and wealth, they are increasingly cynical about the motivation and pronouncements of its leaders; even about its founding myths. When one museum visitor said he had come to see China's past, his friend interjected mockingly: "Is it real history?"

Early records are sparse because it was hazardous for party members to be caught with revolutionary materials. But the reconstruction of the first congress was a political, not just historical, act.

In one room a tableau of waxwork figures shows Mao, clad in a flowing blue scholar's gown, holding forth to rapt delegates. But he was not then the party's pre-eminent figure, as Yeh pointed out; it was Chen Duxiu who was elected general secretary, albeit in his absence.

Samuel Liang, an assistant professor at Utah Valley University and visiting research fellow at Shanghai Jiaotong University, has described the monument as "an empty shell that attempts to eternalise a reinvented past by terminating the place's living, natural history".

Mao noted that a revolution is not a dinner party, Liang wrote. "Neither is revolution, I might add for Mao, a peaceful pilgrimage to a revolution museum."

That the birthplace of Chinese communism now stands in one of Shanghai's ritzier shopping and entertainment areas, Xintiandi, makes an odd sort of sense. Superficially, the complex's developers preserved the traditional housing. In reality, they reconstituted facades and remade interiors. Residents made way for boutiques and restaurants. Both party and neighbourhood retain their shells, but the spirit that animated them has long since vanished.

"The transformation of the area into what today is known as Xintiandi is indicative of how the party has changed from leading political – or ideological – driven governance to leading economic or developmental-driven governance," added Liang.

How the Communist party will evolve next remains to be seen. When Hu Jintao took power a decade ago, many hoped it would bring significant change. "The major challenge for Xi Jinping is that he will have to confront things to convince the public that they are really serious in political and economic reform," said Zhang Jian, a political scientist at Peking University. "The steam of reform has been lost in the past 10 years and the deterioration of those economic and political conditions is making reform more and more urgent."

Upstairs at the museum, there are no such doubts: the achievements of Hu's tenure fill two large walls detailing the 16th and 17th party congresses. No mention is made of this November's meeting. "The 18th congress? It hasn't even started yet!" exclaimed a receptionist, when asked how it might be recorded. It would take several years to add, she suggested.

To include it, the party's historians must once again rearrange the past.

Voice from China

Michael Anti (born Zhao Jing), 37, blogger and journalist

"People and media had hope [in 2003, after the last congress] that a new administration means new hope. But now you read newspapers and hear people talk, nobody has any kind of hope. Some people still think there could be a degree of political reform but most of the public and media have stopped buying it. People see no difference between this administration and the next administration, no matter who is the leadership. So people have stopped dreaming about the future.

"When politics and the economy have problems, you have the nationalism, the anger of the people at the messed-up society. So you see like in the anti-Japan movement, you see this anger, this violence. Basically we see, I personally see, in these coming years, it will be tough for both government and citizens."

"The greatest challenge facing China in the next decade will be democratisation. Corruption has all Chinese most concerned, and it often causes great anger and instability among society, it may even shake the government …The second issue is the economy. With so many problems remaining in China's economy, and now facing the global recession, it's hard for China to maintain over 10 per cent GDP growth as in previous decades. The wealth gap has caused serious problems to social stability and legitimacy of the governing Communist Party.

"Deregulating media is also an issue. Let media play a role in supervision, let the people's voice be heard. Internet shields, monitoring and censorship are the most important and urgent problem with great international and domestic pressure. If public opinion cannot be fully expressed online, it will be demonstrated on the streets … Political reform has not really been done in the last three decades of opening and economic reform. Political reform is the greatest challenge and is the people's great hope, but it also requires the new leaders' courage."

Xu Binjie, 57, retired company manager

"I'm concerned about issues concerning people's lives. Because I'm old, I worry about things like pensions, and I am worried about the economic crisis. I worry that the economy is slowing down. My family is in Harbin, life is pretty good for them already but I hope it can be a bit better. We are starting to feel the effects of inflation, we hope it can be stabilised. They must make the economy better and improve people's lives. But I feel very confident about our country. I believe no matter how the leadership changes, people's lives will continue to get better."

Yao Guang, 23, shoe salesman and second-generation migrant worker

"Wages are too low in both state-owned enterprises and private enterprises, people keep leaving and the businesses cannot keep them, because the wages are too low. Life is stressful here, it's hard to survive, there's a lot of pressure … My parents are depressed about all the pressure here, they have older people to care for back home so it's a lot of economic pressure on them. To buy an apartment or rent an apartment, and also for me to get married is difficult, because our economic situation is not so good … They should help people who come from villages. The farmers and also those who leave the villages to come into the cities, they need more subsidies, in health, in education, in every aspect."

Li Yifei, 28, art gallery director

"Whether or not the political arena will change, that is the biggest issue now. China has always been like this, for centuries, there has always been one dynasty with one group of officials. For this one, the rules of the central government are not stable. It's possible in 10 or 15 years there will be change, maybe there are even people working on this now … The economy is also not doing well and there are a lot of problems concerning people's survival."

Beryl Li, 36, tourism agency employee

"Housing in Beijing is a big problem now. Housing prices are too high and wages are low, the ratio is all off, there's a huge difference. But the Party Congress? We rarely think about this. We just don't pay attention to these issues … I am hoping the economy will get better. If the economy is good, it's better for people's lives."