Poor El Alto is the satellite city of rich La Paz, overlooking the capital of Bolivia from up in the altiplano. In 1952 it was little more than a village; in 1960 it had fewer than 30,000 people; now it has exploded into a metropolis with up to 1 million or more living in the rarified, 4,100m (13,000ft) air. It's remarkable because this is one of the few cities in the world where the poor live physically above the rich, and where the vast majority are indigenous – from just one of the country's many ethnic groups.

Hundreds of thousands of people have flooded in from the countryside to find work and opportunity, but increasingly the reason they give for moving is that frequent droughts, erratic rainfall, heatwaves, unseasonal frosts and floods have made conditions too hard to grow crops. Bolivia has had five major droughts or heatwaves, as well as floods and major mudslides in the past decade. Few people in El Alto could be classed entirely as "climate refugees", but the changing physical environment is clearly one of the new drivers of people to this burgeoning city.

We talked at some length with Mario Mamani. He came from Pinaya village high in the Andes, below the great snowcapped Illimani mountain, arriving in El Alto about six years ago. It was really tough to start with, he said, but now he gets work as a driver. He hasn't seen his 80-year-old parents for more than a year but he was doing well. His home has running water, his family doesn't go hungry, and he and his wife are building a small house for his parents within their yard. In a few weeks, they would come and the family would be together.

Many others in El Alto, he said, move between the city and their villages, growing crops on their ancestral land, and getting occasional work in the city or in the mines. Mamani longed to return to Pinaya, but thought it would be impossible to survive. "Yes, there has been a big climate change there … That's the reason why we had to move and settle here in the city," he said. "We had hail, we had frost, which made it difficult to make enough money to live. Maybe it was because the weather was already tired, because at times it was OK and at times it was not."

So we went to Pinaya. The five-hour journey took us to one of the most beautiful places in the world, tucked at the head of a valley at 3,600m, under the peaks of the 6,300m Illimani mountain. Society there is deeply traditional, it's bitterly cold in the winter and baking in the summer, and Pachamama – the spirits of nature, or Mother Earth – still dominates how people see the world.

We talked to Faustino Mamani, Mario Mamani's brother, who moves between El Alto and Pinaya. He is a mountain guide so has another income, but he showed us his crops. They had all failed this year because a heatwave after they were planted was followed by a drought. Animals were sicker now, he said, and plants were diseased more often because the warmer temperatures allowed insect pests to live at higher altitudes.

Two old farmers, Felipa Condori and Luciano Huanca, told us how desperate they were. They did not have enough food, and urgently needed someone to advise them how to adapt with different crops or technology to a changing climate. They wanted help from anywhere and were in no doubt whose fault it was that the mountain was losing its glaciers and the climate was changing. "We have offended the Pachamama, but above all it's the rich countries!" Condori said.

Like so many villages in mountainous areas, the young people seemed to have mostly left. At least 15 houses had been abandoned in Pinaya, the families moving to El Alto or even to Argentina. There was talk of Europeans setting up a small hotel for climbers and walkers there.