Under a hot sun, boys herd goats in search of green grass. In a nearby plot a couple weed a crop of maize. The ceremony and fanfare that surrounded the visit of Madonna in 2009 now seems very distant.

It was here in the village of Chinkhota in southern Malawi that the pop superstar broke ground for the construction of a prestigious academy for underprivileged girls. Billed as a "gift" to the African country from which she has adopted two children, the $15m (£9.37m) institution was to take in 500 girls and prepare them to be "future women leaders".

Now local elders have accused Madonna of letting them down, and the Malawi government has been forced to admit it does not know what her plans are, after she announced that the academy had been scrapped.

Madonna stated that she would instead focus on building secondary schools across the country, rather than one school, but officials in Malawi have been left scratching their heads as to what this will entail. "I want to reach thousands, not hundreds of girls," she said in a statement this month. "I want to do more and I want to do it better."

The news has come as a bombshell to villagers who surrendered their ancestral land to make way for the school. The Malawian government said it had not been consulted and will now summon the 52-year-old singer to explain herself.

"We'd like to know why she has changed," said Peter Mutharika, the education minister. "Yes, we do appreciate that it is her project; she devised it and she knows best how to implement it. But still, as government, we'd be interested to know why there is this change." Mutharika, tipped to be the next president when his brother, Bingu wa Mutharika, ends his final term in 2014, added: "I honestly don't know the number of schools she is going to construct, where she will build and for how long. So until we talk to her, we can't comment much." He said the government will also review the memorandum of understanding it signed with Madonna to "see what it says before we can embrace her new approach".

Malawi is one of the world's poorest nations, with more than half of the population of 12 million living on less than one dollar a day. Some 200 people were removed from Chinkhota village, on the outskirts of the capital, Lilongwe, to make way for the Raising Malawi Girls Academy.

Most of the villagers were born here and thought the land belonged to their ancestors. They threatened to block the project until Madonna dipped into her pockets to compensate them with $500,000 (£312,800). The Malawian government, which has often defended Madonna's charitable work, helped evict the villagers, claiming it was state land meant for development projects.

Chinkhota became a building site – but without a building. Construction of the academy was delayed over a dispute between the Raising Malawi charity and villagers who claimed they were not adequately compensated for land.

The building was due to be completed in December this year. However, the earth-moving machines that had been landscaping the area have disappeared. Villagers feel betrayed and accuse Madonna of breaking her promise. Maxwell Matewere, Malawi's leading children's rights activist, said Madonna should not "dump" the project.

Matewere, director of Eye of the Child, led two unsuccessful attempts to prevent Madonna adopting two children – David, five, and four-year-old Mercy – from Malawi after their mothers died.

He said the pop star was trying to "run away from long-term obligations" and she should go ahead with the academy because "you educate a few to educate others. She must borrow a leaf from others like Oprah [Winfrey] who have done it in South Africa."

Madonna is funding several charities in Malawi, including homes for children with Aids. She has built a multi-purpose community centre 30 miles from Lilongwe which looks after more than 8,000 orphans. "In a country where only 33% of Malawian girls attend secondary school, I realise that the plans we had in place for the Raising Malawi Academy for Girls simply would not serve enough children. My original vision is now on a much bigger scale."

She said she was in the process of implementing several changes and additions to the management of Raising Malawi in the US and Malawi.

Madonna said she had teamed up with Trevor Neilson and his team at Global Philanthropy Group to "shift the strategies so that we can accomplish our goals with more efficiency as we continue to consult our government partners in Malawi".

She said Raising Malawi was focused on an approach which includes building schools within communities across the country. A pilot school is on the cards that will "address the barriers keeping girls from secondary education".

Neilson said the new community-based approach by Raising Malawi "will provide the opportunity for many more girls to receive a quality education".

Madonna's spokeswoman did not comment today but said she would seek further information from the singer's management.