The benefits of big farms are still being debated. Photograph: Martin Godwin

In the 18 months since the Katine project began, a common thread has emerged from those involved in the international aid debate: transformative change is not going to be achieved by the slow underpinning of livelihoods envisaged at the outset of the project.

To the outsider approaching Katine from the main town of Soroti, it is impossible not to be struck by the apparent potential of the land and the obvious poverty of the people who live by it. Watch a young woman scratching away with her hoe in the rough ground, and the answer seems equally obvious: investment in machinery, even a hand plough, would surely transform her prospects.

How much more so if a few hundred acres – 50 or 60 of these village gardens – were amalgamated into one big farm, able to afford machinery, pay decent wages to its workforce, and offer food security to the village?

Whether big farms are the key to Africa's future has been a matter of debate for more than a generation. While increasing agricultural productivity is a key to poverty alleviation, achieving it is more complex. African history offers few reassuring precedents, for successful development has come at a high price – the landlessness of Zimbabweans, say, or the long grip on political power of the white Kenyans.

Worse, Asia's famous Green Revolution now looks flawed. Soils made productive through the use of fertiliser are becoming exhausted; more and more fertiliser is needed just as its price – linked to the cost of oil – soars. Suitable land is a finite resource. Improving seed is a process, not an event, as pests adapt to meet new conditions.

Even genetically modified BT cotton, which produces spores poisonous to many insects and seemed to offer astonishing returns, is now being challenged by the unexpected invasion of different pests that were – it is being understood – once kept at bay by insects now killed by bacteria. As any New Zealand farmer plagued by the rabbits once introduced as food could explain, you interfere with the balance of nature at your peril.

Meanwhile, with climate change, water scarcity and falling commodity prices, the agricultural environment becomes more challenging just as an era of unparalleled economic growth and spiralling demand for more and better food comes to an end.

Yet the engine for the great economic expansion in Europe was the transformation of agricultural productivity, accompanied – in the enclosure movement of the late middle ages - by one of history's more spectacular land grabs.

In Britain, millions were impoverished. The wholesale disregard for traditional tenure uprooted villages and created a rural underclass. But it also led to innovation, transformed productivity and led to a waged labour force.

So, as the demand for food and jobs expands exponentially, the question is less whether big farms are necessary to making a country's food secure than how to get there as equitably as possible.

An economist like Paul Collier is convinced that radical steps have to be taken. "African peasant agriculture has fallen further and further behind the advancing commercial productivity frontier," he wrote at the end of last year in the journal Foreign Affairs. "Based on present trends, the region's food imports are projected to double over the next quarter century." Only large scale farms, he argues, are capable of providing the investment and market access that is essential to produce the surge in food production necessary to keep up with demand.

Rubbish, says development expert Steve Wiggins. "Yes, he is correct to emphasise the need for commercial farming. But no, he is wrong to imagine that this requires doing so on a large scale. His solution is unnecessary, flies in the face of history and carries important dangers."

The Guardian's partner in the Katine project, FarmAfrica, would agree. They lined up with the International Food Production Research Institute (IFPRI) at an international conference organised by the ODI three years ago. Its findings were reviewed recently in a discussion paper where the IFPRI called for African governments to prioritise support for small farms while developing exit routes for those whose land is swallowed up by more successful neighbours.

Unlike the Collier cry for rapid commercial development, their emphasis is on organic growth. They point to the unhappy experience of attempts to impose commercial farming, with its history of poor labour standards and clumsy machinery maintenance, and a disregard for the good of the land itself.

The most pressing challenge to such development, apart from the sheer need for much more productive agriculture, is the appetite of China, the Philippines and some Gulf states for new agricultural land to provide food security for their own people.

There is also pressure for land for bio-fuels. Last year there was a furious row over allegations that land in northern Uganda that belonged to people displaced by the Lord's Resistance Army was being acquired illegally by Ugandan army officers and businessmen who wanted to grow sugarcane. The Ugandan government has stepped back. A plan to grow palm oil on an island in Lake Victoria has also been abandoned.

To get a fair deal for the small farmer in fertile regions might require some radical thinking. Commodities like coffee and cotton could be one source of extra income. Ugandan small holders have a long tradition of growing coffee in the central and southern regions, and cotton in the north and east.

The roadsides around Katine are scarred with ruined cotton gins. Now, backed by USAID and some major seed producers, efforts have been made to improve production and marketing. There was even a fashion show featuring Ugandan cotton in Kampala last year.

But cotton seems to be responding to investment more slowly than coffee, where trade has recovered. Deregulating a market that was once entirely controlled by the government has greatly improved the amount received by growers.

Working the land in Malawi. Photograph: Martin Godwin

An experiment in Malawi is attracting world attention. A 10-year programme backed by Cru Investment's Africa Invest has acquired the leasehold of land to form two large (more than 100 ha) farms and two smaller ones. They employ local people, offering training and experience in working with new techniques and machinery, and teach skills like lorry driving and management.

At the same time, the operation supports local smallholders by buying in and marketing their surpluses at fair trade rates (it boasts of "the discipline of investment" over "aid hand-outs"). Jon Maguire, the chairman of the project, promises the villages will meet all the Millennium Development Goals within two years of start up. He claims that already he employs 2,000 people and pays well above poverty rates.

The idea behind it is that commercial investment can overcome some of the sustainability problems that dog aid projects. If the investment is economically viable, over the 10-year period of commitment – the theory goes – people involved in it will build up the expertise and knowledge to be able to take it on themselves. Alternatively, Africa Invest might renew its interest.

On paper, it has overcome all the obvious hurdles: it has security of tenure, but pays its landlords a share of the profits. It is spreading resources and knowhow throughout the community as well as contributing directly to poverty eradication. It can bring capital investment and marketing clout and it is seeking relationships with supermarkets to improve the share of the eventual sale price received by the grower. Too good to be true? Or a future for African agriculture?