Lynn Margulis, who has died following a stroke aged 73, was a world-renowned evolutionary biologist, professor of geoscience at the University of Massachusetts at Amherst, a member of the US National Academy of Sciences and recipient of the National Medal of Science. Yet when, as a junior academic in 1966, she wrote a paper that many biologists at the time regarded as a wild evolutionary heresy, few would have predicted her subsequent eminence and the extent to which her theory would transform our understanding of evolution, becoming so mainstream that it now features, entirely uncontroversially, in school biology textbooks.

Then, as now, the leading evolutionary theorists were committed to the idea that the main motor of evolutionary change was competition – between organisms within a species, and between species. Margulis instead expanded on an idea that had been first proposed by the Russian anarchist Prince Peter Kropotkin in 1902 in his book Mutual Aid and then developed by Soviet biologists in the 1920s – that co-operation is as important a feature of evolutionary change as competition (a view that Charles Darwin himself would not have been unsympathetic to, despite the protestations of some of his more fundamentalist disciples).

Margulis's great insight was that, in the early history of life on earth, some 3.5bn years ago, when the only living forms were single-cell creatures without complex internal structures, evolutionary success could arise through increasing cellular complexity. This complexity could emerge through symbiosis between organisms of different species. The most effective form of symbiosis occurred when the two cell types merged, each contributing to the creation of a harmonious living whole, the eukaryotic cell.

Among the several examples of this process of symbiogenesis, Margulis proposed that those cellular powerhouses, the mitochondria, tiny membrane-bound organelles that inhabit in their hundreds each of the trillions of cells that make up every organ of our bodies, were once free-living creatures, before being incorporated into the symbiotic life forms which were our distant ancestors.

Margulis recounted how the paper was rejected by around 15 leading biology journals before being published in the Journal of Theoretical Biology in 1967. It took more than a decade, however, before she was vindicated. Today it is recognised that mitochondria retain traces of their original free-living existence in the form of their own characteristic DNA. She went on to extend her symbiogenetic hypothesis to embrace other cellular organelles, such as the flagellae that many bacteria employ to swim, though here she has not yet carried the mainstream with her.

Picking up on the fertile ideas of the biologists and philosophers Humberto Maturana and Francisco Varela, she argued, once again against the evolutionary mainstream, that organisms are not passive "vehicles" – to use Richard Dawkins's well-known phrase – ground between the upper and nether millstones of genes and environment, but are instead active self-organising constructors of their own destiny, a concept known as autopoiesis. Unfashionable in last century's heyday of genetic reductionism, the concept is more in accord with the richer biological understanding that has followed the sequencing of the human genome.

When, in the 1960s, James Lovelock developed his Gaia concept of the earth as a self-regulating living organism, Margulis embraced it enthusiastically, although not in the more mystical form popular among some of its adherents. She subscribed to a weaker version, seeing the planet as an integrated self- regulating ecosystem. However, at the same time, she looked dourly forward to the prospect of humanity's extinction through our insistence on trying to dominate, rather than live harmoniously with, nature and thus upsetting the self-regulatory processes. At that point, she argued, those great evolutionary survivors, the lowly slime moulds, would inherit the earth.

She was born Lynn Petra Alexander in Chicago, Illinois, and entered Chicago University when only 14. Five years later she met and, in 1957, married the cosmologist and science populariser Carl Sagan, with whom she had two sons, Dorion and Jeremy. By the time she left graduate school in Berkeley, California, in 1965 the marriage had ended. She later married the crystallographer Thomas Margulis, with whom she had a son and a daughter, Zachary and Jennifer. They divorced in the 1980s.

Margulis was long committed to the public understanding of science. She was a charismatic lecturer to audiences at all levels, and made several visits to the Edinburgh Science festival, enthusiastically and generously developing experiments that the schoolchildren could carry out in the festival's ad hoc labs. She wrote and co-edited a range of popular science books, covering topics from global warming and Gaia to the biological significance of sex and the joys of microbiology, as well as reflections on scientists and the doing of science. Several of these were written or co-edited with her eldest son Dorion.

Her children survive her.

•Lynn Petra Margulis, evolutionary biologist, born 5 March 1938; died 22 November 2011