Benoît Mandelbrot, who has died of pancreatic cancer aged 85, enjoyed the rare distinction of having his name applied to a feature of mathematics that has become part of everyday life – the Mandelbrot set. Both a French and an American citizen, though born in Poland, he had a visionary, maverick approach, harnessing computer power to develop a geometry that mirrors the complexity of the natural world, with applications in many practical fields.

At the start of his groundbreaking work, The Fractal Geometry of Nature, he asks: "Why is geometry often described as cold and dry? One reason lies in its inability to describe the shape of a cloud, a mountain, a coastline or a tree." The approach that he pioneered helps us to describe nature as we actually see it, and so expand our way of thinking.

The world we live in is not naturally smooth-edged and regularly shaped like the familiar cones, circles, spheres and straight lines of Euclid's geometry: it is rough-edged, wrinkled, crinkled and irregular. "Fractals" was the name he applied to irregular mathematical shapes similar to those in nature, with structures that are self-similar over many scales, the same pattern being repeated over and over. Fractal geometry offers a systematic way of approaching phenomena that look more elaborate the more they are magnified, and the images it generates are themselves a source of great fascination.

Mandelbrot always had a highly developed visual sense: as a boy, he saw chess games in geometrical rather than logical terms, and shared his father's passion for maps. Photograph: Nigel Lesmoir-Gordon

Mandelbrot first visualised the set on 1 March 1980 at IBM's Thomas J Watson Research Centre at Yorktown Heights, upstate New York. However, the seeds of this discovery were sown in Paris in 1925, when the mathematicians Gaston Julia, a student of Henri Poincaré, and Pierre Fatou published a paper exploring the world of complex numbers – combinations of the usual real numbers, 1, -1 and so on, with imaginary numbers such as the square root of -1, which Gottfried Wilhelm Leibniz had labelled "that amphibian between being and not being". The results of their endeavours eventually became known as Julia sets, though Julia himself never saw them represented graphically.

It was Mandelbrot's uncle Szolem who initially directed him to the work of Julia and Fatou on what are termed self-similarity and iterated functions. In my documentary The Colours of Infinity, shown on Channel 4 in 1995, Mandelbrot told me how he set about developing his approach: '"For me the first step with any difficult mathematical problem was to programme it, and see what it looked like. We started programming Julia sets of all kinds. It was extraordinarily great fun! And in particular, at one point, we became interested in the Julia set of the simplest possible transformation: Z goes to Z squared plus C [where C is a constant number. So Z times Z plus C, and then the outcome of that becomes a new Z while C stays the same, to give new Z times new Z plus C, and so on]. I made many pictures of it. The first ones were very rough. But the very rough pictures were not the answer. Each rough picture asked a question. So I made another picture, another picture. And after a few weeks we had this very strong, overwhelming impression that this was a kind of big bear we had encountered."

In his view, the most important implication of this work was that very simple formulas could yield very complicated results: "What is science? We have all this mess around us. Things are totally incomprehensible. And then eventually we find simple laws, simple formulas. In a way, a very simple formula, Newton's Law, which is just also a few symbols, can by hard work explain the motion of the planets around the sun and many, many other things to the 50th decimal. It's marvellous: a very simple formula explains all these very complicated things."

Mandelbrot, born into a Lithuanian-Jewish family living in Warsaw, showed an early love for geometry and excelled at chess: he later admitted that he did not think the game through logically, but geometrically. Maps were another inspiration. His father was crazy about them, and the house was full of them.

In 1936, the rise of nazism in Germany persuaded his family to leave for Paris, and eventually Lyon, in the south of France. A year of studying mathematics there after he left high school brought home to Mandelbrot his extraordinary visual ability.

At the end of the second world war he returned to Paris for college entrance examinations, which he passed with distinction, winning a place at the École Normale and then moving on to the École Polytechnique. From there he went on to the California Institute of Technology, Pasadena, to study turbulence and gain a master's in aeronautics.

After obtaining a doctorate in mathematics (1952) in Paris, he returned to the US, this time to the Institute for Advanced Study in Princeton, New Jersey. There he came across the idea of the Hausdorff-Besicovitch dimension – the revelation that there were phenomena that existed outside one-dimensional space, but in somewhat less than two dimensions. Mandelbrot took up the concept on the spot: it provided an all-purpose tool and was a special example of his eventual notion of fractal dimension.

Fractals have structures that are self-similar over many scales, the same pattern being repeated over and over. Photograph: Nigel Lesmoir-Gordon

His interest in computers was immediate, and his use of the new resource grew rapidly. He returned to France, married Aliette Kagan and became a professor at the University of Lille and then at the Centre National de la Recherche Scientifique in Paris. His academic future looked assured. But he felt uncomfortable in that environment, and in 1958 he spent the summer at IBM as a faculty visitor.

The company asked him to work on eliminating the apparently random noise in signal transmissions between computer terminals. The errors were not in fact completely random – they tended to come in bunches. Mandelbrot observed that the degree of bunching remained constant whether he plotted them by the month, the week or by the day. This was another step towards his fractal revelation.

During the 1960s Mandelbrot's quest led him to study galaxy clusters, applying his ideas on scaling to the structure of the universe itself. He scoured through forgotten and obscure journals. He found the clue he was looking for in the work of the mathematician and meteorologist Lewis Fry Richardson: he took a photocopy, and when he returned to consult the volume further, found it had gone to be pulped. Nonetheless, he knew he had struck a rich seam.

Richardson loved asking questions others considered worthless, and one of his papers, Does the Wind Possess a Velocity? anticipated later work by Edward Lorenz and other founders of Chaos Theory. One of Richardson's great insights was a model of turbulence as a collection of ever-smaller eddies.

Mandelbrot was struck too by Richardson's 1961 observations on the lengths of coastlines, and published a paper called How Long is the Coast of Britain? This apparently simple question of geography reveals, on close inspection, some of the essential features of fractal geometry. At IBM in 1973 Mandelbrot developed an algorithm using a very basic, makeshift computer, a typewriter with a minute memory, to generate pictures that imitated natural landforms.

While the ideas behind fractals, iteration and self-similarity are ancient, it took the coining of the term "fractal geometry" in 1975 and the publication of The Fractal Geometry of Nature in French in the same year to give the quest an identity. As Mandelbrot put it, "to have a name is to be" — and the field exploded.

He had bolstered his own presence by adding a middle initial that stood for no particular name, and Benoît B Mandelbrot became a fixture at IBM, with visiting professorships at Harvard and MIT, the Massachusetts Institute of Technology. He started teaching at Yale in 1987, becoming a full professor there in 1999. His many awards included the Wolf prize for physics in 1993.

Fractal geometry is now being used in work with marine organisms, vegetative ecosystems, earthquake data, the behaviour of density-dependent populations, percolation and aggregation in oil research, and in the formation of lightning. Lightning resembles the diffusion patterns left by water as it permeates soft rock such as sandstone: computer simulations of this effect look exactly like the real thing.

Fractals hold a promise for building better roads, for video compression and even for designing ships that are less likely to capsize. The geometry is already being successfully applied in medical imaging, and the forms generated by the discipline are a source of pleasure in their own right, adding to our aesthetic awareness as we observe fractals everywhere in nature.

Their beauty and power are displayed in the just-published book The Colours of Infinity, updating the account given in the film. In the course of making that and a further film with Mandelbrot – Clouds Are Not Spheres (2000, now a DVD) – I became aware of his great kindness and generosity. At the end of Clouds Are Not Spheres he reflects: "My search has brought me to many of the most fundamental issues of science. Some I improved upon, but certainly left very wide open and mysterious. This had been my hope as a young man and has filled my whole life. I feel extremely fortunate."

He is survived by Aliette and his two sons, Laurent and Didier.

• Benoît Mandelbrot, mathematician, born 20 November 1924; died 14 October 2010

• This article was amended on 20 October 2010. The original referred to the Institute for Advanced Study at Princeton University, New Jersey. This has been corrected.