When Anish Kapoor and his long‑term collaborator, the designer, architect and engineer Cecil Balmond, agreed to enter the competition to create a sculptural tower for the London Olympic Park they were well aware of the potential pitfalls. "There have been a lot of them," laughs Kapoor. "In fact, there is actually a history of famous towers, so you can't just pretend they don't exist. They've been made for good functional reasons and for other, less functional, reasons. They exist in reality, with the Eiffel Tower remaining the most iconic structure built for a festival, but they also exist in the imagination, so in some way or other we had to take on Tatlin" – whose never-built constructivist tower was intended to celebrate the Russian revolution. "I was also pushed towards thinking about the Tower of Babel, in terms of the participation of people, making a processional meander upwards, being integral to its completion."

Kapoor's ArcelorMittal Orbit, the name reflecting the £19m contribution made in cash and steel to the total cost of £22m by steel magnate Lakshmi Mittal, was officially unveiled just before the 2012 Olympics. At 114.5 metres tall, it is the largest sculpture in the UK and can command views of more than 20 miles. The work comprises a central trunk that houses the lifts and viewing platforms around which curl massive loops of knotted red steel. "There was a lot of pressure to have it in grey like everything else in the park," explains Kapoor. "So we had a quarrel and then it was made in red as we wanted." And not just any red. When the painted steel was first delivered it was coated in a colour with a catalogue number of RAL 3002. Kapoor sent it back to be painted in RAL 3003. It is surprising there is not already a shade called "Kapoor red", so often has he used the colour in his work over the years.

The initial critical response was decidedly mixed and included an attempt to dub it the "Eyeful Tower" – intended as an insult but a phrase that could work equally well as a compliment – and some sniggering about a badly made helter-skelter. "There were also a few who said it was the most extraordinary structure ever," adds Kapoor. "But there were indeed more saying it is an absolute horror. And I'm not blaming people for negative views. We are pushing at the edge of something and these things can take some time. However, because of the way things went in the runup to the Olympics, an awful lot of those criticising it would only have seen it from afar and it is a very different thing when you're up close."

Anish Kapoor in his Camberwell studio. Photograph: Graeme Robertson for the Guardian

Since the Olympics and Paralympics – when 130,000 people journeyed to the top to see a view that provides a revelatory perspective on the eastward development of London over the last few decades – the sculpture has been closed as the site was redeveloped. But on 5 April the south section of the new Queen Elizabeth Olympic Park reopens, and there will be another chance to see the piece close up. The Orbit joins an increasing number of Kapoor's large public works that are dotted around the world. His Sky Mirror has been shown in Nottingham, New York and London; Cloud Gate, a 100‑ton stainless-steel sculpture, is in Chicago, where it is known locally as "The bean"; and an ongoing project on Teesside, Temenos, features large-scale steel structures, and will include "a tower or two" by the time it is finished. Most recently, Kapoor has filled the Grand Palais in Paris with his giant, inflatable, Leviathan.

But while the scale and profile of these projects seem to fit with an artist who turned 60 this year and who features on the Sunday Times rich list (estimated worth £100m, although he was not paid for the Orbit, "these public objects never make you any money"), his progression to such bold spectacle has been relatively slow, and his early career triumphs – at the 1990 Venice biennale and as winner of the 1991 Turner prize – were based on more physically modest work extending back to simple mounds of pigment on the gallery floor.

"But colour was always important to me in a very particular way," he says, "as a kind of immersion, a single absolute condition that inevitably has the sublime somehow associated with it. Then somewhere along the way I discovered that scale and the sublime are also closely related. The sense of awe, or fear, or whatever it is that one can feel when immersed in colour, one can also feel with something big. It is like standing on the edge and looking over a cliff. You can't not do it. Is that a place for art to go? It seems to me yes."

A key work was Kapoor's 2002-03 installation in London's Tate Modern Turbine Hall, Marsyas, his 10-storey high, 120-metre-long deployment of red PVC membrane stretched between huge metal hoops. Working with Balmond he evolved a new technical language while retaining his longstanding psychological and artistic preoccupations with shapes that fold and enclose and hide and reveal. Or as one interview with him began: "Anish Kapoor is extremely keen on vaginas … Here a chasm, there a crack, over there an abyss that takes you plunging into a void. This, clearly, is a man who's read his Freud." "How can I put it?" he laughs. "I'm endlessly obsessed with this question of the interior. In terms of form there is only, really, convex, concave and flat. Everything else is either inside or outside. It seems to me such an obvious thing. We live in a world of night and day, male and female, inside and outside. A world of opposites. Of course I was in psychoanalysis for 20 years, so some of this stuff did come up."

Now out of analysis, he says his studio practice exhibits many of the same characteristics. After working for 25 years in a street of industrial units in Camberwell, south London, he has acquired six of the seven – the other is used by fellow Royal Academician Bill Woodrow – where you can find his 30-odd staff, some in hazchem suits with breathing masks, moving among large chunks of Spanish alabaster, piles of huge half-spheres reminiscent of a crazy Jodrell Bank, and flayed canvases of silk and hessian daubed with thick dollops of dark-red paint and resin. They work on a variety of projects that involve applying corrosive solvents to blocks of polystyrene, polishing huge reflective pieces of stainless steel and making cement shapes through giant icing guns.

"Although I do make a lot of things, my studio practice is not really geared towards the making of things," Kapoor explains. "I am much more interested in process and experiment and possibilities. It is not unrelated to psychoanalysis in the sense that you lie there and talk your stuff and suddenly there is something in the room and you're asking, what's that? Where did it come from? What does it have to do with? It's the same in the studio."

But how does this content and these practices transfer to a mass public audience for whom the artwork might be akin to a tourist attraction? "The element of participation is not just OK, it is essential. You want people to say: 'You have to look at this thing', and for them to go and see it again and again." He says when Cloud Gate, the massive reflective bean, opened in Chicago to huge audiences he did worry about the difference between it and Disneyland. "Traditionally, we are very suspicious of anything popular or attractive in serious culture, so I went every day to the site to watch people. And why the work is not Disneyland comes from its very strange ability to shift scale: because of the way it is made you can't tell if it is big or small. When you are near it is this massive thing, and just a few feet away it seems not so big. What happens is very mysterious and, while it can be pretentious to talk about the sublime, something of the sublime is in that sense of the ungraspable. That is the difference between an art experience and a thrill, and a piece of public art, or whatever you want to call it, that can do something like that is completely worthy of a life's endeavour."

He hopes there is something of this in the "awkwardness" of the Orbit. "I can do sleek forms, but this is a determined and deliberate process and there is no doubt that the Orbit is an odd and strange form that doesn't sit comfortably, and part of what I'm interested in is that it doesn't sit comfortably." Taking off his spectacles, he explains the groundbreaking engineering by manipulating the arms to illustrate how "one piece of collapsing spaghetti props up another" resulting in a sensation of instability and an aspect that changes as you move round, and within, the structure. "The public won't know about these innovations behind the scenes and in the guts of it, but I think they will sort of sense them in the unusual elbows-sticking-outness of it. And I hope that quality of awkwardness is the thing that will give it life."

Looking at contemporary architecture, he says that the Orbit's "hybrid status" is increasingly common. "It is probably best described as a sculpture because it is not a building, but it does this building-type thing. There is also the same question around Frank Gehry or Zaha Hadid as to what the status of their buildings are. Describing a Gehry as a Cubist object viewed from the interior may well be a useful way of talking about it. Hadid's work is more and more sculptural. The lines are blurred and I think in the future, partly because of computer technology, the status of big form making, let's not call it sculpture or architecture, will be increasingly confusing and, correctly, disputed."

And as to whether the Orbit will be understood, and even loved one day, he says time will tell. "You try your best to control something, but there is a moment when you have to hand it over and let it go. I wish now it was bigger, but I think that is a common thing from people who have made new structures. I don't worry about taste as we are never fully aware of where taste eventually goes. Mostly I am pleased that out of such a complicated and political thing as the Olympics, we managed to build this thing, awkward as it is, and I hope its problematic aspects will be what matter most in the end. But we'll see."

•The Orbit will be open daily from 5 April at the Queen Elizabeth Olympic Park, London E20.