Thirty years ago Gary Tovar ran California's largest marijuana operation, smuggling famous varieties such as Acapulco gold and Thai stick into the US and distributing Afghan seeds to growers in California to kickstart the homegrown industry.

A music fan, he took the name of one strain, Goldenvoice, for the name of his concert business which specialised in bringing British punk bands to Southern California – the same company that this weekend will put on America's answer to Glastonbury on a polo field outside Indio in the Mojave desert.

Tovar, now close to 60, will be at the Coachella Valley Music and Arts Festival as he is every year, enjoying the spectacle from his all-access golfcart. Coachella is the most profitable festival on the US circuit and now wildly popular with the Hollywood set. The LA Times calls it "more than an event, it's a state of mind" and "the biggest gift to fashion since Kate Middleton."

Tovar is understandably chuffed. He may no longer be active in either business but he considers the festival and the marijuana legalisation movement gathering momentum in the US a validation. Since recreational use of marijuana was legalised in Colorado and Washington in January, a dozen more states have indicated they are looking at introducing at least partial legalisation or medical use measures.

"Yes, you could say I feel gratified," he says over breakfast at Phillipe's, a downtown LA institution with woodchips on the floor. He looks every inch the renegade, from his shades to his stripped shirt.

"When I was doing both my things – smuggling and concerts – I considered them crusades. Now I think we won on both ends. Our music won – you can hear a Ramones song in an elevator – and we won on the marijuana front."

Tovar shows me his California medical marijuana card. The transition to legality marks the end of an era, and no one is quite certain if the hippies or the squares of the 60s culture war prevailed.

According to Marijuana Business Daily, the legal marijuana business is projected to grow to $8 billion by 2018. While purists don't begrudge the availability of the drug, they worry the legalisation is ushering in a business dominated by intensive agriculture techniques, grow-boosters and genetic science – a far cry from the hippie ideal of organic, soil-nourished weed cultivated in sunlight and wind.

Tovar is concerned that the old growers are being pushed out. In this "free for all", the business could be taken over by "opportunists with no regard for the principles or cultural roots of the industry in its illegal manifestation".

Still, his devotion is unwavering. He recalls how a US serviceman on R&R from Vietnam delivered a knapsack filled with Thai stick to him in Hawaii and how Acapulco gold was burned yellow by the wind coming off the Pacific. The older the pot got, the more it faded into a wheat colour. "Americans are obsessed with marijuana, and we were obsessed on finding the best," he says.

Tovar started Goldenvoice (named after a strain of cannabis said to produce the sound of angels singing in your head) three years after his sister took him up to San Francisco to see the last Sex Pistols show. He was smitten. He wore a badge inscribed "Move Over Hippies, Your Time is Up" and used smuggling profits to bring British bands like Public Image Ltd, Siouxsie and the Banshees, GBH and the Anti-Nowhere League to Los Angeles. Orange County, where bands like The Dead Kennedys and Black Flag were popular, proved especially fertile. The suburban kids were bored and the police were suppressing punk shows – a recipe for punk success.

"The hippies were getting old and turning into yuppies. They behaved toward their children as their parents had to them. San Francisco wanted its own scene but southern California embraced the British bands."

Tovar visited the UK to scout for acts, found 25 he liked and invited them over to play in intervals. Californian punks even started their own dance – the slam dance. "They went up and down in England, we went sideways." At its peak, during the 1984 Los Angeles Olympics, Tovar put on an international festival of punk at the Olympic Auditorium.

But he never turned a profit. Punk's lifeline was cash from cannabis, and music provided a way to wash the proceeds from the trade. He kept it in a floorsafe so packed he stepped on it to close it. "I never had to go anywhere for money. I had all I needed."

"I lost $4 million dollars – and I mean 1980s dollars – trying to break this music. But I was torn. I didn't want it to become big because I didn't want to see it watered down."

At his peak, Tovar, who had started smuggling fireworks in the US from Tijuana aged 14, didn't merely limit his business to importation. He guaranteed a market and distribution to growers in Humboldt and Mendocino counties, a relationship that had begun in the late 60s when he distributed 50,000 Afghan seeds he received from his guru, a man known only as LaRue, a member of Timothy Leary's Brotherhood of Eternal Love who had famously smuggled hashish from East Asia in hollowed-out surfboards.

Despite misgivings about music he loved going mainstream, he need not have feared. "Punk was like a shooting star and after 85, I started losing money and it got too violent. I pulled the plug."

By then, however, the Feds were catching up. "It was Ronald Reagan's 'Just Say No' era. There was hysteria. They were telling schoolchildren to tell on their parents. It was a total waste of time and money, and a waste of people's lives."

Tovar served seven years in prison. Goldenvoice continued with Paul Tollett, his business partner, at the helm. It launched Coachella in 1999 with Morrissey and Beck on the bill. The event lost money but the notion of an annual desert festival with a communal atmosphere took hold. Each year, organisers aim to reunite an cherished act. Last year it was the Stone Roses; this year the honour falls the Atlanta rap duo Outkast.

Nowadays, Tovar does what he pleases – but nothing illegal. He calls it "English Fun".

But he still wonders what took us, or at least the authorities, so long to recognise the benefits of music and marijuana. "I served 10 years in prison but they should have given me a parade down main street."