With his snowy white hair, neat moustache and spectacles that sit low on his nose, Van Dyke Parks may look like a kindly shoemaker from a fairytale but don't mistake him for a soft touch. Between songs at the Borderline in London, the 70-year-old mocks rock critics who apply words such as "smarmy, quirky, idiosyncratic: adjectives that have lost their special charm to me". When we meet in an empty hotel dining room the next day, he peers over his glasses and says: "Inevitably you will want to use the word eccentric in your writing. If you run out of quirky."

Parks falls prey to such reductive shorthand because his career defies categorisation. He is best known for co-writing the Beach Boys' ill-fated Smile with Brian Wilson in 1966, an album that wasn't completed for another 38 years. Aficionados cherish his troubled 1968 fantasia Song Cycle, which established a lifelong pattern of huge acclaim and modest sales; his new compilation of seven-inch singles, Songs Cycled, has some of that earlier album's expansive magic. Younger listeners have discovered him via glowing endorsements from Joanna Newsom, Fleet Foxes and Grizzly Bear. Everyone else has probably heard his arrangements without knowing it, from The Jungle Book's The Bare Necessities to U2's All I Want Is You. He has often been around fame and wealth – he tells self-deprecating anecdotes about encounters with Bob Dylan and John Lennon – without accruing much of either commodity himself. He thinks that's probably for the best.

"When somebody blew John Kennedy's brains out in the face of his fame, I realised there might be value in anonymity," he says. "Celebrity did not have the maddening, fawning dysfunction it has now, but I saw clearly that fame could be an inconvenience and time has borne me out."

Van Dyke Parks: 'I saw that fame could be an inconvenience and time has borne me out.'

Compressed quotes can't give the full flavour of Parks's raconteurish charm; his fondness for looping tangents, pungent opinions, historical trivia, useful quotes, deadpan puns, colourful aphorisms and mischievous asides. He's like a well-loved college professor whose classes are lessons in far more than the subject at hand. At one point he gives me a business card which reads: "Mr Van Dyke Parks apologizes for his behaviour on the night of ————— and sincerely regrets any damage or inconvenience he may have caused."

Parks was born in Mississippi in 1943 (you can still hear a creamy Southern twang), a psychiatrist's son, and moved to Princeton, New Jersey at the age of nine, where he attended the American Boychoir School. One day Einstein visited and Parks sang in German while the professor played violin. He was also a child actor who appeared alongside Grace Kelly in 1956's The Swan. A lively childhood, all told.

Parks studied composition at Pittsburgh's Carnegie Tech, where he was taught by Aaron Copland and conducted by Toscanini, but became bored with contemporary classical music ("It ran the gamut from ebony to slate grey") and moved to Los Angeles in 1962 to join his older brother Carson in a coffee house folk group. Within months of his arrival two life-changing events occurred: the mysterious death of another brother, Ben, who worked for the US state department in Germany, and his first professional job, arranging The Bare Necessities.

"It paid for us to go to our brother's funeral and buy some black suits," says Parks. "That first cheque, with Mickey Mouse waving a three-fingered glove at me, was my introduction to music as a profession."

Parks fitted in everywhere and nowhere. He was well-connected enough to get Carson's song Something Stupid recorded by Frank and Nancy Sinatra. He was briefly in Frank Zappa's Mothers of Invention but quit "because I didn't want to be screamed at". He bonded with the likes of Randy Newman and Harry Nilsson, fellow members of what he calls "the counter-counterculture". He still prefers classical music to most rock. "I listened to all the waltzes of Chopin on the flight over. Oy! It beats fellatio in my book, and that's saying something."

What should have been his big break came when Brian Wilson invited him to work on Smile, but it was doomed by Wilson's drug-damaged fragility and the bullying hostility of Beach Boy Mike Love. Parks brings it up unprompted, then wishes he hadn't. "It's a dull issue," he growls. "I hope it doesn't need any further elaboration. To have been victimised by Brian Wilson's buffoonery."

The words "victimised" and "buffoonery" seem somewhat harsh in the context of Wilson's mental illness, but Parks doesn't care to elaborate, other than to say: "It just got too much for me. It was an expensive decision for me not to continue my association with the most powerful artist in the music business at the time, but I made the only decision I could. I walked away from that funhouse."

He concentrated instead on his solo debut, Song Cycle. His instincts were partly political, delving into America's musical history in order to shed light on the era of civil rights and Vietnam.

He remains inspired by his old friend, the late protest singer Phil Ochs. "I still think the song form is the most potently political tool that we have in our kit. Learning from the past demands that music have a retrospective aspect and it must migrate what we know into the future through a contemporaneous lens. Creating that perfect lens is the songwriter's dilemma." He grins. "Doesn't this sound hifalutin?"

Skrillex. Photograph: Bruno Postigo

But Song Cycle also stemmed from grief and emotional torment. "It's fair to say that I was in psychological collapse – not so evident, perhaps, as the one Brian Wilson was in but I had to work my way through it. It's good work for a 24-year-old boy. We're not talking Bach here but then again it's not Andrew Lloyd-Webber. It's an entirely individual effort, and I think terrifically entertaining, but you can hear that it's highly troubled. You cannot accuse a songwriter of being guarded. In spite of his or her best intentions, the song is so fucking transparent."

He's still furious with Warner Brothers copywriter Stan Cornyn, who made light of the album's commercial failure in a notorious press ad titled: "How we lost $35,509.50 on 'The Album of the Year' (Dammit)". "I believe the music was beyond Warner Brothers' comprehension and flippancy was the only conclusion that they could come to," snaps Parks. "And it hurt."

Bruised, he edged towards the sidelines. Although he has released seven solo albums since, including 1972 calypso travelogue Discover America and 1984's Brer Rabbit-inspired Jump!, he's been much busier as a producer (Randy Newman, Ry Cooder, Joanna Newsom), arranger (Rufus Wainwright, Saint Etienne, Scissor Sisters) and composer for film and TV("By being 10ft from the mic I found some startling perspective. I see greater definition in twilight than I see in Hockney light."). Even so, it was still surprising to hear of him working with US dubstep kingpin Skrillex last year.

"I had two choices," he recalls. "One is to remember that I'm above this kind of shit, and the other is to remember that I'm not above this kind of shit and what I have to do is serve this man who represents people who couldn't give a rat's ass about an orchestra. I've never had a harder job. It was the most fascinating, epiphanic, transformative experience that I've ever had."

Skrillex is hugely successful but generally Parks is drawn to tales of valiant failure. He scored The Company for Robert Altman, who drily told the orchestra: "My only regret is that more people played on this picture than will see it." In the mid-90s, he reunited with a reclusive Brian Wilson for the album Orange Crate Art. "When I found him he was alone in a room staring at a television. It was off. It took three years and $350,000. The record came out and sank without a trace." But during a promotional interview Parks mentioned his love for long-forgotten French group Les Compagnons de la Chanson and later received a thank-you letter from one aged member. While telling the story, Parks's voice trembles and he removes his glasses to rub his moistening eyes. For the first time he looks all of his 70 years. "Goddammit!" he rasps. "That's why I did that record."

Parks admits a higher profile would be nice because he'd love to play more concerts if the demand was there. "Robert Frost said an empty auditorium is a poet's nightmare. I understand what he's saying." Other than that, he thinks he got what he wanted. "Anonymity has been pretty good to me," he says, knitting his fingers together. "I think of old age as the same twilight – a value in being relieved of light pollution. You start to see the fundamentals in the sky. And that's where I am now. I can see the constellations."