Planning on making a space in your hearts, or at least on your children's toy shelves, for Nigel the lovably eccentric koala from animated movie The Wild? Or how about Hammy the lovably eccentric squirrel from Over the Hedge? The lovably eccentric McSquizzy from Open Season? And there are many more celebrity-voiced, digitally rendered delights to come this year, jostling aggressively for your cash and affection. Thanks to Pixar's run, from Toy Story to The Incredibles, bolstered by a few Shreks and Ice Ages, animation has been one of the few movie sectors of the past decade that has consistently combined critical acclaim, commercial success and technological progress. But it looks like the golden age is over. Now there's an excess of product and a dearth of quality. Even Pixar could only muster middling reviews for its latest, Cars. But adults need not fear - for we may be entering a different golden age, of grown-up animation.

This week there are two animated movies on release in the UK with 15 certificates: Renaissance and A Scanner Darkly. Neither of them features talking animals, but both are lovably eccentric. The former is a French-made sci-fi thriller whose striking black-and-white visuals recreate the feel of a graphic novel. It's something of a niche product, but a quality one, with an English-language voice cast including Daniel Craig, Catherine McCormack and Jonathan Pryce. Richard Linklater's A Scanner Darkly could also be labelled sci-fi (it is based on a Philip K Dick novel) but it is a broader proposition - an unlikely mix of stoner comedy and paranoid thriller set in a near future where the vicious circle between drug users-turned-informants and undercover cops-turned-drug users is spinning so fast it's no longer possible to work out which is which, or who's high and who isn't.

Neither of these movies is animation in the same sense as Shrek or Cars. Rather than using 3D computer graphics, they both employ the technique of rotoscoping, in which footage is shot using live actors, then drawn over, frame by frame, to create realistic-looking animation. The technique has been around almost from the start of the movies: Max Fleischer used it from 1918 for his Betty Boop and Ko-Ko the Clown cartoons, and Walt Disney rotoscoped live-action models from Snow White and the Seven Dwarfs onwards. With the help of computers, both Linklater and Renaissance director Christian Volckman have moved rotoscoping into the computer age, so that it is now possible to achieve high-quality results relatively cheaply. Purists might not accept rotoscoping as true animation, but neither movie could have worked in live action - at least not without skiploads of extra cash.

Renaissance plays out in the Paris of 2054, where landmarks such as the Eiffel Tower and Sacre Coeur are still recognisable above the ramparts of a dense, futurist-influenced mega-city that we tour from top to bottom. Linklater had already used animation in his 2001 movie Waking Life, a roving philosophical enquiry set somewhere between dream and reality - something like Sartre meets the Simpsons. Similarly, in A Scanner Darkly, animation proves to be perfect for representing the unstable drug-addled reality of Dick's fiction. There are hallucinated insect infestations, characters' thought processes pop up like comic-book bubbles, and then there's the "scramble suit" that Reeves' character wears - an outfit that projects a constantly shifting collage of physical features, to preserve his anonymity. The actors are also recognisable as themselves, visually, rather than being mere voiceovers, and the animation rarely gets in the way of their performances.

Linklater might be ushering in a new age of grown-up animation, but he's by no means the first person to go there. If there's a precedent for A Scanner Darkly's hip, druggy, adult-oriented cartoon kicks, it would have to be Ralph Bakshi. A Jewish New Yorker with a background in children's TV cartoons, Bakshi alone carried the anti-establishment values of flower power and the New Hollywood into animated realms. In 1967, he created a counterculture classic in Fritz the Cat, the first X-rated cartoon. Based on Robert Crumb's comic books, Fritz the Cat certainly merited its certificate. Within the first 15 minutes, its feline hero initiates a dope-smoking orgy in a bathtub - between cats, rabbits and an aardvark - then steals a gun from a police officer (rendered as a pig, of course) and sets fire to his college. You can imagine the hippies of the time cheering it on, even as it satirised the hollowness and hypocrisy of their scene.

"A lot of people got freaked out," says Bakshi. "The people in charge of the power structure, the people in charge of magazines and the people going to work in the morning who loved Disney and Norman Rockwell, thought I was a pornographer, and they made things very difficult for me. The younger people, the people who could take new ideas, were the people I was addressing. I wasn't addressing the whole world. To those people who loved it, it was a huge hit, and everyone else wanted to kill me."

Bakshi went on to produce a string of cult movies using his own characters. Fritz the Cat's follow-up, Heavy Traffic, tackled the same mix of New York lowlifes, urban alienation, and racial and sexual tensions that Taxi Driver would come round to a couple of years later. Coonskin turned Disney's Song of the South into a broad-to-the-point-of-tasteless blaxploitation spoof, while 1981's remarkable American Pop traces four generations of an immigrant musical family, with a soundtrack spanning Gershwin, Hendrix and Pat Benatar. Bakshi had his share of noble failures, though, including his cartoon version of The Lord of the Rings, and his final project, 1992's awkward, semi-animated Cool World, starring Brad Pitt and Kim Basinger, which was effectively taken out of his hands.

"They rewrote the script while I was on location," he says. "They were going to sue me if I didn't do the picture. So I thought if I did the animation well, it would be worth it, but you know what? It wasn't worth it. So that was the last picture I did. The moment I couldn't say what I wanted to say any more, I left. I didn't want to make their dumb pictures." Now Bakshi lives in New Mexico. "I basically retired to paint pictures because I then didn't have to argue with anyone. But I have a lot to think about when I'm painting."

Therein lies the contradiction: in the right hands, animation is the perfect vehicle for subversion, but in the economic landscape of movie-making, the only way to make animation profitable is to pitch it broad. As Walt Disney proved, if you did it well, the rewards could be huge. Disney set the standard for high-yield family animation, creating an association between animation and family entertainment that precluded alternatives. A similar scenario has been playing itself out in the computer-age Pixar era, in which the investment in animation has been huge. The Incredibles, for example, had estimated production and marketing costs of more than $200m.

But costs have started coming down and it's becoming possible to target niche audiences again. Japan has been doing it for years, of course, with a broad universe of anime genres ranging from robot sci-fi to teenage girls' fantasy. You could say Hollywood, too, is leaning in that direction with vaguely child-unfriendly animations such as Corpse Bride or dark semi-animated fare such as Sin City, although the only truly subversive animation features to have come out of the US in recent years have been the South Park movie and its puppet-animated follow-up Team America.

Meanwhile an adults-only Danish animation, Princess, opened the Director's Fortnight at Cannes this year. Made by the cartoonist Anders Morgenthaler, it is a violent assault on the degrading power of pornography, in which a priest avenges the death of his porn-star sister. It features sexual abuse, child abuse, drug use and bloody violence. If you took a minor to see it, you'd probably be arrested. If Morgenthaler had filmed the movie in live action, he'd probably have been arrested, too. At the lighter end of the spectrum, there's Free Jimmy, an anarchic computer-animated comedy from Norway. Despite a voice cast including Woody Harrelson, Simon Pegg and Samantha Morton, it's essentially a crude, sub-Bakshi alternative comedy revolving around a circus elephant addicted to heroin. Its director, Christopher Nielsen cut his teeth on a Norwegian animated TV series called Two Wasted Wankers.

Bakshi is also at work on another animation, a detective story called The Last Days of Coney Island. Bakshi was turned away when he approached animation giants such as Pixar and Dreamworks with the idea, he says, but the technology is now advanced enough for him to make it on his own resources. The Last Days of Coney Island doesn't sound like it will shift many cuddly toys, but Bakshi, of all people, knows better than to be deterred by the prospect of not appealing to the mainstream. "It's probably higher quality than anything I ever made, at a cost so low it's embarrassing," he laughs. "Everything I used to do in my old movies that required hundreds of people and huge salaries is now done in a box. It took 250 people to make Heavy Traffic, now I'm down to five. I kiss the computer every morning - fuckin' unbelievable!"