Gillian Anderson as Miss Havisham and Oscar Kennedy as Pip in the new BBC adaptation of Great Expectations. Photograph: Nick Wall/BBC

The opening credits of BBC1's new three-part adaptation of Great Expectations (27-29 December) show a chrysalis cracking open to reveal a pair of trembling wings. A few seconds later this delicate emergence is replaced on screen by the escaped convict Magwitch (Ray Winstone) erupting from the stagnant waters of the Essex marshes. Covered in blood and slime, he is at once the monster of nightmares and a huge misshapen baby gasping its first breath.

In a single sequence, the director Brian Kirk gets to the heart of Dickens's novel as a fable of rebirth and renewal. Together with Sarah Phelps, the screenwriter, he has created a world in which characters are forever seeking to transform themselves – or each other. A spookily young Miss Havisham (Gillian Anderson), still cocooned in her tatty wedding dress, absent-mindedly scratches at her hands as if trying to free herself from her own skin. Pip (Douglas Booth) is introduced as a grimy apprentice blacksmith, receives a life-changing sum of money from his mysterious benefactor and rapidly metamorphoses into a velvet-clad dandy. But the scene of Magwitch rising from the water is also a playful acknowledgment of what underlies all attempts to adapt classic novels for the screen. A moving body breaks out of a flat surface; two-dimensional print gives way to the three dimensions of real life.

The fact that this scene reworks one of Dickens's most famous narrative openings is hardly a coincidence. No other novelist has been adapted for the screen so often or to such popular acclaim. Around 400 films and TV series have been made so far, and the number will rise again in 2012 with the release of another version of Great Expectations (directed by Mike Newell) and a two-part BBC2 drama from Gwyneth Hughes based on The Mystery of Edwin Drood. From lavish costume dramas to spin-offs and spoofs including Scrooged (1988) and The Muppet Christmas Carol (1992), not to mention movies such as It's a Wonderful Life(1946) that have lovingly ripped off his plots, Dickens's stories run though the history of cinema.

In the BBC4 Arena documentary Dickens on Film, made in association with the BFI, which is to host a three-month retrospective of TV and movie adaptations on London's South Bank, the producer Adrian Wootton and dramatist Michael Eaton make a strong case for thinking that Dickens was one of the fathers of modern cinema. They are not the first to make such claims. In a famous essay published in 1944, the Soviet director Sergei Eisenstein argued that "only very thoughtless and presumptuous people" believed in "some incredible virgin birth" of cinema, and that the film pioneer DW Griffith found many of his storytelling tricks, including close-ups, dissolves and cutting between parallel narratives, in novels such as Oliver Twist. Griffith admitted as much himself. One of his first films was a 14-minute version of Dickens's The Cricket on the Hearth (1909) that featured some early experiments with montage, and when he was criticised by his cameraman for employing the technique in a later film ("How can you tell a story jumping about like that?"), he is said to have replied: "Well, doesn't Dickens write that way?"

Like many origin myths, this all sounds a little too neat to be true, but Dickens was undoubtedly a key figure in the emergence of new ways of looking at the world. In Dombey and Son, the first time we see Solomon Gills he has red eyes from looking through the lenses of the optical instruments in his shop, and although he grumbles about being old-fashioned, he is plainly a man of his age. From cameras to kinetoscopes, when it came to visual entertainment the Victorians were spoiled for choice. Dickens had personal experience of the magic lantern, which projected illustrations from popular tales while the operator put these scenes into words, making stories come out of pictures rather than the other way round. He was equally fascinated by railway trains, which gave passengers unprecedented access to moving pictures, an ever-changing panorama framed by the carriage window. And while none of these inventions was as important to him as less sophisticated forms of entertainment, such as staring at the fire and imagining pictures leaping into life, once flickering flames were replaced by the flickering images of cinema, his fiction quickly found a new home.

Stories such as A Christmas Carol had long been familiar through theatrical adaptations, meaning that far more people knew about Scrooge or Tiny Tim than had ever read about them, and many of the earliest films set out to woo the same audience. At a time when moving pictures were still treated largely as a sideshow novelty, Dickens added ticket sales as well as a sheen of cultural prestige.

The surviving four minutes of Scrooge; Or, Marley's Ghost, for example, an ambitious British film made by RW Paul in 1901, are clearly modelled on a popular play by JC Buckstone first performed that year. Paul may even have used the same painted sets, and every aspect of his film, from the overblown acting style to the use of a camera fixed in the front row of the stalls, seems to have been designed to give each member of the cinema audience the best seat in the house. Yet even here there are clues that the film was trying to find a new way of telling the story. Marley enters dressed in a sheet, and then dissolves into a phantom through some impressive trick photography. Later a series of tableaux depicting "Scrooge's Visions of Himself in Christmases Past" are projected on to the living-room curtains, while Scrooge is powerless to interfere – a film-within-a-film that powerfully captures the experience of going to the movies and watching stories unspool to their inevitable end.

Later adaptations of Dickens made on both sides of the Atlantic show how quickly cinema developed. A nine-minute version of Oliver Twist (1909) again slims down Dickens's crowded pages into a series of tableaux, ending with a scene of Fagin in the condemned cell that is closely modelled on Cruikshank's illustration for the novel. Far from speeding up the story, the main aim of the director (J Stuart Blackton) seems to have been to slow it down, as if nervous of a medium that operated at the speed of light. Frank Lloyd's 1922 Oliver Twist, by contrast, while it retains authentic fragments of the novel in its title cards, cleverly switches between scenes of slapstick and pathos to create a visual equivalent of Dickens's "streaky bacon" narrative style (comic, tragic, comic). The film even finds a way of depicting Dickens's conviction that Oliver represents "the principle of good" untouched by his surroundings. Playing Oliver for laughs as well as tears, Jackie Coogan is less like a waif than a little vaudeville comedian unleashed on 19th-century London. He is as invulnerable to genuine suffering as a clown receiving a custard pie in the face.

Having recently starred alongside Charlie Chaplin in The Kid (1921), which had borrowed large chunks of its plot from Dickens's novel, Coogan was perfect casting as the tiny but tough hero. His success set a trend, because in the years that followed it became increasingly common for actors to reprise the same roles in different productions. Familiar stories required familiar faces. Francis L Sullivan played Jaggers in Great Expectations in 1934 and 1946, while Donald Wolfit went one better, playing Sergeant Buzfuz in The Pickwick Papers three times – 1952, 1955 and 1959. Such tactics rarely produced thoughtful or challenging drama, especially when directors treated Oliver's "Please, sir, I want some more" as an invitation to give audiences more of the same. The most startling example was Carol Reed's musical Oliver! (1968), which reproduced several scenes from David Lean's Oliver Twist (1948) almost shot for shot, but replacing Lean's gritty realism with a Technicolor world in which nobody seems capable of walking down the street without breaking into a complicated dance routine.

The development of TV costume drama in the 1950s and 1960s created a more subtle version of the same phenomenon: an imaginary "Dickensland" in which fog always swirled and bonnets always bobbed along murkily lit streets. It was an all-purpose Victorian age that was no more authentic than the plastic beams in Ye Olde Tea Shoppe. The episode of South Park that opened with Malcolm McDowell twinkling into the camera – "Ah, Dickens! The imagery of cobblestone streets, craggy London buildings, and nutmeg-filled Yorkshire puddings" – was only a lightly satirical version of much Sunday teatime viewing.

Attempts by film-makers to update this cosy ritual have rarely been successful. The practice of modernising Dickens began early, with The Death of Nancy Sykes (1897), a short sketch that turned the teenage prostitute from Oliver Twist into the more socially acceptable figure of Bill Sykes's – it should be "Sikes" – wife. Since then it has produced some memorably awful films, most recently Alfonso Cuarón's 1998 Great Expectations, which not only turned Pip (Ethan Hawke) into a struggling artist named Finn and renamed Satis House "Paradis Perduto", but was re-edited after some disappointing test screenings to make Estella (Gwyneth Paltrow) more likeable, which is to say less like Estella and more like a movie star.

The main alternative to updating has been to make adaptations increasingly realistic – a technique perfected by Christine Edzard's Little Dorrit (1988), where meticulously detailed sets and a soundtrack by Verdi form the backdrop to six leisurely hours of screen time. But even the glossiest production values cannot prevent accusations that adapting Dickens's novels strips them of all but their characters and plots, and that what happens on screen is inevitably a betrayal of the images in a reader's head.

Other adaptations have tried both to offer both a window on to Dickens's world and to hold a mirror up to their own. This can make for uncomfortable watching – or listening. Ralph Thomas's version of A Tale of Two Cities (1958) climaxes with a stirring scene in which Sydney Carton (a wonderfully laconic Dirk Bogarde) mounts the steps of the guillotine, but until then the film is largely a tale of two accents: Cockney for character actors and the crowd, and stiff-lipped RP for everyone else. While Thomas's decision to shoot in black and white undoubtedly adds a patina of age to his film, its sound is pure 1950s. Yet for all the difficulties involved in repackaging Dickens for the screen, the best adaptations prove that no novelist is better at encouraging the reinvention of tradition. The BBC's 2005 miniseries Bleak House reworked the novel's serial publication as the bite-sized chunks of a soap opera, creating a brilliant fusion of old and new forms of popular entertainment. (Dickens's contemporary Charles Reade memorably explained how to keep the public coming back for more: "make 'em laugh, make 'em cry, make 'em wait".) There have been several excellent cartoon versions of A Christmas Carol, such as Richard Williams's Oscar-winning 1971 feature, which came close to reproducing Dickens's own way of looking at the world, forever reshaping everyday reality to fit the needs of the imagination. Best of all, and a highlight of the BFI season, Lean's Great Expectations (1946)opens with an attempt to capture a child's vision, full of sudden close-ups and distorted perspectives, and ends with Pip (John Mills) ripping down the dusty blackout curtains in Satis House, crying out "I have come back to let in the sunlight!" In 1946 he might have sounded like a demobbed soldier returning home. In 2011 he sounds more like a screenwriter picking up Dickens's novel and setting to work.

• The Dickens on Screen season at BFI Southbank runs from Jan-March 2012. Arena: Dickens on Film will be shown on BBC4 on Tuesday 10 January 2012 as part of the Dickens on the BBC season.

Sign up for the Guardian Today

Our editors' picks for the day's top news and commentary delivered to your inbox each morning.