In November, the Globe's first Broadway transfer opened in New York: two ensemble productions of Twelfth Night and Richard III starring Mark Rylance and Stephen Fry. The theatre is partly lit by candles and both actors and audience are bathed in the same light, with some audience members seated on the stage. Instruments and costumes are crafted using the same materials and methods as in Shakespeare's day. The productions aren't cluttered by concepts, they don't play any tricks, they aren't trying too hard to be clever. They just are what they are.

The popular response has been overwhelming and, hearteningly, the reviewers were equally thrilled. New York Times critic Ben Brantley wrote: "These productions are suffused with that most fundamental of Shakespearean virtues, faith. The performers here trust wholly in Shakespeare's words and in the ability of the audience to understand them." The enthusiasm with which American audiences and the east coast intellectual establishment have embraced this way of playing has served as a pleasing vindication of what we do at the Globe – but it has made me wonder, not for the first time, why things have been so different on our side of the Atlantic.

When the Globe first opened in 1997, the negative catcalling was almost deafening. In its inaugural production of Henry V, Mark Rylance stepped out on to the stage and delivered the prologue into a culture thick with scepticism and suspicion. Epithets such as "heritage" and "ersatz" rained down (along with a fair bit of actual rain). Critics wondered aloud if the space would ever be more than a "tourist-trap-cum-playpen-for-cranky-academics". The extremity of the reaction, thankfully not shared by an enthusiastic public (a public made up of only 20% overseas visitors by the way, to knock that one on the head – though why the English are so scornful of tourists has always bemused me). Subsequent success and artistic achievement have encouraged all but the most doggedly dull to change their minds, but the extremity of that original antipathy, much of it from the theatre profession itself, for a long time confused me. What was it that the Globe was doing that so unnerved some people? Why was it so important to keep this part of our theatre and national story so tightly under wraps? Why was history so challenging?

Despite the transforming insights of feminist, Marxist, queer and post-colonial theorists over the last few decades, much of our history is still written by the victors – and so our modes of behaviour in the present are dictated by a meagre and highly partial version of who we were in the past. But history does not happen in soliloquy. We need to crowd the stage of history with as many voices as possible in order to understand it in all its ambivalent, cacophonous diversity, and thereby learn to live with the same polymorphous perversity and trump those who would have us live thinner lives.

So what is it in the Globe that so upset theatre folk? Many of the answers are contained in the first words that Rylance spoke, in the prologue to Henry V. The prologue is a paean to collaboration and imagination. Shakespeare prefaces his great celebration of Englishness with an explicit acknowledgement of fictionality. He makes no bones about the fundamental absurdity of trying to stage the battle of Agincourt within the Spartan confines of the "wooden O". He begs the audience to "piece out our imperfections with your thoughts". Now consider Alfred Jarry's notes on the staging of Ubu Roi at the turn of the 20th century. He asks for costumes with as little specific local or historical accuracy as possible. He suggests that an actor enters at the beginning of each scene to put up a sign indicating its location, as in the guignols (puppet shows). And he directs that King Ubu is to wear a cardboard horse's head "as in the old English theatre".

Costumes for the new production of John Webster's The Duchess of Malfi. Photograph: Sarah Lee for the Guardian

What is Shakespeare's prologue but a prescription for avant garde theatre? In the 16th century, the original Globe demanded of its playwrights, directors and actors a freshness, honesty and ingenuity that by 1896 had come to seem dangerously radical – even in the relatively permissive atmosphere of fin-de-siècle Paris. In the intervening 400 years, this way of theatre-making had been gradually smothered. As the proscenium arch reared triumphal, the audience was plunged into darkness. They were told where to sit and what to look at. The machinery of scenery and props became more elaborate, and could be carefully concealed when not in use. Thick walls and velvet curtains excluded the world outside.

Within that proscenium, what happened on the stage was physically cut off from the audience. Where Shakespeare had written for a shared light where people discovered meaning together, now one group sat in passive darkness, being told where to look and what to think. And beyond the physical separation, a further more insidious separation took place, where one group became artists, and the other became non-artists; where the hieratically appointed initiates talked down to the lay people. A weird closed conversation began between directors, critics and academics, a conversation that short-circuited the central heart of any theatrical presentation, the audience. And since that conversation could only be about interpretation, since that was what had to be commented on, it elevated adaptation above actuality, the version above the real.

In 1997, the Globe was hardly the first space to challenge theatrical orthodoxy, but it was the first to return the event so wholeheartedly to the audience, and the first to do so in a way that felt so essentially English. A great many excellent things were being done in theatres all over the country, but there was perhaps a lingering sense that theatre-making was a European import. The Globe offered productions of Shakespeare in a single shared light, stripped of bells and whistles, productions that put power back into the hands of the spectator. The new "wooden O" housed a small revolution in which, night after night, benign dictatorship was swept away and democracy – some might say anarchy – was installed in its place. It reminded us that the greatest playwright who ever lived wrote for an audience capable of great joy and irreverence and sensuality, for a society in which different strata were constantly and publicly challenging each other, and for a space in which there was nowhere to hide. Understanding that we come from this tradition helps us understand who we are today.

It took an American, Sam Wanamaker, over two decades of tireless work to get the Globe rebuilt, encountering stubbornness, snobbery and scepticism at every turn. Sometimes we need an outsider's perspective. Wanamaker ultimately helped show us much about the theatre of the past – which can help us towards a bolder future – but also much about the English character, which had perhaps been lost in the great fog of empire and post-empire. That the Elizabethan and Jacobean eras were as much about the vitality and wildness of the Boar's Head (Falstaff's den of booze and wit and broken hearts) as they were about the decorum and intrigues of the court. That merriment is not just tankards and quaintness and mimsy Morris dancing, but a witty, angry and tender fire at the centre of Englishness. That the English – to speak in generalisations – were not simply the tortured inheritors of Victorian repression and Edwardian good manners: before that we were a robust and raucous people, happy to welcome outsiders, eager to embrace the new in any form, who would cheerfully congregate in all weathers to hear stories and to celebrate stories.

This week we opened a brand new theatre bearing Wanamaker's name. In many respects it is the "anti-Globe" – a small, intimate, indoor space, hand-crafted from oak and lit entirely by beeswax candles. The Sam Wanamaker Playhouse is based on the earliest known designs for an indoor English theatre, which lay dusty and overlooked for many years until they were discovered in Oxford in the 1960s. We haven't tried to replicate them perfectly – in fact, we came to believe that the building they depict was architecturally impossible – but what we have tried to do is to create an indoor playhouse that Shakespeare would have recognised.

These drawings have offered us yet another opportunity to gain exhilarating new insight into the plays of Shakespeare and his contemporaries, and the society they were written for, about and within. Rehearsing Webster's macabre tragedy The Duchess of Malfi in the Playhouse in recent weeks, the space suffused with flickering amber light, has been a lesson for all of us. The light produced by candle flames is organic, alive, permanently in slight motion. It forces the eye to see with different intensity and variety. It beguiles and intrigues, illuminating the textures of the actors' skin and the fabric of their costumes in a way that can feel a bit voyeuristic. I've been reminded of film director and artist Steve McQueen's compulsive and compelling materiality, that feeling of intense, almost uncomfortable proximity to the physical stuff of human existence. Or the eloquent folds of flesh in a Lucian Freud portrait. There is a shadowy eroticism to watching people under this light, that unmistakeable whiff of sulphur that eventually caused the anxious Puritans to close down the playhouses in 1642.

Beyond the physical, performing in that space has made us all newly aware of the absolute centrality of psychology in the Jacobean period. TS Eliot wrote that "Webster was much possessed by death/ And saw the skull beneath the skin" but in The Duchess of Malfi he goes even deeper – having peeled away skin and flesh, he cracks open the bone and delves into the darkest recesses of human psychology. In the second scene of the first act, Antonio and his friend Delio discuss the Cardinal, the former warning the latter: "observe his inward character". Later on Antonio gently menaces Bosola with the words: "I do understand your inside." The play is full of these moments – one character literally or metaphorically holding up a light to another, searching his or her face for clues to their true personalities, intentions and motivations. In fact many of the plays of Webster, Fletcher, Kyd, Beaumont – too often caricatured as unthinking bloodbaths – display extraordinary psychological acuity and a great relish for character.

Many of the excitements and delights of playing in the Globe came as a surprise – it's difficult to predict the outcomes of the kinds of dynamic conversations between past and present that happen there. We hope the conversations in and around the new Playhouse will be completely different, but just as enlightening. Whatever happens, the candlelit space will offer our audiences a new narrative about the past, and about how people felt, thought, spoke and wrote then. And with luck that new understanding of the past will help, in a modest way, to push forward to a richer and fuller future.

Mark Rylance recalls Shakespearean traditions and his first encounters with the Bard, from a school production of Macbeth to watching Olivier's Richard III on TV and seeing Judi Dench at the RSC in the 70s