Bankers on stage in The Power of Yes. Photograph: Allstar/Cinetext/Rizzoli

This month I celebrated a melancholy anniversary. It was 40 years since the premiere of my first full-length play at the Hampstead Theatre Club on 6 April 1970. Those old enough to remember will know that the prefabricated building was moved first from one side of the Swiss Cottage car park to the other – and then back again. Somewhere in transit the word "Club" dropped from the shingle. In other words, in four decades, theatre culture has changed, if not out of all recognition, at least significantly.

One further example. If you set to writing plays in the postwar years, it was necessary, or at least expected, to pass through a portal of approval. In prospect, this gave a comfortable, orderly feeling to the idea of being a British dramatist. Kenneth Tynan, a humanist dandy, guarded the portal on one side from his position at the Observer. Harold Hobson, a Conservative Francophile whose life had been changed at the age of 10 by the sight of a Bible in the illuminated window of a Christian Science church, guarded the other side from the Sunday Times. A novice playwright had every reason to expect that a life in the theatre would involve attracting and then retaining the interest of at least one of these two men. Hobson's name was inextricably linked with Beckett's and with Pinter's. Tynan's fortunes rose with his advocacy of the work of Osborne. These were the writers they championed and whose view of the world fired them up. They were interlinked by a profound correspondence of belief. Today, no such correspondence exists. No living theatrical figure is associated with any particular critic. Tynan, just turned 83 years old had he not been taken by emphysema, would be devastated to know that to work seriously in the British theatre it is no longer necessary even to know the name of the Observer's theatre critic.

Some people have understandable nostalgia for what then felt like a common culture, even if, over the years, bitter experience left few practitioners with much trust in those delegated to be its guardians. In fact, the growth of diversity both in the audience and in the places it sharpens its opinions has brought only benefit to any dramatist whose first love is experiment and innovation. And newspapers that once enjoyed such power are themselves discovering what it is like to live with the threat of working in a minority form.

Throughout the 1980s, propaganda for the free market aimed to reach out into many spheres beyond the economic. The aims of the revolution were cultural as much as political. Norman Tebbit was recently asked why the Thatcherism in which he had played such an important part had created a society he so heartily disliked. Tebbit replied that Conservative governments of his generation had taken on such a massive task in fixing the economy that they had had no time to fix the culture. He was being too modest. In those days, in my memory, there was certainly no shortage of fellow-travellers happy to direct their fire against the most communal of art forms. For years no Murdoch paper let a week go by without some loyal employee railing against their own definition of elitism – elitism often being represented by the state-subsidised British theatre.

Today, such attitudes themselves look dated. Having so long prophesied The Death of the Theatre, the prophets have woken to find themselves writhing in the coils of a problem rather closer to home: The Death of the Newspaper. What a reversal of fortune! If the free market is indeed the moral courtroom that its admirers claim, then what a judgment is being visited on Fleet Street. What a pack of failures the editors must be! No artistic director of a theatre could survive such a plummeting loss of income and popularity without being sacked by their board. Surely it must be, according to those iron laws of the market which newspapers have done so much to propagate, that consumers are today buying fewer newspapers because those newspapers are poor products. The people writing for them must be no good at writing.

The obvious absurdity of this proposition, and its roots in that same false logic which has had such currency throughout the Thatcher-Blair ascendancy, should not divert us from more interesting questions about the relationship between art and journalism. When asked to name the best British film of the last 25 years, without thinking I nominated Adam Curtis's documentary series The Power of Nightmares, which examines how politicians have exploited the so-called war on terror in order to transform themselves from managers to saviours of their nation. My mind went straight to a BBC documentary because The Power of Nightmares seemed imaginatively more ambitious than most better-known fictional works. If we give Curtis's film a full-body scan, it passes some of the most traditional tests of art. For a start, nobody could miss the fact that it had been created by one exceptional imagination. Behind its images lay some rich associative thinking. It advanced a way of considering Western leaders which made you see familiar figures in a new light. But if you also investigated its technique, its use of scrap footage – from advertising, from feature films, from training films, as well as from more usual documentary sources – it not only adopted the methods of some of the 20th century's most important visual artists, it also attained some of their haunting strangeness.

You must not think that I sharpen all my aesthetic thinking by attending to Norman Tebbit, but on another occasion, Tebbit showed impatience with some fellow guests in a radio studio by declaring that he was tired of hearing about the claims of art. In his view, a Rolls-Royce aeroplane engine was far more beautiful than most things living artists had created. Why was an engine not a work of art? There are certainly many different answers to his question – plenty of people would say it was – but my personal response would be that an aeroplane engine is an object without metaphor, and without metaphor we have no art.

In another provocative statement, Waldemar Januszczak, then head of arts programming at Channel 4, went further than Tebbit. He proposed that there was no need any more to make television programmes about artists because a television programme was itself a work of art. Television had spent too long in the position of a waiter, bringing art in to the viewer on a silver salver. In the modern world, he said, television had no good reason to continue feeling subservient. Fawning profiles of artists were old hat. Television did not have to illuminate Bacon. It was Bacon.

Of course, such economically convenient reasoning at once blew through the offices of the channel controllers. Words cannot convey how delighted they were with this cute piece of post-modernism. What a rare pleasure to be let off an expensive hook by one of their own. Television no longer needed to bring news of art. Strange this, because Ken Russell did not seem in the least subservient when he made his TV films about Delius and about Elgar. Russell seemed all too happy to accept these composers as masters, and then, in his turn, to demonstrate what their mastery had released in him. Namely, mastery.

The example of Russell making such brilliant television out of brilliant music might, you think, once and for all dispatch certain questions of category. But there is still plenty of dismaying evidence to prove how closely conjoined art and snobbery will always be. Over 40 years ago Philip Roth put his finger on a problem which was going to challenge the novelist's profession as much as mine. He wrote: "The actuality is continually outdoing our talents, and the culture tosses up figures daily that are the envy of any novelist." It would be an unusual artist who did not feel that this difficulty had deepened in the new century. I once observed that it was always valuable for a writer to go out and be rebuked by reality. But recently we have not been so much rebuked as overwhelmed. As a working dramatist, I have written plays about the privatisation of the railways, about the diplomatic process leading up to the invasion of Iraq, about Labour Party funding, about Foreign Office complicity in torture and about the financial crisis. But this kind of timely writing which seeks, as Balzac's work once did, to provide society with its secretarial record, continues to attract reproach from those good souls who believe that the results cannot be regarded as "proper" plays – in the sense, say, that Sophocles or Racine wrote "properly".

Particular objection is made to the use of other people's dialogue. No sooner had a genre called verbatim drama been identified than sceptics appeared arguing that it was somehow unacceptable to copy dialogue down, rather than to make it up. People who did this, it was said, are called journalists, not artists. But anyone who gives verbatim theatre a moment's thought – or rather, a dog's chance – will conclude that the matter is not as simple as it first looks.

In the autumn of 2006 I was working in New York. My luck was that I could go to the Lincoln Center Theatre to see Jack O'Brien's superb production of Tom Stoppard's trilogy of plays, The Coast of Utopia. After the first evening, we were set fair. Here was the kind of epic work which wove the lives and ideas of 19th-century Russian philosophers into the overall movement of history. It was exhilarating and it was also well made. Yet at the climax of the second play the audience was jarred out of involvement. After Alexander Herzen's son Kolya was killed in a shipping accident at the age of five, Stoppard had given his father a speech in which he argued there was no need to mourn or be upset. We do not ask of a lily that it be built to last. "The death of a child," Herzen is made to say, "has no more meaning than the death of armies, of nations. Was the child happy while he lived? That is a proper question, the only question."

Stoppard is self-evidently a humane and decent writer, but I came out of the production seething at his callousness. How could a child's life properly be compared to that of a flower? Really, this was taking the religious view of things too far. Even a priest mourns. However, when talking a few days later to a member of the production team, I was gently put right. The words were not Stoppard's. They were Herzen's. The dramatist had felt compelled to include them, against objections, because he had wanted to give full rein to the character's unpredictability. He had included Herzen's words precisely because they seemed so surprising. But nowhere in the coverage of The Coast of Utopia did anyone try to belittle it by labelling it a verbatim play.

You may say that I had committed nothing more than the vulgar crime of ascribing to an author the opinions of one of his characters. What a crass mistake for a fellow playwright to have made. But you may also observe that the inconsistent detail, the confounding, non-fitting fact, is often the giveaway mark of material drawn from life. In James Marsh's documentary Man on Wire, Philippe Petit celebrates his achievement of spending 45 minutes crossing a steel cable suspended 1,350ft above Manhattan between the twin towers of the World Trade Centre by sleeping with a previously unknown admirer in the street. By doing so, he destroys his long-term relationship with the partner who has helped him plan his feat in the first place. Our response is to exclaim: "Oh my God, this must be true. It must be true because it's so unlikely." Or maybe we use that other comforting phrase, so hated by all writers of fiction: "Nobody could make this up." Asked for his reasons for making the trip, Petit replied: "When I see three oranges, I juggle. When I see two towers, I walk." But significantly he omitted to add: "And when I come down from two towers, I make love to the first woman I see."

"Is this true? Is this a true story?" is a question you hear asked frequently in cinemas. Before a film a message regularly appears: "This is based on a true story." This functions as a kind of prophylactic, a way of protecting the subsequent proceedings from undue criticism. By declaring in advance that something is true, the film-makers seek to absolve themselves from the highest demands of art. The implication is that, if it's true, it must be interesting. The updated CV piously tacked on to the end of the film – telling us that the tousled hero is today running a brothel in downtown São Paolo, or that the lovable heroine overcame her crack habit to open a small bakery in Montecito – is also intended to ward off our more spontaneous reactions to the film itself. It acts as a sort of dousing of the flame of our potential dislike.

"You cannot improve on the facts," Hilary Mantel has said in a ringing defence of accuracy in historical fiction. But Mantel was most certainly not arguing that it was enough to offer only the facts. Nor is it a guarantee of importance in art that the art directs itself to an important subject. Every Sunday, like so many students in the 1960s, I fell out of bed and read Kenneth Tynan. But his biggest misstep surely came in the week when he reviewed Peter Watkins's film The War Game. Because it speculated on the effects of a nuclear attack on a typical English city, "it might," Tynan said, "be the most important film ever made . . . We are told that works of art cannot change the course of history . . . I believe this one might." Its screening, he continued, made it impossible to take seriously the other films released that week. An Italian director had come up with "an ecstasy of chic self-indulgence, a gigantic parcel which conceals beneath its gaudy wrapping, a fragment of old rope". How could anyone bother with "an immensely pretty film of monumental triviality" when it was placed beside a harrowing vision such as The War Game?

Now admittedly, Juliet of the Spirits is not La Dolce Vita. But nor is it nothing. It is a film to which you may return many times for pleasure and instruction. And yet, oddly, in their characteristic mix of autobiography, of fiction and of fantasy, Fellini's films raise the very same question I wish to ask. How can there be a wrong way to make good art? And, indeed, what point does criticism serve when it asserts only "This is not the sort of thing of which I approve"? When a literary critic such as James Wood twists himself into a pretzel explaining exactly why the novel he has under review is the wrong kind of good novel, he sounds like nothing so much as a Railtrack official railing against the wrong kind of snow. All the rules dictate that, in 8½, Fellini ought not to be able to construct a counterintuitive masterpiece out of his own pampered indecisiveness. But he does.

All over the world serious work is being made in all sorts of unauthorised ways. Old-fashioned opinion, meanwhile, is tying its shoelaces and not noticing. In the face of the evidence, it is still held as an article of faith by high-minded bystanders that it takes time for artists to absorb events. Any response that appears too quickly must, it is claimed, be journalism, not art. The fact that Wilfred Owen wrote the greatest poems of the first world war in the heat of battle does not shake the prejudice. If the high-minded had their way, Owen would have waited to lend the events more distance. He would, mind you, have been killed in the meanwhile, and his poems would never have got written, but at least Owen would have died with the consolation of knowing that he did plan to compose on a critically approved timescale. Addressing a similar conviction – that films about Iraq and Afghanistan are bound to be flawed because they lack perspective – the critic David Denby asks this excellent question: "Box office wisdom holds that it is too early to make movies about this conflict; but how can it ever be too early to make a good film?"

It is this question of prematurity – "Hold on, I'm not ready for this" – which bedevils the reception of any work on a contemporary subject. It is very hard in these circumstances to ensure that the question the work provokes is neither "How soon has it been made?" nor even "What has it been made from?" but the far more lasting question: "How deep does it go?" I said earlier that the example for my recent plays had come from Balzac – "What was it like living under New Labour? How did Tony Blair seem when he went to war?" But I need to make clear that I therefore regard The Permanent Way, Stuff Happens, Murder in Samarkand, Gethsemane and The Power of Yes as something entirely different from journalism.

Journalism is reductive. This is not always the fault of journalists. It is in the nature of the job. At its best and worst, journalism aims to distil. It aims to master, even to subjugate, a particular topic. In this ambition, the journalist will always run the risk of tipping over into contempt. As soon as something can be summarised it can also be dispatched. Anyone who has ever attended morning conference at a national newspaper will know the form: everyone taking part in the human comedy is a fool. What was once the humorous stance of Private Eye has become the humourless stance of the entire press. The gap between what people are and what they are treated as in journalism has never been wider. Only the very best journalists know how to suggest that a person, theory or event is not just what the journalist believes it to be. It is also itself. Holding that balance between your account and a proper respect for the truth of what something or somebody is outside your account involves a level of self-awareness hard to achieve in 600 words.

In the west a journalistic culture which takes in both the internet and television has now become both tiring and ubiquitous. It has also led to a curious deformation in society. As citizens, we consider our family, our friends and, most of all, our children as likeable and virtuous. But we are encouraged to consider everyone we don't know – and most especially those we know only through newspapers – as ridiculous or vicious. To this tendency, this desire to bundle people and thereby to dismiss them, art and death are the most powerful antidotes. Art frequently reminds us that things are never quite as simple as they seem. Nor are people. Journalism is life with the mystery taken out. Art is life with the mystery restored. Put people on the stage, in all their humanity, propel them into a course of events, and in even the most savage satire or preposterous farce, characters may acquire a sympathy, a scale, a helplessness, all of which draw forth feelings eerily reminiscent of those elicited by people you actually know.

Meanwhile, to the objection that plays and novels about contemporary events are too hastily conceived to be profound is added the confident counter-objection that such works are unlikely to endure. Shakespeare's plays may be crammed with incomprehensible Elizabethan references and jokes which amuse nobody, and these have hardly damaged his continuing popularity. But the example of literature's highest achiever does little to blunt the popularity of this line of attack. How on earth, it is asked, can either foreign cultures or generations unborn ever be interested in such local doings? On this question, I can only say I am willing to take my chances. Like most writers, I have at best a sceptical attitude to posterity. But wherever playwrights gather, you will find them telling stories of plays, performed in far-off places and years after their premieres, which have somehow acquired what seems like an accidental shimmer.

Of a recent revival of Stuff Happens in Canada – six years after the National Theatre first conceived a then-topical account of the lead-up to the Iraq war – the director wrote me a letter: "I find the play infinitely sadder than a few years ago . . . I think there is something potent about these people now officially out of office and firmly set in their historical place. At the same time, the references to both Afghanistan and Iraq are eliciting vocal responses from the audiences that I don't recall having happened in my previous production." In response to such a letter, any playwright will argue two things. First, no proper play is ever just "about" the events it describes. The whole intention of a play in describing one thing is to evoke another. Bush and Blair, after all, are not the only warmongers in history. But, secondly, in celebrating this play's bewildering success in Toronto six years on, the director was, in fact, celebrating the special nature of theatre itself. In Stalinist Russia the most powerful protest you could make was to stage Hamlet.

When standing as a candidate for the Governorship of Alaska in 2006, Andrew Halcro observed of Sarah Palin that she had what any politician would kill for. "And that is the ability to make substance irrelevant." Theatre has the very opposite ability – to find substance out and weigh it with devastating accuracy. But when deeply felt, theatre also has, like the music of Delius and Elgar, the potential to inflame the imagination of others. When The Power of Yes opened last October in Angus Jackson's production, it laid out the progress of the current banking crisis. It was therefore inevitable that the immediate response be journalistic. It either was or wasn't accurate. Its diagnosis was or wasn't correct. The play suggested that bankers now have the upper hand over anyone elected to power, firstly because complicated financial practices are beyond the understanding of most democratic politicians, but secondly because bankers have refined all their various blackmail notes into one single threat, left like a bomb in a litter-bin: "If you don't rig the market in our favour, we will drag you down with us." The legitimacy of such intellectual terrorism was hotly debated.

The Power of Yes dealt with issues that might well have been batted back and forth on a lively edition of Newsnight. Because the play portrayed real people, the dish arrived hotly spiced for journalistic carving. But then, interestingly, a second wave of reaction followed which addressed not so much the play's ideas as its techniques. Many things were expected of a play about high finance, but it was not foreseen that it should resemble Michael Bennett's production of A Chorus Line. Friends reported that they found the sight of 20 suited bankers lining up beneath the proscenium arch curiously moving. From then on, nothing was as they'd anticipated, least of all their own responses.

Many spectators noticed that this was a unusual example of a verbatim play which did not seem set on righting a wrong. Verbatim theatre was, they thought, a known genre, at its worst a touch hectoring and solemn, and overly dependent on direct address. On this occasion an attempt was being made to jazz it, to take it places it had never been. A few smart people were clever enough to guess that the inspiration had come from Glenn Gould's radio programmes, made over a 10-year period from 1967. In The Solitude Trilogy, Gould recorded the impressions of northern Canadians, many of them living lives of extreme isolation, and then sent their voices spinning like music, weaving them, fugue-like, into something the pianist called tapestry.

The most interesting reaction of all came from a painter friend, who was struck by how much the look of the play resembled an installation by Bill Viola. How strange, he said, that an ostensibly prosaic, ostensibly factual play should unleash some of the purest visual poetry he had ever seen in a theatre.

It is safe to say that I was more flattered by this response than I would have been 40 years previously. When I set out in the theatre, I was part of a fringe movement that often sought to crash the problem of aesthetics by doing plays as crudely as possible. If you made no attempt to do things overly well, then people would not be distracted from what you were saying. But as the years went by, it became clear to me that I had not understood aesthetics. They were not your enemy. They were your opportunity. Style was the only means by which you could suggest that what you were writing about was something more than what you appeared to be writing about. Without style there was no suggestiveness, and with no suggestiveness, no metaphor. The processes of art could begin nowhere else.

"Are you the notorious composer Arnold Schönberg?" someone once asked Arnold Schönberg. His reply has gone down in history. "Yes. Somebody had to be." In the last few years I admit I have felt a mild degree of fellow feeling. "Are you the person who makes plays out of what's going on in the papers?" is never a question asked in a friendly manner. Nor is the answer much liked. "Yes. Somebody has to." "How on earth do I review that?" said one well-known theatre critic, crying out to a colleague as he left the first night of The Power of Yes. It as if the doors of our theatre, of their own volition, blow shut all the time, and the task is always to prise them back open. Plenty of people get their poetry from science, from the physical universe, from the contemplation of mathematics, or of animals, or of solitude or of the stars. An audience arrives fearing the theatre will be one more medium like any other. If the subject of the play comes from political life, then they anticipate a form of animated journalism, journalism on legs, the usual mud-soup of opinion and sociology. But the performing arts can deliver high-flying bankers who are at once contemptible and deeply sympathetic. If we accept the simple distinction that factual work asks questions for us, whereas fictional work is more likely to ask questions of us, then why can some work not do both?

We are living through curious times and they demand curious art – in both senses of the word. "Aren't you telling us what we already know?" is the last question, always aimed between my eyes, potentially lethal in the questioner's view, but not even causing a skin-wound when fired. "No, I am not. You may think you know about something. But it's one thing to know, and another to experience." The paradox of great factual work is that it restores wonder. Thinly imagined work takes it away. "I never knew that, I never realised that, I never felt that" is what you hear from the departing audience when their evening has been well spent. Because we think we know, but we don't.

This is an edited version of a lecture given at the Royal Society of Literature.