If you want to make dramatic fiction for the screen you must first strangle your creative impulses. The alternative is even more painful. It is to put your creativity at the service of the formula and take instructions from the executive apparatchiki. They need to feed off your creativity because they have none, and to control it because they are told to.

This totalitarian micro management is not confined to just one area of television, nor even to television. It's just the one I know best. It grew up under Thatcher as the bosses recovered their self confidence and new management was encouraged to crack the whip. It has achieved its apotheosis in the grand years of New Labour's incursion into every crevice of our public services. We cannot understand what is happening in screen drama unless we place it in the context of the wider society.

Working in art film or commercial cinema is like dancing through a mine field, and every broadcaster is now racing down market in a desperate attempt to survive. But what is happening at the BBC is the real scandal: it is bigger than all the rest combined, it is free from direct commercial pressure and its public service obligations carry cultural responsibilities. There are no excuses.

You need flair to ignore real problems, then to identify as problems those processes which are in fact working well enough. But it takes genius to apply solutions to these fictional problems in a way which actually does create real problems. Only management consultants at many thousands of pounds a day have such chutzpah, which they call best practice, because they know they will be called in again to solve the very problems they have just created.

At least in the Seventies, if Huw Weldon's possibly apocryphal story is correct, the consultants were more honest. He recalled being asked by the top man just how many decision makers there were in the BBC. Decision makers? Yes, people who actually have the power to decide what you make, the core products, for example. So Huw wondered how many producers there were and said maybe several hundreds, all told. The consultant went white and said, "Just as I thought. I'm afraid in that case, there is nothing I can do to help you, Mr Weldon."

The decision the BBC made in the Eighties as it attempted to impress its political masters was to increase the height of the management pyramid, instead of flattening it out. This allowed it to claim that it was becoming more professional, tough minded and responsible. After Alasdair Milne resigned and John Birt achieved power, this centralisation was accelerated. By the time New Labour got into its stride, Birt had consultants all over the BBC like a rash. As an institution it fitted in perfectly with the ideology of the day. It is no accident that Birt's two jobs since have been at number 10 and at McKinsey's.

In fact Birt has been unfairly turned into a wicked uncle. The truth is more nuanced. He was resolute and brave in his attempts to bring some proper financial discipline into a ramshackle system. The gentlemen did not wish to be turned into players. He was percipient about New Media and the imminent upheavals the Internet would bring and made sure that the BBC had a head start. It is now reaping the benefits of his detached understanding of the technology. His problem, in addition to a leadership bypass, an inability to charm and persuade, was to have faith in out of date management theories about structure. So under him layer upon layer of supervision and new job descriptions were piled onto the programme makers. The old joke that the BBC would be an efficient, well oiled machine if it were not for the pesky programme makers, seemed to be taken seriously at the top. Better to get rid of them all together, and if that were impractical, at least, supervise the life out of them. It was complicated, because by the Eighties there was a lot of complacent dead wood in Drama. It was a slack outfit needing a clear out and some inspiring leadership. It got neither, just more irrelevant bureaucratic supervision from senior management.

The BBC did become a recognisably modern outfit, at least if you looked at the management charts. The pyramid was tall, reporting was clear right up to the Director General ,and power was where it should safely be, with the grown ups in senior management.

There was only one problem. This sort of control is the enemy of creativity. The more you have, the more difficult it is for artists to do original work. This is not to argue against all controls. If your work needs lots of money both to make and to distribute, you should expect to work exactly to an agreed budget and to deliver roughly what you promised. But the main effect of the kind of supervision which penetrates into the details of productions, leading to artistic decisions being made further up the hierarchy, is to stifle the creativity which the organisation is supposed to be encouraging.

Senior management still does not understand that detailed supervision by more and more layers, reporting to more and more senior executives, does not result in higher standards. A writer will get notes from a story editor and a producer anyway. The real motive must be neurotic control borne out of fear. Let's make sure everything is safe with no embarrassing surprises. Better to squeeze the life out of it than run the slightest risk of getting into trouble.

Let's see how that looks, not from the eagle's perch but from the worm's eye view, the writer. We assume he or she has an idea for a renewable one hour series. That is what they want to buy these days.

A pitch is worked up and taken to the BBC executive. There will be some discussion. Can the characters be skewed young? Well, considering they are senior hospital consultants, it might be difficult to go very young, but we will try. I don't mind where it's set, really. I don't want to be prescriptive, but somewhere other than Birmingham, perhaps? Manchester would be good. And so on. Eventually a pilot script may be commissioned. The writer has a few weeks of bliss, the only time alone with the characters. Then the producer gives notes on the first draft and another is written. It goes to the BBC. Long delay. Maybe months. They are very busy. Then notes from the commissioning editor. After a tactical discussion with the producer about how to avoid alienating the editor yet stop these silly notes killing the project, a new draft is written. It is submitted. Long delay. Then more notes, possibly by same editor, or with luck a higher executive. Finally another meeting, possibly with the writer not present – we can speak more frankly, can't we? More notes, but contradicting the previous ones. First editor now fulsomely backs his boss. The higher you go the more valuable your ideas. Naturally. Yet another draft, or two. More long delays. The senior editor is in America or on away days or waiting to speak to the Controller.

You are now maybe two years in. If you are lucky, the show is green lit. But don't think you can just go off and make it. The problem is not the gap between the scale of the production the BBC requires and the budget they allocate. You are used to putting a quart into a pint pot. It is the refusal to accept your suggestion of a writer for episode four and the directors who are turned down. The demand is for someone coming off a hit, someone in fashion, the flavour of the month. They need to be reassured, so if you are hot you work, if you are not, you are rejected. The pool to pick from naturally reduces. Various stars are suggested, all miscast and not even stars. There is no point arguing. Of course the BBC did not invent this system. They copied it from the Hollywood studios. The only difference is that in Hollywood the poor writer gets jerked around but ends up with a swimming pool. Eventually a compromise is negotiated. The producer naturally sees the brilliance of their ideas, so the production starts.

There are notes on rushes. Notes on each cut. An executive presence at the each stage of post production. Eventually the show is delivered.

This experience is typical. Sometimes it is smoother, sometimes it is worse. It often takes about three years.

Remember, these executives are mainly benign people with good intentions, keen to work hard and to achieve. Many are intelligent, some even talented. If they were working properly in the industry, learning their trade, they would become good producers. But what they do is largely unnecessary. They put spanners in the works. In their place should be an experienced grown up whose help would be welcomed, who knows when to do nothing and whose taste is informed. People like this are difficult to find. But we would need very few of them. This sits uncomfortably with New Labour busyness.

The trajectory of energy is in the wrong direction. Instead of erupting upwards in ways which surprise, delight and occasionally shock, it travels censoriously and prescriptively down the pyramid. The writer is left to second guess what might please the power at the top in a grotesque game of pass the parcel of notes as they travel from hand to hand, changing their meaning on the way.

Note giving is an art and a craft. It must happen in an atmosphere of earned trust and approval if it is to avoid defensive resistance. It must be specific and concrete. "Make it funnier" will not do. Nor will half understood jargon from a weekend screenwriters' course. Talk of "narrative arcs" and "epiphanies" and the writer will politely nod and go home to look longingly at the gas oven.

The problem is that all executives think they know how to read a screenplay. They were taught to read at school, after all, and have even written stuff. E-mails now, mostly. They've seen plenty of drama and have strong opinions. Power seems to confirm their ability. I have watched Channel Controllers come and go over fifty years. Few started in Drama, but something magical happens the moment they are appointed. They instantly become authoritative experts not only on scripts but especially on casting. Producers who spend their lives learning about these matters are, of course, grateful for their advice.

If the producer does not heed it, the best that can happen is the show is cancelled. The worst is the producer, thought to be too difficult, finds new orders even more elusive. This is a commercial relationship now in a buyers' market. Behave as the buyer wishes or get another job. Much better to make the director you think is wrong for the job work with an actor neither of you wanted on a screenplay the writer is now ashamed of.

The reality is that over the last 25 years producers have lost their role. All the important decisions have been stolen by executives, not because they are now making the shows, but because they have the power. They have reached down, taken the "what" and the "who" questions and answered them to their own satisfaction, leaving the producer with the responsibility for the "how" questions. In army terms, there is a vast officer class, well decorated – you only have to see them strutting proudly at awards ceremonies – with the producer, now an NCO, out on location with the crew, trying to win the war, but hindered by friendly fire.

Traditionally producers could only justify themselves by showing that they had taste, sympathetic detachment, leadership skills and above all the confidence to devolve their own power further down the line, giving room to the writers, directors, actors, DOP's, editors and so on. The very best had a vision for a show which they were able to share with the vision of others. Helping to meld everyone's creativity into an artistic whole is different from handing down dictats from on high, even if they are dressed up as helpful suggestions. Of the three promises you should be wary of, two are of a sexual nature and the third is, "hello, I'm from head office and I'm here to help you." Run for your life.

There is a case to be made for the existence of producers, then, by the other creative elements. But there is also a case for their abolition. They were introduced to the BBC by Sydney Newman in the early Sixties. Before that directors carried it all, with the minimum of detailed supervision from Heads of Departments. Like everyone, producers need to be seen to be useful in the eyes of their peers. But the BBC's gradual move to abolish them is another matter. That will be a consequence of management's victory in the campaign to take all decisions as far up the institution and as far away from the creative community as possible.

Writers are becoming executives' scribes. Directors are mere bus drivers: only they have the skills to handle such a big vehicle, but they are told which route, where to stop and who is to travel on the bus. Inspectors are never far away to check up on any use of the imagination.

To commission a project well you have to predict what a Controller will need a year or more in advance and what an audience will respond to, not always the same thing; and do this on the evidence of a pitch and a few personalities. In addition now are required to micro manage all aspects of a production, expected to give detailed notes on scripts, direction, performances, editing and other technical matters. This would be an impossibly tall order for even the most sophisticated and experienced practitioners. The present commissioners pick up what they can from the professionals they supervise, but they are on a hiding to nothing. Instead they concentrate on the dark arts of executive survival. One day they will wake up and admit that they are just junior cops in a conspiracy to corral creativity and neutralise spontaneity. They will be sad.

So the next time you watch a fine piece of TV drama, grateful for the brilliance of the writer, the director, the actors and the crew, remember the aggravation they had to endure and the guile they had to deploy and the energy they had to waste. You will no longer be puzzled at how rare this experience is or be surprised at the formulaic, repetitive, machine made, emotionally dishonest junk food you now get for your license fee. The people making most of this predictable junk called drama would love to be creating something better and more nourishing. But they are not allowed to.

They are herded together on an assembly line and given specific functions to perform. They have little choice because high volume shows provide most of the work now. Everyone likes some junk food occasionally. It is immediately satisfying, cheap and addictive. So what that it doesn't nourish you, might even be bad for you with all that fat, sugar and salt? Lighten up, a bit now and again will not kill you. But over the last decade or so the BBC, in perhaps its worst public service dereliction, has skewed its money and airtime decisively towards high volume junk which runs across the year. In addition to "Eastenders" and "Casualty" it now has "Holby City" and numerous other lengthy series. There are very few single pieces or mini series, the kind of original writers' work where a voice can communicate directly with an audience. The BBC has the duty and the resources to make a full range of programmes, but in this shift in balance they expose their opportunistic cynicism. Ratings are their default argument, as though this were the only criterion. By opting to get an audience the easy way they short change both the audience and the programme makers. Better to pack them in with junk. Cost per thousand viewers cannot lie. But a high volume show is a branch of manufacturing. The artists are put at the service of the product. Watch them and weep. They are usually set in a place called Holby and run throughout the year. You will occasionally see bursting out of the bland predictability a scene written, acted and even directed with originality and verve and emotional honesty. It must have escaped being ironed out by the machine.

Before the BBC corrupted itself and became just another marketing exercise, it understood that its business was not the commodification of ersatz culture. It understood that we make sense of the world, and of each other, through the telling of stories. The more these stories are authenticated by an author, the more emotionally truthful, the more complex they are, the richer the society which encourages them. The more they are taken from the shelf, unfelt, manipulative and false, the poorer the society which lazily allows them. The BBC, for all its faults, used to take its leading role in our culture seriously. It knew that the very health of a society depended on the quality of its national debate and it knew that the storyteller was a central force in that debate.

Now, every night, we die a little as we suffer what cynically they call entertainment. They fail to realise that good work is more than that. It lives and feeds our minds long after the entertainment fades.

The senior management at the BBC simply do not understand the creative act. This would be a deficiency in any organisation, but in one for which it is the main raison d'etre, it is crippling.

In particular, they do not realise that an artist is childlike, not childish. Good parents will erect boundaries, around personal safety, for instance, but will leave room for the children's imagination to flourish. The children, with few material resources, will invent elaborate worlds, not knowing from one moment to the next where their actions will lead. No matter if some prove to be cul de sacs. They will start over and go in another direction. This creative absorption needs room and time. The parents should not interfere, preferably not even eavesdrop. The results are magical and satisfying, not least in the healthy growth of the child.

Anal retentive, anxious parents help and stifle. They know best. They cannot relax and trust. They are prescriptive. Play becomes a duty, imagination becomes second hand, the goal of the children degenerates into guessing what will please the parents and earn praise. It is no fun, but the child has to pretend it is fun, because the parent insists that is what it is. In fact it is a lifeless desert. Spontaneity is dead. But the world is safe from the children's journey into the unknown. Dictators first kill the imagination. For the people's good.

The BBC management in a previous age existed in another social and political context. There was much to criticise ( and I did ) but when it worked it was largely management by benign neglect. They had the confidence for that. It may have left many problems unattended, but it allowed creativity to breathe, calmly accepted that failure was one of the prices of success and had faith enough in its people not to micromanage them. They were paternalistic, but they had the good sense not to continually open the oven door to check if the soufflé was behaving itself.

A personal memory from the Seventies will illustrate the point. Standing at the BBC bar my Head of Department, Gerald Savory, was also ordering a drink. Haven't seen you for a while, he said. What was I up to? About to go out on location to shoot a couple of films for you, Gerald, was my reply. Who are the writers? There are no writers. What are they about? I don't know yet. The directors will work that up with a few actors over the next few weeks. Oh……who are the actors? No one you've ever heard of, Gerald. Who have you got directing? Clutching at straws by now. Just a couple of lads you wouldn't know. What have they done? Nothing really, a low budget little film no one's seen. But I think they have something. Well…..Jolly good luck. The next time we spoke was when he saw the finished films.

The point is not that all shows were made like this, but that it was possible at all. The directors, by the way, were Les Blair, who went on to have a distinguished career directing "Law and Order", G F Newman's mini series, among many others, and Mike Leigh, who didn't exactly fade away either. Without the BBC neither would have been given a chance to explore their style, creating work in their own way. Would they get a chance in today's BBC? A centralised structure where there are many who can say no and only one who can say yes results in a narrowing of taste on the screen. We need a range of sensibilities at work. This can only be achieved if power to commission and transmit is passed down to more people. The writers will have more chance to find a sympathetic ear and the audience will have a bigger range of tastes to sample.

Unfortunately the BBC hired McKinsey's and ended up as MacDonald's. It used to hold a subtle creative tension between the writer and the audience. The writer was pressed into reaching for a large audience, straining to be both serious and popular; the audience was invited to try the unfamiliar, to be challenged and disturbed, not just spoon fed with the pre-digested and familiar. It was a nuanced, complex relationship. Now the BBC aspires to be Proctor and Gamble. It doesn't take ads for soap flakes. Instead, it makes its programmes as though they were soap flakes. When a senior executive says, without shame or the merest blush, that they do not believe in authors, they believe in strategy, what she means is that they follow the advice of the marketing executives who have pored over focus group results. The programme makers are then instructed to construct a series which will attract young men, because that is the strategy; or that a show will not be renewed because the large and appreciative audience is too old, and that is against current strategy. The game now is not about the writers having the freedom to make their sense of the world; it is about creating products and brands which the research has indicated will sell.

But even on its own depressing criteria, the BBC is failing to embrace the new and the young and the technology – words which give them all an erection at Television Centre. Its early lead in the Internet has become bogged down in supporting traditional television. The internet is not just a new means of distribution and exhibition any more than broadcasting was. That started by putting cameras on West End theatrical plays, but soon creative people were discovering ways of telling stories which were peculiar to television. It broke away from the cinema and the theatre and became itself. The same will happen on the internet. So why hasn't the BBC voted some real money to Drama, instructing them to commission directly for the Internet, in a two footed leap into the future? No one knows what that future will be like, until they try, fail, fail better and then come up with something wonderful and in retrospect obvious. We don't know what sort of stories will work, at what length, with what, if any, audience participation. Individual devices can connect in real time, with each audience member experiencing the same story. Stories can be broken up into different POV's. And so on. At the moment there is no adequate revenue model, so the brave souls who have experimented are not being encouraged. What is stopping the BBC? Why not a batch of low budget dramas made for the internet, focusing on ideas, innovation, the writer and the actor, not the technology, nor the production values? Then broadcast them if you like. But broadcasting is over. It will survive on the internet in a different form. Perhaps the ideology of senior management does not encompass the creative entrepreneur. Like their New Labour masters, they talk the talk of modernity, but deep down they are traditionalists.

The BBC is porous. It makes the culture but also the wider culture makes it. Over the past decades there has been a renegotiation between producer and consumer interests throughout society. It continues, made more difficult by the fact that each individual is at work a producer and at leisure a consumer, shifting attitudes accordingly. Guiding and policing this renegotiation have been a priority for New Labour, and consultants from the private sector have seen rich pickings beckoning them into the public sector. We should not deny the reality of the renegotiation, nor the attendant problems. We should regret the policies adopted. They are worse than the disease they purport to cure.

I could have written this piece with few changes if I had been a nurse, a teacher, a social worker, a cop or almost anyone in the front line of any public service.

I am not suggesting that the lunatics should take over the asylum from the lunatics who now run it. I am suggesting that now, in what one hopes are the death throes of the New Labour diversion, we have an opportunity to slough off totalitarian management, solve our problems in a creative way and let joy replace fear in our national broadcaster.