Get out of our living rooms. This country is in danger of becoming a politically controlled nation closer to communist China. That's all very well if you have three hours to wash the dishes, but some of us need to get things done. Gee, these toddlers are up to no good. What are they up to? Wait for it – they're watching television!

The outrage that has greeted reports that the Australian government is to issue cautious guidelines advising parents and carers to prevent children under two from watching television seems remarkably acerbic. Across the world, however, the same debates flare up every time it is tentatively suggested that the electronic screens we began by placing in one room at home and now carry everywhere in our pockets may not be good for the development of children's brains.

Television is no longer merely the drug of the nation, it is the pacifier, babysitter, wallpaper and teacher for our children. Increasingly it intrudes on the very first months of their lives. In Australia, young children spend more time watching television than any other activity. The average four-month-old gazes at the box for 44 minutes every day. In the United States, under twos watch 1.2 hours a day on average. In Britain, older children have been calculated to spend five hours and 18 minutes watching TV, playing computer games or online each day, just over an hour less than the US average.

Behind the fury about strictures suggesting television is bad for our children is guilt. Parents are uneasy about the effects television has on their children and are quick to get defensive about switching it on. "Whether it is the slack-jawed look their children have when they put them in front of the television or the tantrum when they turn it off, most parents have this unease about it but it's a battle they choose not to fight. They have enough battles getting them to eat the right food," says Dr Michael Rich, director of the influential Center on Media and Child Health at Boston Children's Hospital.

We may now be highly tuned to what we feed our children's bodies but we are less careful about what we feed their minds. Academics researching the impact of television on the very young compare debates over its adverse effects with those over smoking a generation a half ago, or seat belts and cycle helmets more recently.

A draft of the Australian government's guidelines says that screen time for young children "may reduce the amount of time they have for active play, social contact with others and chances for language development", and may also "affect the development of a full range of eye movement [and] . . . reduce the length of time they can stay focused". Jo Salmon, associate professor of epidemiology at Deakin University, was one of the researchers who informed the Australian government's draft guidelines. "Children aged six to 30 months who are watching television have less developed vocabulary, display more aggressive behaviour and have poor attention spans," she says. "Parents and childcare centres are not justified in encouraging children, under the age of two, to watch television." While there is no evidence that so-called educational programming is harmful, she would discourage under twos from watching it. "I really would not put my young one under two in front of a television. Generally, the evidence that's out there says it could be detrimental," she says.

We may sense TV is bad for young children but what evidence is there really? There is a booming market in educational computer games and DVDs, such as the Baby Einstein range, and if our modern multitasking lives are saturated in electronic screens, isn't sitting children in front of them at least good training for the modern world?

Rich worked in the film industry before having a "midlife crisis" and retraining as a paediatrician. He is not evangelical about governments enforcing how television is used in homes but barred his own two young children from television and computer games before they were 30 months old. While there is good television that children can consciously learn from at a later age, he says scientific studies show young children are not able to consciously learn from television.

As Rich explains, humans have the most sophisticated brain on the planet because it is relatively unformed when we are born. Our brains triple in volume in the first 24 months. We build our brains ourselves, by responding to the environment around us. The biggest part of this is a process called pruning, says Rich, whereby we learn what is significant – our mother's voice, for instance – and what is not. "TV killing off neurons and the synaptic connections that are made in order to discriminate signals from 'noise'," he says.

Experts in child development have found that three things optimise brain development: face-to-face interaction with parents or carers; learning to interact with or manipulate the physical world; and creative problem-solving play. Electronic screens do not provide any of this. At the most basic level, then, time spent watching TV has a displacement effect and stops children spending time on other, more valuable brain-building activities.

Scientists concede that they do not yet know precisely how TV affects the cognitive development, not just in terms of understanding the inner workings of the brain but because the way we use television and other electronic screens is changing so rapidly that we do not know how it will affect people by the time their brains stop developing in their mid-20s. But the weight of evidence about the deleterious impact of TV on child's ability to learn is alarming – to say nothing of its impact on children's sexual development

The Australian government's advice is supported by the recommendation of the American Academy of Pediatrics that under-twos are not exposed to any television time. Dr Dimitri Christakis at Seattle Children's Research Institute found that for every extra hour watching DVDs, 8- to 16-month-olds learned six to eight fewer words than children who spent no time in front of the screen. Marie Evans Schmidt at the Centre for Media and Child Health found that even just having television on in the background while under threes play with their toys disrupted their attention span even when they appeared to pay little attention to it.

While there is a paucity of evidence that television is beneficial to early cognitive development, there are studies that show it is not as influential as the educational status and income of parents. Schmidt found that an apparently negative relationship between TV viewing and cognitive development disappeared when she factored in the mother's educational status and household income – parents' education and finances mattered more. "TV viewing is an outgrowth of other characteristics of the home environment that lead to lower test scores," said Schmidt. Other research suggests these include less mother-and-baby interaction and less reading to children.

Unlike the Australians, the British government does not offer any guidance on how much television toddlers should be allowed to watch. It has introduced an "Early Years Foundation Stage" for 0-5s which implies that television should be part of children's learning. Carers, the guidance states, should help children become familiar with "everyday technology" and use it to support their learning. Only the French government has been brave enough to ban stations from showing programmes targeted at under-threes. Last year it also insisted that overseas cable channels must incorporate a tobacco-style warning: "Watching television can slow the development of children under three, even when it involves channels aimed specifically at them."

Aric Sigman, a UK psychologist and author of The Spoilt Generation, a broadside against permissive parenting, says while governments are happy to offer advice on suncream and portions of fruit and vegetables, they are less willing to provide guidelines about TV. "Of course they don't want to because it is a vote-loser," he says. "It is society's favourite pastime and it makes parents feel guilty. The convenience of us parents is seen as paramount as opposed to the wellbeing of our children. When it comes to our childrens' wellbeing, our guilt as parents has to come second."

Part of the problem, argues Sigman, is we have a nostalgic view of our own experience of television when we were young. "We say, 'I watched Blue Peter and I'm OK'," says Sigman. "But the editing speeds and the colours and the number of hours spent watching TV and the age at which TV watching starts are a whole different thing now. We can't compare now with before."

Rich agrees. Television is so different now that the "it didn't harm me" argument is irrelevant, he says. Instead, frustratingly, he finds the debate around young children watching TV is played out as part of the culture wars in which the educated, ruling classes of academia and, when they dare, politicians, are perceived as self-righteously restricting the freedom of expression of ordinary people. Rich instead hopes the debate could be examined more neutrally – and scientifically – as an issue of "health and development". He accepts that TV is not like smoking: it is not simply bad for your health. He would like a return to a "respect" for TV and other electronic screens so they are treated like a trip to the theatre or a novel, as something to be consciously watched in moderation. This may sound like another culture wars value judgment but, as he argues, it is not about good or bad TV but about the good and bad ways in which we consume it.

Additional reporting by Toni O'Loughlin.

• This article was amended on Wednesday 14 October 2009. In the article above we referred to sexual activity, we should have said sexual development. This has been changed.