It is Sunday afternoon on a windswept sports field in the depths of south-east London, and the quarter-final of the London under-13 youth cup is under way. There are 30 or 40 family and friends gathered on one sideline. Some are shouting a mix of encouragement and advice to their children, most of it conflicting, interspersed with agonised, expletive-enhanced appeals to the referee. The rest remain silent, grim-faced and anxious.

On a hill behind one of the goals stands a well-built middle-aged man with cropped grey hair. He is barking an instruction at the boys running around on the pitch, but the wind prevents his words from reaching their target. Louder and louder he shouts, his arms waving, as he marches back and forth. "There's always one," another parent says, nodding towards the man on the hill.

I move goalwards to try to hear what the man is saying. I can see that he's in an apoplectic rage of frustration. He looks a little like Mussolini on a civic balcony, addressing an indifferent crowd. "Push up!" The words cut through the air. "Push up! Push up! Push up!" If the boys he's shouting at can hear it, they don't seem to be taking much notice. They are two goals down and struggling to get out from their own penalty area.

The man on the hill is not the coach, because the coach is wearing a tracksuit and standing on the other sideline. So who is he? In the second half, he picks up the assistant referee's flag and runs the line. He is, in fact, the linesman. And as he runs up and down the line with his flag, he screams at the forward line: "Stay on! Stay on!"

Strictly speaking, the linesman, who is meant to be neutral, shouldn't be organising one of the team's forward lines to avoid being caught offside by him, the linesman. But this is not the Premier League. This is not the fantasy destination of all the kids and many of the parents gathered together on this blowy Sunday. This is the reality of youth football, and things are different here. The linesman, it turns out, is also a parent, the father of the centre-back he had been thunderously trying to direct. And parents in youth football sometimes get carried away.

When I was boy and used to play club and school football, the matches seldom attracted much noise from the sidelines; very few parents bothered to turn up. I would travel all over London by public transport to get to games, as did most boys I knew. The current generation of parents have become weekend sports chauffeurs. If a parent is going to give up a substantial chunk of their Sunday to ferry a child around, he or she may feel they've earned the right to provide vocal encouragement and possibly let off a bit of the steam that built up while trying to find the correct exit from the A2.

AC United were set up two years ago by a couple of friends in their late 20s, Bol Joseph and Reece Darcheville. Some weeks, taking into account matches and training, the two men volunteer five mornings, afternoons or evenings to the team. That's a lot of unpaid commitment, but both men say they don't regret a second of it.

AC stands for All Cultures – it's a multi-ethnic team. Joseph, who also doubles as a rapper called Joe Black, is a former gang member. He spent five years in prison as a youth and was stabbed multiple times by a rival gang. Having renounced that life, he and Darcheville have sought to combine football coaching with life coaching. "We try to merge the football and the mentoring," Joseph says. "Most of the kids are not going to become professional footballers, but there are skills from football you can put into everyday life, like punctuality, like respecting your parents or employers."

Back in south-east London, ACU score their third goal and some of the parents don't try to conceal their celebratory mood. The atmosphere farther along the line, where the bulk of the home support is bunched, is less jovial. "He hasn't got a fucking clue," says one mother about the referee, who looks not much older than the players.

The match is taking place at a well-respected club with impressive facilities and friendly staff. But some of the ACU parents sense an underlying tension. The team they're beating is entirely white, as are their parents, and the colour difference does not go unnoticed. "The farther east you go, the whiter it gets, and the more aggressive the atmosphere," says John Slyce, an American whose son is one of ACU's goalkeepers. "Barking's the worst. Nobody wants to go to Barking."

AC United secretary Pete Wickham with his son Charlie. Photograph: Sophia Evans for the Guardian

The game finishes 3-0 to ACU, without a major incident. That has not always been the case, as ACU's secretary, Pete Wickham, tells me after the game. His son Charlie is ACU's star defender. Last year he was playing in a game that ACU were leading 5-1, and the opposition and their coach were getting increasingly irate. "Two of their players came in from either side and broke Charlie's leg," recalls Pete, a small but resilient insurance agent given to clear-eyed analysis of sporting friction. "I'm carrying him off the pitch and their manager's shouting, 'Nothing wrong with him. You two are cheats. Father and son – cheats.' I was fuming. I saw red. I said some things I shouldn't have. He said, 'I've never been called a fucking twat before.' I said, 'I'm sure you have.'"

Charlie had been due to have a trial for Norwich the following week. One of the factors that has helped increase parental involvement in youth football, but has also raised tensions, is the possibility, however slim, of a boy making it to professional level. In an era when the top pros can earn more in a week than many parents earn in a decade, the fantasy of Premiership glory is one to which few parents are completely immune. Many are looking for their child to be selected for trials with a professional club, then to join the youth academy. Several of ACU's best players have been lost to academies. It's like a mini-version of what happens with big clubs and small clubs in the professional leagues. As soon as a player stands out, he's at risk of being lured away. ACU's best player this season has been Charlie, a lanky, tough-tackling defender with good technique. In March, he got another chance: he was asked to try out for Southampton's academy – the one that produced Gareth Bale, Theo Walcott and Adam Lallana.

Pete was pleased and proud, but still focused on ACU. If Charlie passed the trial, the chances were still overwhelmingly against him. Most academy kids don't make the grade, and of those who do, only a small minority make it into the first team.

Pete knows all about overenthusiastic parents and managers, having been and done both himself. With his first son, Ryan, who is now 18, he was an active participant from the sidelines. "I used to shout instructions to him, not understanding that it would confuse the kid."

When Ryan was 10, he played with the under-14s, because back then no one was checking birth certificates; nowadays, players have to produce identification. Pete would often find himself in disputes, protesting at what he felt was the vicious tackling that was allowed to take place. "Then, when Ryan was 14, I took my coaching badge. It teaches you loads, the welfare side of it. It makes you think massively about these things. I thought, fuck me, I must have really upset him and the manager. And I must have confused him, because you don't know what the manager's telling him."

AC United make parents sign a code of conduct that stipulates no coaching from the sidelines. But Joseph is relaxed about its enforcement. "As long as parents are in the right place and not swearing or saying anything offensive, then shout your lungs out. I tell the kids, 'This is how it'll be if you're playing in a stadium – people screaming from all areas of the pitch.' It's all experience."

Although he says he's seen parents of other teams punch each other and even attack a coach, Pete says behaviour has improved markedly in recent years. This may be partly down to the Respect programme. One feature that has been introduced in recent years is a sideline cordon that discourages incursion on to the pitch of those who feel the need to get nearer to the action.

Dermot Collins is responsible for the Respect campaign. "There are 35,000 or so youth fixtures that take place each weekend," he says, "and the vast majority of it happens without incident. Everyone enjoys it. But there is a proportion of people who can make that experience unpleasant."

His own son plays in a league, so he's familiar with the problems. "There's one team, they've got a manager who likes to walk up and down the line and take a position five metres on to the pitch, and a parent who runs behind his child and advises him on every kick and run. But they are exceptional. What we've tried to do with Respect is educate people about the best way to support their kids, and about what their kids want out of football."

What's notable, watching youth football, is that what kids want is to play football – and that's really it. In that sense it's different from the adult game, where often players also want to argue, shout, threaten, cheat, complain and sometimes fight.

AC United dad Reda Imansouren with his son Yusuf. Photograph: Sophia Evans for the Guardian

This distinction is confirmed by John Hyde, a legal journalist who referees in youth and adult leagues. On one of the few occasions he had to send a boy off, he was worried about doing so because of the way the boy's father was behaving. "It was an under-13s game, and as I was taking my card out, his dad was screaming at him, stuff so bad, I was inclined to keep the lad on the pitch."

In spite of stories such as this, Collins would not like to see Lineker's suggestion imposed. "We've got some leagues and county FAs who want silent football. They don't want parents to say anything. I've never supported that. Youth football is only possible through the support of parents. If they are bringing them along, their involvement is legitimate. It's positive. It's just that bit when it steps over the line and becomes them and us, and you can't recognise good play for what it is."

This winter was a washout for grassroots football across the country, with thousands of games called off due to waterlogged pitches. But in the middle of the rainy season, AC United managed to play the semi-final of the London under-13 youth cup on an all-weather pitch in Elephant & Castle. It was a full-size, adult 11-aside field, and some of the boys seemed dwarfed by the dimensions, as if the opposition goal was in another county.

Pitch size has been identified as one of the problems that afflicts British youth football and results in few players making it to the elite grade. In Spain, children play on smaller pitches where they are taught how to make short, accurate passes and develop an appreciation of what's called "possession football" – namely keeping hold of the ball and not giving it away to your opponents.

The larger pitches British kids often have to play on encourage a boot-it-up-front-and-run style that has limited technical development and hindered the England team in international competition. Another problem has been the style of coaching. "Unless you're about development, you shouldn't be in youth football," Joseph says. "A lot of coaches have an adult mentality. All they care about is winning, winning, winning."

In the semi-final, ACU came up against Docklands Junior, a team of kids who looked as if they could have earned a decent living unloading ships. Despite some minor grumbling about the size disparity, the parental support was good-spirited, with both sets applauding displays of skill, regardless of the team. Here we were in inner-city London, with scores of boys of all ages, sizes, cultures and races thronging the perimeter, and you couldn't have found a more benign mood at a village cricket match. Perhaps most sporting of all was Anam, a short Asian man with a long beard. In each game I saw, his son appeared to be forever on the subs' bench, but it didn't for a second diminish his support for the team.

ACU lost 1-4, the dream of the youth cup over for another season. The next day, Pete got an email from the London FA praising both sides for "a great advert for youth football". He was moved in a way that would have surprised his younger self.

A few weeks later, Pete was on the sidelines in Enfield, at ACU's home ground, as it were, a municipal park with a broken sewage pipe under one goalmouth, which turns making a save into a health hazard for the unfortunate keeper. Kick-off was 10am, but games often begin at 1pm – Sunday lunchtime – round trips of 40 miles are not unusual, and arrival time is an hour before kick-off. Does Pete ever resent giving up a large slice of his day off? He squints at me in incomprehension. "No, I love it. I wouldn't miss it for anything."

That day, ACU were playing Broadwater Farm who, Pete says, are renowned for their volubility. The Broadwater parents were affable. There was Sandra, who joked they always believed the refs were against them, and Paul and Eileen, here to watch their son, a strapping striker with a cornrow braid. Paul is a lorry driver, and had driven down from Scunthorpe the previous night, arriving home at 6am. He looked exhausted, but he'd never miss a match if he wasn't working. When he was working, Eileen stood in for him – and got to love watching so much, she comes now all the time.

The first half was uneventful, apart from when one of ACU's players went down injured. "He's carrying a groin strain," his dad explained.

"He ain't got a groin," growled John, a stout cabbie with a tendency to wear short trousers in long-trousers weather. "He's a kid."

At half-time, Paul told me about living on Broadwater Farm. "We're always under surveillance. Look at your Wi-Fi, it will say MI5."

AC United dad Piers Feltham with his son Sam. Photograph: Sophia Evans for the Guardian

In March, the scene shifted to Hackney Marshes, the vast spread of football pitches in east London. ACU were playing their bitter rivals, the team that had broken Charlie's leg, coached by the man Pete had insulted. Before the match, a boy from the opposition who used to play for ACU suffered an epileptic fit. The joint effort to provide aid seemed to ease the tension between the two teams and their parents, grouped behind the cordon that was placed about two yards from the sideline.

All except one parent. There's always one. He was wearing a jacket advertising salsa instruction and stood apart from everyone else, a foot from the sideline, shouting out a stream of instructions and appeals. "Get in there!" "Chase it down!" "Get stuck in!" "Foul throw!" "Offside!" "Whyja pass it to him?" and, over and over again, "Hurry up!" This last command must have been particularly confusing because several other parents were simultaneously telling the boys to take their time.

Fortunately, none of the boys appeared to take any notice. The game was full of neat football, quick movement and sharp passing. Events were nicely poised at 3-3, before the dreaded rivals finally ran out 6-3 winners. Sam, ACU's goalie, was distraught. His father, Piers Feltham, a barrister, told him not to sulk. "Fuck off, Dad," replied the boy. At which point Joseph and Darcheville took the pair aside, and gave Sam a lecture on respecting his father. "We told him," Joseph said, "we're not saying you're wrong to challenge your dad, but it's the way you're doing it. Remember, your dad's only got your best interest at heart."

And underneath all the shouting and bluster, heartache and joy, that's almost invariably true. Every weekend, on windblown, muddy pitches across the nation, there are thousands of middle-aged men and women giving voice, straight from the heart, to their sons' best interest.

Time was when pundits were all meaty thighs in shiny suits. Then along came David Beckham, social media and everyone upped their game. As they gear up for Rio, Adrian Chiles, Manish Bhasin, Jake Humphrey and David Ginola talk to Jess Cartner-Morley about football and the new fashion rules