Our car moves slowly across London, trundling through heavy traffic from Chiswick to Shoreditch, as Clare Balding looks out at the ordinary streets of a city that has just staged the most extraordinary summer of sport. A tumultuous six weeks have slipped away but, on a cool September evening, Balding captures the exhilaration and emotion of London's Olympic and Paralympic Games as she talks in her familiarly relaxed way.

She may have become the most cherished and respected sports broadcaster in Britain but, as befits a woman who believes she was a disappointment as soon as she was born, Balding also uncovers the layers of poignant meaning buried in a rollicking summer. "You know," she says, "Chris Hoy is very impressive. He really gets it. When the floats were travelling through London [on last week's parade] he knew it wasn't a case of the crowd saying thank you to the Olympians and Paralympians. It was much more that the athletes were saying thank you to the crowd. That's why it was so moving.

"I really lost it on Trafalgar Square. I just looked at the crowd and felt really choked because this summer made us all better people. But I don't think I will understand the significance of 2012 for another 10 years. I don't think any of us will. You need a sense of real distance to look back and understand."

Such distance has helped Balding to write a fine new book about her often dysfunctional family, where a recurring half-joke has been that "women ain't people", and a battle to accept her own appearance, character and sexuality. Balding's conscious decision to stop worrying about how people judge her, and concentrate instead on being honest and passionate and open, prompted a usually bile-ridden columnist like Jan Moir of the Daily Mail to ask a frothing question: "Why can't everyone be Clare Balding?"

Yet remembering how, just before the country was swept away by happiness and wonder, Frankie Boyle tried to dredge up another mocking crack about Rebecca Adlington's face, Balding says: "I still got cross on air. I didn't realise it was 10 o'clock on a Sunday morning. The chimp got me and I said: 'Becky Adlington's had to put up with people saying bloody stupid things about her on Twitter.' In my ear, my editor said: 'Steady, Clare …' Your rage on behalf of other people is stronger than for yourself. But Boyle's power is diminished. That alpha male bullying which belittles women, and particularly successful women, is dying."

A contrasting warmth underpins Balding's success but, as she points out, her work is "defined by knowledge. This summer a lot of presenters reaffirmed the value of knowledge over style. We don't just need to know enough to ask the first question. We need to know enough that we can pick up on a reference in the answer and respond with a new question. I was also relaxed because I'd done four Olympics before and three Paralympics. I knew I had stuff deep in my head and if it comes up I'll remember it. So from the start I thought: 'I'm going to enjoy this.'"

Balding excelled in both the serious and the lighter moments – as when she allowed Chad le Clos's dad, the burly Bert, to emote about his beautiful boy after he produced one of the surprises of 2012 and beat Michael Phelps in the 200m butterfly. "South Africa's media woman came to me later and said: 'Thank you for Bert. That was unbelievable for South African men.' It was a big deal – having this big Afrikaner crying: 'Look at my beautiful boy.' It made Chad. He wasn't just the guy who beat Phelps. He became a star in a great TV moment. It could only happen on live TV where I had the freedom. It also helped that I've done racing, where you're constantly doing things on the hoof in the paddock. It doesn't matter what you look like then.

"But we do judge each other on appearance. I did a couple of Ramblings on radio with blind people and I once asked a stupid question: 'Do you ever wonder what your guide looks like?' The blind man said: 'It has no relevance to me. It's only relevant to you because you live in a visual world.' I thought: 'God, you're right. Of all our senses our vision is the least trustworthy. You make a judgment of someone when they're halfway down the street.' Society does it all the time – particularly to women. That's why I like sport – because sport is more honest. This summer sportswomen of all shapes and sizes promoted the idea we can look different to each other."

The Paralympics, which Balding fronted effortlessly on Channel 4 after her outstanding Olympics on the BBC, carried that same freshness. "Oscar Pistorius getting beaten by Alan Oliveira was extraordinary," Balding says in the midst of recounting her favourite moments of the summer, recalling how the celebrated Paralympian's defeat to the young Brazilian was followed by Pistorius claiming the length of Oliveira's blades gave him an unfair advantage.

"Oscar comes in for the interview and my editor's in my ear saying: 'This is what we're going to do …' I went: 'Shut up! Listen!' I did what you do when it's a big story. I clicked into overdrive. And that evening went by in a nanosecond because I was so excited by this huge story. It was a sporting story and a proper Paralympics story."

Balding happily reels off her sweetest memories. "I was really lucky. I got to lots of events and that makes a huge difference. I was at the diving, watching Mo Farah on the telly with Miranda Hart, who is a great friend, just before Tom Daley fell in the pool in celebration. That was such bizarre fun. I was really moved by the show-jumping medals – and the boxing. The night Nicola Adams and Katie Taylor won. What an atmosphere. How come there were so many Irish there? How did they manage to get all those tickets? Oh my God! Amazing. I also felt hugely privileged to watch Phelps in his final races. Coming back to win again after losing to Chad? That was immense. And I watched Katherine Grainger win gold and I had tears running down my face. Jonnie Peacock and Hannah Cockroft were great too.

"This summer changed things and the challenge is to harness it and make it last so your kids say: 'I want to be an Olympian' – or if your kid is an amputee they say: 'I want to be a Paralympian.' That's different from wanting to be famous. That's a big deal. I know sport can change the world and that matters to me."

For many years Balding struggled to find a place for herself in a family obsessed with racing. "I hated that world. I deliberately didn't engage with racing because I thought: 'This excludes me.' Maybe that's why I now try so hard to not exclude people."

There is an involving immediacy to Balding's wry but often painful account of the way in which women have always been demeaned in her family. "Oh, it's a girl," Balding's grandmother said when Clare was born. "Never mind, you'll just have to keep trying."

Her relationship with her father, Ian Balding, the former champion trainer, is more tangled. "I was very nervous about him reading the book because I've shown him a mirror. I was honest when I could've sugar-coated it. But I thought: 'No, it's important, because he did say those things to me.' There's that line where I ask him: 'Do you love me more now that I'm thin?' and he said: 'Yes, I do.' That hurt me for a long time. When you get hurt like that you need to say it loud because he needs to know he was wrong."

Ian Balding read the book and responded with a note that said 'Brilliant – x'. Have she and her dad since discussed these issues? "Don't be silly," Balding says, chiding me playfully. "Don't be ridiculous! How's that going to happen? That would be like therapy. Dad is charming and funny. He's had an interesting life. But it hasn't turned him into a feminist – unfortunately. But I think the book's made him understand why I am – possibly."

For Ian Balding, 1971 will always be the year that his horse, Mill Reef, won the Derby. And yet, as Balding points out, it will never be remembered as the year she was born. "Absolutely not," she laughs.

Yet the book centres on Balding's need to be true to herself. "We all get hoodwinked into thinking we have to be a certain way. It takes strength to be yourself. I've learnt how to do that – and part of that is down to Alice [her partner]. Alice is so important."

Balding suggests that, when going out with men, she should have been browsing in a "different section of the library". And yet her parents have accepted Alice. "They were initially worried it would affect my career. And they worried because society has created a shame around homosexuality, which is why all these sports stars won't come out. They fear it will affect their sponsorship. I think it affects their performance not to be honest. I hope this summer showed if you work hard and you're consistent nobody gives a shit. They really don't. That's a huge leap and I'm very lucky it happened to me."

We finally reach Shoreditch and Balding's arrival at a 'literary salon' results in a throng of uber-cool men and women reacting in communal delight at first sight of her. But Balding is nervous on the night before her book's publication and we find a temporary escape before her reading. Alone on the roof, hearing the wailing police sirens as darkness sinks across London, Balding relaxes again. She laughs loudly when revealing that: "I got asked this morning to do a naked cover shoot by some magazine called Closer. I've never heard of it but I think it's like Heat."

A day later the French edition of Closer publishes topless photographs of Kate Middleton. But, oblivious to such machinations, Balding smiles. "Of course I'd never do it but I found it hysterical. You should've seen my mother's face when I said: 'Mum, I'm really considering it.' She half believed me. I'm just riding the wave because you can't be flavour of the month for anything longer than, well, a month. So enjoy it. Keep working hard. And remember, I didn't turn into a different person this summer.

"This is my life and my career and I want to be around for the long haul. I don't make decisions for money or popularity. I do things because they're right for me and they're interesting and challenging. And, like this summer, because they matter to all of us."

Clare Balding's My Animals and Other Family [Viking] is available at www.guardianbookshop £13