At the age of 40 Vincent Deary jacked in his job as an NHS psychotherapist, sold his house in south London, moved to Edinburgh and locked himself in a small room for two years to write a book. Or, more accurately, to think about writing a book. He spent the first year mostly writing Post-it notes. By the fifth year, having turned 45, he finally finished it and called it How to Live. This could have been an act of enormous hubris… if anybody had known about the title. Or the book for that matter. Except nobody did. He simply left it on his hard drive and went back to work as a research fellow at Northumbria University. In his mind, he says, he always envisaged it being the first part of a trilogy – though how that would work, given that he couldn't bring himself to give it to anyone to read, is just one of many aspects of his story that he concedes, in retrospect, doesn't make sense.

For five years he did nothing with the book, until, just turning 50, he finally got round to showing it to a friend, a former editor at Granta, whom he'd been at university with. "A few days later she sent me a text saying, 'I love your book!'"

From then on, he had a steady stream of people loving his book. Three agents competed to take him on, and then, having chosen one, a handful of publishers entered the bidding for it, including the one he finally went with, Allen Lane, Penguin's most prestigious imprint, which stumped up a six-figure advance for three books. He doesn't say how much exactly but concedes it was a "life-changing amount".

"And it does feel like my life is changing, which is funny because the book is about transitioning, and a lot of what I said in the book is happening to me now."

One reason publishers stump up a lot of cash is because this in itself drums up publicity, but in Deary's case it's still notable as the kind of advance that doesn't really exist any more, or at least not for a 50-year-old unknown. But in some ways, How to Live is a book about middle age, for the middle aged, or at least about how difficult it is to change, an affliction that's most likely to hit in middle age. Deary is the subject of his own book, man on the cusp of change. Or, for a lot of the book, what looks suspiciously like a midlife crisis. "I am dedicating myself to this project on transformation in the hope that it might be a transformative project," he says at the start. And throughout the book he divulges snippets of his life. How he has recently moved cities, houses, started afresh. How he's embarking on new things, including full-time single parenthood: halfway through the book his 16-year-old daughter, Vicky, comes to live with him. He's writing about change – the psychology of it, the philosophy – while living through a time of change. It's left frustratingly opaque in the book as to why. Though, of course, I have my theories…

You get the sense that it was quite a big disruption in your life, I suggest.

"It was a very deliberate disruption in that I decided to leave London and go and write this book… I was working as a clinician and I had a really good life in London, but I wanted to write this book, and it wasn't conducive to that."

He had already been thinking about it for a decade, he says. "For about 10 years, before I started writing, I'd been collecting notes. I had an old Dictaphone and I'd always be walking around keeping notes and reading and writing. I knew that I wanted to write something about us and how we live – the kind of creatures we are and how we get stuck and how we can get out of it, because I think that's always been my preoccupation really."

What he didn't want to write was "a psychology book", though as a therapist he must have come across a lot of material, I say. "I have but it was important to me that the book was authentic. I didn't want to write about 'cases'. So none of my patients are in the book. I write about me, about my friends, about my daughter, my family, because I think too many of the popular psychology books are monkey see, monkey do, and the author keeps himself in the position of being the expert. Whereas I don't want to say how to live. It's just me figuring out how to live, for me. I just wanted to go, 'OK, this is what I've thought. This is what me and my friends have gone through.'"

This is the small stuff of life. Do not go into this book expecting life-shattering events. Deary frets about the fact that he and his daughter have not yet established a Sunday evening routine. In one episode he talks about his friend Maureen's new bathroom. There are several paragraphs devoted to this, or, as he sums it up, acknowledging its absurdity. "Maureen misses her shower." And then: "The shower is not the stuff of movies, nor of philosophy, where recently the talk is all of Acts and Events… There are many ways to change. The end of the world doesn't have to be The End of the World."

Yet lives, like movies, he points out in his opening chapter, follow the rules of drama. Screenwriters refer to "inciting incidents" that set the plot in motion. Deary's own inciting incident, he says, was simply turning 40. "I just thought: if I don't make a go of it now when am I going to make a go of it?"

That was the push, he says. "And there was also a pull: my daughter had moved up to Scotland, to Aberdeen, with her mum, so there was a push and a pull, but in the end I took a big risk."

When I read the book, I tell him, I assumed there had been some terribly bitter divorce, and his wife had run off with his best friend. At the very least. But no, they divorced when his daughter was two, and remain good friends. "It's funny that comes across in the book. I was single but I didn't go north in search of a partner. It was quite the reverse. I wanted to be on my own… But it did feel like a major upheaval. It also felt like a leap of faith, so maybe it's the size of the transition, but I really didn't know where life was going to take me. I think it was my brother who said the other day: 'You know that risk you took when you were 40? It's paid off.' Because for a long time it didn't."

This is why the story of what happened next – the agents, the publishers bidding, the "life-changing" sum of money – is such an essential coda to the book. Because it's not clear from what he writes that it will end this way. And yet: it actually turned out to be a highly successful midlife crisis, I suggest.

"There's a Homer Simpson phrase: it's a 'crisistunity'. It felt a bit like that. There are certain decisive moments in people's lives that for some people come on as crises. But there is certainly a midlife point, and it's why I do make an allusion to that, in the beginning, when I say I could just carry on this life but I want to do something more."

What comes across most strongly in How to Live is just how bloody difficult it is to change. Or, as is more often the case, to handle change. Deary had a choice – to stay in London or to go – but many of those he cites in the book don't have a choice. Change has been thrust upon them – partners leave, work dries up, people die. "There are many ways our worlds can end," he writes in the book. "It may start as a distant rumour, a noise outside your small world, or an unexpected intrusion within it… sooner or later your current world will change, the present season will end."

And even the perfect people of Facebook, with smiling kids and sunny skiing holidays, are not immune. "They will fall for their lover, their dog will die, they'll have to move house, they'll go bankrupt, they'll die, they'll age and if they stay the same their circumstances will change so their old responses won't produce the same response from the environment. So even if they stay the same, that will mean change." The problem is that we are "habit machines". We suffer from "character sclerosis". "Left to [our] own devices, the result will be the downhill slide of a life dictated by whatever happened last, by happenstance and habit."

The book is not particularly cheering on this point. We think we are in charge of our destinies, that it's our consciousness that's at the wheel, but in fact, he says, we're mostly a collection of old habits and ancient routines that we drag around with us like a shrivelled limb. "Who's in charge?" he says. "Ask to talk to the manager. Who's in charge of your life right now? Are you living the life you want to lead?"

Gulp. But wouldn't a lot of people say: "Actually, I'm not in charge of my life because I've got responsibilities. I've got three kids to feed, therefore I have to do this job whether I like it or not"?

"True, but there are bits of it they could do differently, or want to do differently."

In print, he's less didactic than this. Part of the reason he did nothing with the book for five years, he says, is: "I didn't know what it was. I didn't know what I'd written." He has a point. Before I read it, his publishers had told me how "unique" it was, how it uses examples from Buffy as well as the Bhagavad Gita, how it's "a sort of memoir but not really a memoir". I'm told the story of the publicist in the Penguin office who read it, promptly jacked in her job, and went off travelling.

His voice is different. There's a whiff of Alain de Botton perhaps. A possible distant debt to Oliver Sacks. A touch of self-help for the literary set. Deary tries to go beyond psychology and its chosen words – cognition, affect and behaviour – and draw on older terms: mind, heart and will. "My profession dresses [them] up," he says, "claims them for its own and charges a fee for their management." And if it's a bit self-conscious at times, Slavoj Žižek this and Melanie Klein that, for the most part it keeps its feet on the ground.

Now, though, there's volume two to write, or as he calls it: "The Emperor Strikes Back part of the trilogy. It's the one about the triumph of the dark side, how we break, or get broken, or get stuck." And then, finally, "The Return of the Jedi", or how we get over it. He admits he's "slightly dreading it… oh, that's terrible, isn't it?"

But then one of his old habits is "catastrophising", he says, as well as being "fairly neurotic" and "an introvert". "I'm always scouting for what's the worst that can happen."

So what did he come up with?

"That everyone will hate it. I also wonder if that's why I waited so long to get it out there. I assumed people would say, 'Who does this guy think he is?'" They still might. Or they'll cheer him on as one of their own: middle-aged bloke takes risk. Makes it to the other side.