As head of the department where "exotic" financial products turned toxic, he was at the centre of the financial storm in 2008 and beyond. We are meeting for lunch one day in January. He is a very engaging, slightly restless English man in his mid-30s. Though a great lover of cider, he orders a ginger beer with his pork pie.

"Those were scary days. You think: we are in a new paradigm. Nothing works any more the way it used to. My department's potential losses were hundreds of millions of pounds and several billions across the whole of the bank. We began to realise: this could sink the bank. In fact, we were bankrupt three or four times; our bank owed more than it owned. We were lucky to have a parent company with very deep pockets. I was struggling to keep it afloat. If the market had crashed further we would have gone down.

"All that started more than three years ago. Almost a year ago I left the bank, and I am still nursing myself; the night sweats have gone but I have this skin condition. It was stress. I pretty much engineered my own exit from the bank. Earlier they couldn't let me go, I was one of the few who had intimate knowledge of the products that blew up. In fact, in those last years they paid me more than ever. This was a bomb and I was basically the only one who could defuse it.

"How credit derivatives work is not easy to explain and I'll come back to it later. When credit derivatives were still young, the Financial Times called people like us the 'F9 model monkeys', after the key you press to get the algorithm to tell you how things are going; what the value is of your portfolio. 'Monkey' alluded to the fact that some of us didn't understand what the algorithms did.

"In the years before, we'd know in our heads to within a few thousand what profit or loss we'd made for the day, then press F9 and have it confirmed by our systems. When the crisis hit we would press F9 and get a number that was totally unexpected. We'd ask: how can we have lost so much money? What happened?

"You can imagine that all of a sudden our department began to receive a lot more attention. In the old days we would calculate our P&L [profits & losses] for the day, I'd send off a report to my superior and you'd hear no more about it. Now my report would be pored over by 10 risk managers in London, then 20 more in my bank's parent country.

"This was enormously stressful. You really don't want to be caught out by a risk manager in HQ in the parent country with a mistake. Because that risk manager won't go back to you, they'll go straight to the board. What was much more stressful still was that most in the bank didn't understand our products. Even the risk and compliance people who were supposed to be our internal checks and balances … We began to realise that we had to teach them how to monitor us. Then there were the people I reported to, who were getting calls from the people they reported to. I learned that the people high up know just enough for the role they're in.

"'Just enough' is not enough in an emergency. I would be on the phone for hours explaining to people of increasing seniority what we were doing. And I realised, they don't understand, not on a fundamental level. They will not be able to spot a mistake, correct us when we fuck up, or take an informed decision. It was down to us, and I was losing precious time talking to the top people. And to their underlings. This was when I would explode; when people would be sent to me with a question they did not themselves understand. I would tell them, 'Look, you're adding no value. If you tell me you don't understand the question you're asking me, that's fine. I will sit down and explain. But how can my answer to your question be of any value when you don't even understand what you're trying to get me to tell you?' I would get very mad.

"I learned about human nature in those days. The arse-covering … At one point we worked out a solution to the biggest of our problem positions. We'd have to take another loss but then the thing would be back under control. So we propose this and it gets shot down by the top. The bank did not want to acknowledge more losses at that point, and chose to let it fester and face considerably bigger losses later on. When we realised this, our team began to send emails, purely to cover ourselves: 'As discussed in meeting X, this is our proposal. I am strongly encouraging…' We knew they wouldn't go for it but we were not sending these emails to them. We were sending them to potential future investigators and prosecutors.

"It was during this time that I realised that all major banks and corporations are doing this. They are burying and pushing back bad stuff till the books for that fiscal year close, just to make the numbers look better. I don't trust their annual reports at all any more. They massage them until they look the way they want them to look.

"I said earlier that structured credit is not easy to explain to the bloke in the street. It covers a range of products to do with derivatives of some sort. The two terms here are credit default swaps or CDS, and collateralised debt obligations, or CDO.

"I am going to simplify. Suppose you are in some way dependent on a corporation or government. Maybe you are an investor who lent them money. Or a company that's in business with that corporation or government. In either case you worry about them defaulting and causing you to lose your money.

"So what you do, you take out a CDS to insure yourself. If the corporation or government defaults, the provider of the CDS will pay out. Great. But it leaves the following risk: what if the CDS provider itself defaults? Who is insuring the insurer?

"That would be the role of the CDO. They are exactly like a CDS but with two key differences. One, the corporation or government is replaced with a portfolio of assets. In our case the portfolio was a load of CDS. Two, the provider only pays us when certain conditions are met, which is usually after a certain proportion of the portfolio has already defaulted.

"It's on one of these CDOs where we had our biggest potential losses when the provider of a CDO we had bought started to get into financial strife. We'd bought a huge amount from this one provider because it was so unlikely to get into trouble that the possibilities of loss were miniscule. Of course, there was always the possibility that this provider would default and in normal markets other banks would have stepped up to that challenge. But this wasn't normal any more and by the time we realised we might need some extra protection people weren't answering phones.

"We had to be content with offsetting as much risk as we could by hedging. Hedging your bets when the bets are exotic credit derivatives is not easy. We had this algorithm that allowed us to calculate the trades we needed to execute to ensure we were 'dynamically hedged'. The program would know what positions to take on the financial markets, making sure that when one element in our portfolio lost value, another would increase. Our algorithm allowed the traders reporting to me to estimate what CDSs and currency needed to be bought and sold in order to maintain a balance.

"The program with the algorithms for 'dynamic hedging' was not a black box. I mean, the PhD guy working on my team, he wrote his PhD on that algorithm. Perhaps you might compare it to a game of chess. At the end of the day the algorithm reveals a certain position. When the crisis happened we ended up with positions that were very different from what we were expecting. But we did know the moves that each piece could make. It was a matter of reconstructing them, of back-casting and figuring out what had happened. We'd always find it, if not after hours and hours of trying and building and testing hypotheses.

"We'd go over all the variables, the possible chess moves if you will: currency issues, interest rates in all these currencies, prices of CDSs, prices of different CDOs, some of which are not traded very actively so their value needs to be estimated by the program. There's the potential for human error so we'd go: did you update this, did you update that? We'd create hypothetical 'example CDOs' that were similar to ours with which we could tinker to see if it threw up any clues. We'd be there till midnight but we'd get it, eventually.

"The team spirit with my direct colleagues, that was great. There were only a handful of us, like a Swat team, all from different countries because that's the City of London for you. And the environment was intellectually challenging, to put it mildly. It helped that I wasn't the one who had traded the products that were blowing up in our face. They lost our bank a lot of money. They lost our clients a lot of money. But I could be fairly dispassionate because it hadn't been me who sold them the products.

"Could this meltdown have been avoided? Should rating agencies have spotted it? Well, this is how it would work with the rating agencies when we were building a new CDO. They would tell us their parameters and criteria; if you meet this requirement, you get that rating and so on. And they gave out a free model so we could test our product and tweak our portfolio for the CDO until it fit, I mean get the rating that we wanted. We would do a lot of stress-testing ourselves too, of course we would. We'd pretend the market changed and run the models to see how our products would hold.

"But what happened during the financial crisis was like a perfect storm. In our tests we would assume the market moved, say, 10% – while in reality it rarely moved more than 1%. Now the crisis happens and suddenly the market moves 30%. Our models were based on what we saw as normal. Now we saw numbers behave in ways barely conceived possible.

"Say you are an insurer and in any given year about 3% of car drivers get into an accident. For decades this is the percentage. Then suddenly the rate jumps to 25%. It was crazy, crazy. They call it a 'black swan' event; something theoretically possible yet totally unforeseen that changes things forever. You believe for centuries all swans are white. One day, you see a black one. Your model collapses.

"One thing we didn't model for was politics getting involved. Fannie Mae and Freddy Mac, the mortgage insurers in the US, were taken into conservatorship by the American government. Technically this was a form of default and hence it triggered some of the CDS in our CDOs. Nobody saw that coming when we put the CDO together a few years before. How could we?

"It's easy for critics to say 'you should have planned for this'. I don't think they appreciate what a black swan event is; it is by definition something you don't plan for, as it is outside your perception of what is possible. I suppose it's like betting on a tortoise to win a horse race.

"I like to think that I am an honest and reasonable guy. If I see a blind person crossing the road I will get off my bike and help that person. That's me. What drove me all this time was to do a good job. I am the sort of person who gets the punctuation right in every email.

"In my final years I was making 400k including bonus. Our department was losing heavily but they couldn't get rid of me. It was weird how my pay really started to escalate during that time. They practically threw the money at me, I didn't ask for any of it! Each bonus time would come around and they'd say, we're paying you a bit more than we guaranteed to pay you. It's not much more, but it's a gesture of our thanks. And in the same breath they'd say, and for next year your guarantee will be X. Not once did I ask for more … How could I, they were paying me a lot of money. I knew I'd worked hard and I knew I'd helped the bank keep control of some gigantic risks, but that was just me being diligent. My grandad was a diligent milkman and my dad a diligent policeman, I was just being diligent in an industry that throws its money around.

"Four hundred is not bad for somebody who entered the bank at 18 and worked his way up, all the way from back-office. I was the only one in my high school who did not go on to university. It was not for me. I fell into this business. And I don't think I was in it for the money, when I was in high school my parents tried to bribe me into working harder. I never did.

"It was not great in those dark days of the crisis, being known to work in finance. Nobody in my social circle was in banking. My mother would feel compelled to defend me when somebody said something about bankers. Friends turned on me, on my wife. I would go to a party and I'd get fried by a socialist friend ripping into bankers. Sometimes I'd argue back, saying, look, I probably agree that bankers don't contribute as much to society as doctors, but let's extend the argument and let's start ranking every profession. Do you place doctors higher than teachers, or vice versa? Nurses? And customer service staff?

"I am still trying to hide from people that I used to work in a bank; I don't want the parents of my children's friends to know. It can get so ugly so fast. The problem is, people don't really seem to understand exactly what they are so angry about. Do I suffer from post-traumatic stress? I'd rather think I'm experiencing post-traumatic growth. After all I have gone through, I feel there's nothing I can't do."