Inside Customer Success: test IO

Based in Berlin and San Francisco, test IO pioneered continuous testing for agile software development teams. By connecting its customers with thousands of professional software testers around the world, test IO gives teams near-instant real world test results on any device. Hundreds of customer-obsessed organizations -- and their developers, quality assurance engineers, and product managers -- rely on the power and flexibility of test IO’s self-service platform to ship their software faster with confidence.

Implementing Customer Success Management has revolutionized the way test IO relates to its customers and builds relationships. Amity had the chance to sit down with test IO’s COO and CEO, Christoph Püchel and Philip Soffer, to ask them about this transition, and to find out how Customer Success has affected the overall business strategy of the company.

Can you give us a brief overview of Customer Success at test IO?

Christoph: We started with a customer support mentality, but as it became apparent that our customers were on their own journey with us, we realized that we needed Customer Success Management, and we now have 4 dedicated Customer Success Managers. Onboarding is incredibly important for our CSMs, because we want to understand what our customers want to achieve, but also because there are many ways to use test IO. Our platform isn’t a one-way product, it’s very flexible, so it’s that much more important for us to understand the customer, and for the customer to understand our service.

When was Customer Success introduced at test IO, what were the driving factors for this?

Christoph: We realized about a year ago that reactive support wasn’t enough. When our customers had questions, we could answer them but there was not a value-added, follow-up process after that. This was unsatisfying and it was clearly a dead-end street because it didn’t deepen our engagement with the customer.

That’s when we decided to start with another approach, and as I’d begun studying Customer Success Management about a year and a half ago, it seemed like the logical evolution from what we’d been doing. It seems simple, but as soon as you start looking deep into it you realize that it’s a whole philosophy. We had to decide if we wanted to go that way -- the answer was yes -- and then we needed to figure out what that really meant.

And here’s one important realization we came to: we’re already in the Customer Success business! Look at it this way -- our customers work with us because they’re obsessed with having their own customers having a great experience with them. Making sure quality problems don’t get in the way of that is our job. So given that we’re in the customer success business and our own customers are too, it’s particularly important that we carry this philosophy into everything we do. It’s Customer Success all the way down.

So we wanted to do this very fully, and we began looking for a tool that could support our efforts, and that’s when we found Amity.

You have to meet the customer where they are, and where they are going to be.

Philip: It’s also important to note that our customers are dynamic over their lifetime with us, the customer we have in December isn’t the same that we have in May. When you realize that, the imperative for Customer Success becomes that much more important. You have to meet the customer where they are, and where they are going to be. That’s a key difference between a reactive and a proactive approach, and the only way to do that is Customer Success.

How is the Customer Success team structured at test IO and how has it evolved over the past years?

Christoph: At first, we had one dedicated CSM and a team of “Implementation Managers.” Then, we realized that one dedicated CSM wasn’t enough and we trained up our implementation managers as CSMs because we realized that a lot of what they were doing in the past was actually Customer Success, even if it wasn’t called Customer Success. We had to push this a bit further with the team, and we made sure that they had the right idea of what Customer Success Management meant. Now we have 4 dedicated CSMs and growing, and we segment our user base by geography, mainly for language reasons.

What is the biggest advantage of the way Customer Success works at test IO?

Christoph: Before our shift to a customer success mentality, we had to do a lot of guess work. Now we know and we understand what the customers want from us. We can foresee what the customer will need next, and what we need to do to be there for the next steps.

Philip: Building on this, once a CSM is in charge of a customer for a long period of time, they gain a better understanding and it creates a sense of accountability. Internally, this also creates an advocacy relationship between the CSMs and the rest of the team at test IO. When they can name the customer, tell their story over time, and know how the relationship has evolved, they can be much more forceful advocates if something needs to change.

Can customers tell that their Customer Success Manager is there to advocate for them?

Christoph: Most customers are excited that there is someone looking ahead for them. Customer Success has hugely improved our customer relationships.

What was the biggest challenge your team has faced, and how did you address it?

Christoph: I was a bit surprised when we encountered some problems transitioning from reactive to proactive. Talking about it is easy, but if you have to change your daily work and change your behavior, that’s harder than you’d think. So we hired a consultant to train us with Customer Success Management. We did a lot of role playing and workshops with her as a team. We learned together and moved from a “what happened” approach to a “what will happen” approach.

Philip: One of the biggest challenges for our CSMs was to learn how to ask our customers what success means to them. If you are accustomed to asking “what do you want me to do now?” and you have to start asking “where are you going and how can I help you get there?” it takes some learning and practice. Our CSMs are the world’s experts in what they do -- nobody understands crowdtesting better than they do -- and once they know where you are going, they know exactly how to get you there. But asking “where do you want to go” is actually not as easy at it sounds, it’s a skill that has to be taught.

We learned together and moved from a "what happened" approach to a "what will happen" approach.

What is the most powerful part of your Customer Success process? Do you think your process is different from other companies?

Christoph: What differentiates us is the fact that we are dealing with human beings at both ends. Out there, there are real human testers working on our customers’ behalf, so it’s not just about understanding a platform, an application, or a program logic. Our CSMs have a unique role because on one end they have to deal with our (human) customers, and on the other, they have to understand our (human) testers too. Our CSMs aren’t advocates for the testers -- we have a dedicated Community Manager for that. However, they have to understand how the testers think and work, and they make suggestions about how our Community Manager can lead the testers to meet the needs of our customers. This is the most challenging thing but also the most exciting thing: test IO is so powerful and adaptable because we are dealing with the world’s most flexible platform technology -- the human brain -- and we aren’t just telling our customers about a machine, or a program.

What role does Customer Success play in developing the overall business strategy at test IO?

Christoph: Long-term customer relationships are crucial for test IO, and Customer Success is crucial for long-term relationships. Some of our customers come to us believing that they’ll need our service for just a couple of months, and once they’ve done the testing and relaunched their product, they’ll no longer need testing. Most of those customers find out working with us that that’s not how it works. Software quality is an ongoing process, and we deliver ongoing value by being part of that process and standing by the customer’s side, making suggestions based on our experience.

We test more software in a day than most organizations test in a year, so we’ve pretty much seen it all. Without Customer Success we’d only be supporters, but we are now able to be a good starting point and we provide expertise for customers who are entering the next stage of their development.

Philip: And CSMs have an army of smart and adaptable human beings behind them, so they are actually able to create solutions for problems in ways that others would struggle to because they don’t have the brain power, and in our case, the human power. We are able to identify patterns of challenges that customers face, and we can build that back into the platform and make it available for everyone. Our Development team is sitting right next to the Customer Success team, and the process is much more experimental than it might be elsewhere, it’s a very rapid feedback loop.

Without Customer Success we’d only be supporters, but we are now able to (...) provide expertise for customers.

What type of metrics do you watch closely?

Christoph: The most crucial KPI is the activity of our customers: how often do they log in, how often do they set up a test run and work with reports? This is very closely followed by the engagement of the customer: how intensive and regular is the communication between us and the customers? Finally, we look at the life cycle stages of the customer very closely. We are very focused on our customers’ life cycles, and the onboarding stage is the most important for us because it allows us to start everything the right way. We have a checklist that our CSMs go through to decide whether we think our customers have reached the next stage. The renewing stage is also very important for us, of course. Usually, everything else is aligned with these three KPIs, for now, these are the 3 most important metrics for us.

How important do you think managing customer relationships has been to the overall success of test IO?

Christoph: It’s been absolutely crucial. A lot of our customers have a short-term mentality at first, and the closer we are to them, the more we know about them, and the easier it gets to make sure they understand the value of our service in the long term.

Since we’ve started implementing Customer Success, a lot of our customers who initially commit to only three months end up signing up for much longer. We have a pilot program, similar to the one Amity offers, and the conversion rates post-pilot went through the roof as soon as we started doing Customer Success. It’s sad to think about all the opportunities we lost before we adopted Customer Success Management, but now even though we always hope for the best, reality keeps on exceeding our expectations, and that’s amazing!

We are now signing customers up for up to 3 years, which would never have happened before we started doing Customer Success Management. And it’s interesting because the longer we stay with the customer, the more new challenges emerge, but also the more we know and understand them, which in turn makes it easier.

The more we know about the customer, and the easier it gets to make sure they understand the value of our service in the long term.

What is the culture of Success like at test IO?

Christoph: Success at test IO is something that we want to share as an organization. It’s not something that is about the Customer Success team, it’s about making test IO successful through every department, and each team depends on the next for success. I know that sounds a bit cheesy but it really is the basis for everything: everyone knows what Customer Success is all about, and we get great support from our engineering team and our product team, but we are also working closely with sales. We continuously communicate feedback from customers to the other teams to adjust our processes.

Philip: The one word to summarize it all is communication. That’s the one thing we’ve improved upon the most since we started doing Customer Success: we communicate better with our customers and one another and we all know that good communication means success.

What does a typical day look like for a member of your Customer Success team?

Christoph: We really focus on doing things right from the beginning, so our onboarding process is very detailed and intensive -- not because it’s difficult to use our platform but because we want to make sure that we have the right understanding of the customer and their needs. We are very conservative when it comes to whether or not we consider a customer as having advanced to the next stage, and whenever there is a change in their team, we put the customer back to onboarding stage to make sure the new person understands the product. Internally, a change in staff means “new customer” so… I would say that our CSMs have at least one onboarding session per day, if not more!

We also spend a lot of time tracking customer KPIs and trying to understand possible changes in activity and whether or not they are critical.

Finally, we look at the daily results of the test runs, and we make sure we are on top of what our testers deliver to our customers.

Whenever there is a change in their team, we put the customer back to onboarding stage to make sure the new person understands the product.

How can you tell if a customer is successful?

Christoph: We like to see our customers in what we call a “testing rhythm.” That is, they test regularly with us in a predictable pattern. For some customers that’s once or twice a month, others it’s once or twice a week, and some test every day. Once we see that rhythm, we know they’ve integrated our service into their development, quality assurance, and delivery process. On the other hand, if you aren’t doing it that way, it means you won’t get the most out of the platform: doing test runs now and then doesn’t give you continuous monitoring of your software and you only get isolated snapshots of your development process.

Sometimes customers have unrealistic expectations. They might think that they are successful if they can deliver perfect software in the first release, without a single bug. That would be awesome, but most companies aren’t like NASA. Most of our customers understand that their own goals are only achievable through good quality assurance and regular testing. In testing, the more and better you do it, the fewer bugs you’ll end up finding before each release. In our ideal world, our customer runs a test often enough that they don’t find many bugs each time.

The thing is, the customer might then think “why aren’t you finding more bugs?” because at the beginning they felt like they were paying us to find bugs! One of the things we need to get straight with the customer right away is that at first they will find a lot of bugs but as we work together over time, the number will decrease, and that means that we are efficiently working on the quality of their software. They’re not paying us for bugs, they’re paying us for quality -- and their own customers’ success.

What blogs, book, or news sites are you reading at the moment?

Christoph: I read your blog very regularly, of course. I am also currently reading Customer Success, which is a book version of some of Gainsight’s articles. I get some good white papers from the Customer Success Association, and I also read anything I can find online by doing research about Customer Success.

What does success look like in your everyday life?

Christoph: Making music all day long ... that would be great! Seriously, I can summarize it like this: if I see opportunities to change something for the better, and I try to achieve that and I am successful in doing so, that means success for me. Changing something for the better, whatever that means, in every life situation. One example: I got married to my now-wife after spending 13 years together, and I think that that’s success.

What’s one piece of advice you have for teams who are just getting started with Customer Success now?

Christoph: The first thing I think you should do is to have a look at what you are already doing, and what might already be aligned with Customer Success Management. I think this comes first. When we started doing interviews for Customer Success Managers, an applicant told me “Hey, you have a team of CSMs already, right there” and of course I said “No, they’re implementation managers,” to which the applicant replied, “Well, they’re doing x, y and z, and that’s Customer Success!” And she was right. I would suggest trying to find out what you are already doing in terms of Customer Success and it might be far more than you imagine.

Then, be clear on the differentiation between Customer Support and Customer Success. They are two very different things, even if both talk to the customers and make them happy, the approach is completely different.

Finally, we realized quickly that change is hard and you need to train your Customer Success team. Don’t make assumptions or take anything for granted. Every now and then, we’re in a meeting and I tell my team what I have in mind, and I can see in their eyes that they don’t get the picture. Just because you explain something doesn’t mean it makes sense right away, and you need to think like your team. For over a year I’ve been researching the field deeply, and everything is quite clear to me now but I have to remind myself that I might be a couple of steps ahead in thinking about Customer Success, and of course the team is dealing with real-world problems every day, so sometimes they see the trees but not the forest.

Customer Success is a discipline where everyone needs to see both the forest and the trees. Once you figure out how to do that, you’re on the right track.

About the Author

Mathilde is the Manager of Digital Marketing at Amity. After moving from France to complete a degree in Political Science from McGill University, she made her way to Toronto in order to pursue her passion for Marketing and Tech.