Simon Tischer

“We are Awesome at the Crowd” — Interview with Testbirds founder Philipp Benkler

Testbirds tests its customers’ software using their own crowd. Quite an amount of time has passed since our last visit — so we asked Philipp Benkler, Testbirds founder and CEO, to bring us up to date.

Would you still call yourselves a startup?

That is of course a matter of your definition of a startup. We now employ roughly one hundred people. According to many conventional definitions, that means we are no longer a startup.

Yet it is not just a matter of size, but is also about the flair and how employees interact and do things differently than big companies.

As soon as a company grows, a lot of people will tell you: “You need different levels of organization, you need middle management, you need to get senior figures on board.” We decided to do things differently and still rely on flat organization.

“A perfect solution does not exist”

Was there a point in time when the informal style of startup communication reached its limits?

Communication always becomes a problem when you grow. We used to be three, then four, five, six people in one room. Everyone knew everything at all times. We answered questions from across our desks.

The first problems arose when we added an adjoining office. Suddenly no one knew what their colleagues were doing anymore. We then introduced a weekly breakfast where everyone shared what they had planned and if they needed help. It was extremely informal and unstructured. It worked really well.

Until we grew to 30 people. Everyone sat in a circle and only one person would talk – it took a long time and people hated it. At first, it was the best solution for communication that we had found up to that point, and just one year later it was awful. We had to think of something new.

Now, we have several offices and operate internationally, so we communicate several ways: face to face, but also with tools such as Skype, Yammer and Confluence.

What I want to say is: a perfect solution does not exist, but you can find good solutions for specific stages.

What did you do with the funds from your last round of financing in late 2015?

We bought ourselves expensive cars and have been living the good life since then. No, just kidding. For one, the money was invested in internationalization. We opened new offices in London, Amsterdam and Stockholm and expanded our existing offices. We also invested in technology. That includes our Device Cloud. Our platform gives our customers access to every kind of smartphone and tablet that exists in the world. That was not a trivial measure in terms of technology and it cost money.

What is the deal with having franchisees in Russia and Hungary?

Our partners have an exclusive license to market, execute and deliver Testbirds services in their region. The companies in those regions are responsible for sales, marketing, project management and supporting the crowd.

Are you not able to take care of that yourself?

Every one of the companies came to us, and we realized that they were a good match. Because we only have limited resources, we would rather give good partners a license and also get a slice of the pie.

Would you like continue on the licensor route or rather tap into new markets yourselves?

We plan to do both, depending on the appeal and potential of a market. That being said, franchisees have to come to us on their own accord. It is difficult to actively look for people in this field.

Have you increased your investments in cloud services because crowdtesting will not last for much longer?

Quite the opposite: the crowd works wonderfully, is our main business model and the core of our DNA. We are awesome at the crowd.

However, we did receive a customer request in 2012 or 2013 from a company that offers payment services. That company’s technology was causing problems, but only with specific combinations of operating systems with different browser and flash versions. In total, there were a bit more than 1,000 configurations and the customer did not know where there were problems or not. Our crowd was not able to reproduce the problem because most users’ browser and flash programs are automatically updated to the latest version. What we did was piece together a super-simple virtual machine builder, which we gave our crowd access to so they could test those thousand different configurations.

The product then went on the back burner for a while. After the market continued to move in the direction of test automation, we integrated it into our platform. We are the only ones who have united both solutions on one platform.

“When we started talking about the crowd, most people understood ‘Krautsalat’”

The preliminary hype surrounding the crowd was followed by disillusionment: including the dwindling pool of authors for Wikipedia and failed crowdinvestment projects. Are there now less, but more specific applications for the crowd?

When we started talking about the crowd, most people understood ‘Krautsalat’ (German for coleslaw.) We had to do quite a bit of explaining. Now the hype has just come to an end.

What you mentioned are actually two very different topics that cannot be compared. We work in the field of crowdworking and utilize swarm intelligence. We have a large number of application scenarios, and it works really well. A person used to have to test a program on ten different devices. That was a mindless and boring task. In contrast, if you have ten people testing something on one device instead and they also have fun thanks to gamification, then everyone benefits in the end.

That will definitely happen in some cases. As technology develops, algorithms will execute tests that we still perform manually right now. To be honest, that applies to all kinds of work, for example in Industry 4.0 or in the construction business.

What that means for us in concrete terms is that simple test cases will most certainly be performed using a higher degree of automation in the near future. I do not see more complex test cases being taken care of with automation in the medium term. When I say complex, an example would be everything in the realm of IoT. One problem would be how to realistically virtualize a connected car service without having to provide a real car.

In the user experience and user feedback areas, we are much further away from being able to replace real people. Some questions asked are: What do I like about an app? Is the flow good? How is the experience? Do I feel good? Does this app annoy me? I am really skeptical about algorithms with artificial emotions. As an example, we are working on a research project with LMU on smart algorithms.

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.