Test Market Considerations: Countries, Timeline, UA and More!

At Kongregate, all of our published mobile games go through a rigorous test market period (a.k.a. geo-locked launch) lasting at least a month. This is an important phase of the mobile game product cycle, as it allows the team to gauge the strengths and weaknesses of the title through data-driven KPIs, and to make changes to address any issues we observe before we release the game globally, whether they are crashes/bugs, tutorial/retention drop-offs, monetization or ad engagement. In this blog post, we share our learnings from running test market campaigns, with a focus on the pros and cons of running tests in different parts of the world.

Phase 1: The Philippines

The Philippines is a popular country for testing. Players are cheap to acquire with Cost-Per-Installs (CPIs) typically ranging from as low as $0.30 to $1.00. This allows developers/publishers to hit a meaningful number of installs quickly and stay within budget. The caveat is that games with a niche theme and/or art might see higher CPIs; for example, our recently launched Pocket Politics (iOS/Android) saw a much higher CPI because of its American political theme.

For stability and compatibility testing, the Philippines is a great place to consider, as one will observe a wide range of devices and OS versions. Conversely, the in-game KPIs collected in this country are often not reliable or indicative of what KPIs might look like in the U.S. and countries in the Western world. Players are less likely to stay in the game and less likely to engage and make purchases, so don’t despair if you are seeing low early retention and monetization numbers!

Phase 2: Scandinavia (Sweden, Denmark, Finland and Norway)

Once you have verified and improved that stability of the game, you can expand to additional regions. The Nordic countries are great to run tests in because the player behaviors resemble those of U.S. and other Western world countries’ players, while CPIs are usually lower than those of English-speaking countries like Canada, Australia and New Zealand (expect CPIs less than $4).

We often run tests for early retention (D1 and D7), ad engagement, and monetization in these countries, as the KPIs will give us a good idea of what they might look like once a game is launched globally. It is also possible to run A/B tests in this region and get meaningful results. One thing to keep in mind is that since these players are acquired through paid campaigns, KPIs will generally look higher than global metrics. For example, a D1 retention of 50% might turn into 40% globally when you get a mix of organic and paid players, as well as players from both high-performing and low-performing countries.

Phase 3: CANZ (Canada, Australia and New Zealand)

These three countries are the most popular among mobile game developers and publishers because players here most closely resemble those in the U.S. market and thus metrics can be more relied on to estimate U.S. KPIs. As mentioned before, keep in mind that players acquired through UA often show higher KPIs and engagement than those of organics.

The downside to running your test market in these countries is that the costs could be very high, as you will be competing with other developers and publishers who might be willing to bid up the prices to get volume faster.

Practical Things to Keep in Mind

Timeline

The length of the test market period will depend on the metrics you want to test and gain statistical significance on, as well as if any updates are needed to address any under-performing KPIs and/or fix any bugs. For example, to test for D1 and D7 retention, budget your time to get enough players past D7 and make changes to the game. Keep in mind that unless you are flexible with your global launch date, you likely won’t be able to address and test all issues, so it’s important to prioritize!

Budget

To determine what you will be able to test within a given budget, come up with an estimate of CPIs and install counts needed: CPIs vary by region as well as genre (i.e. casual games see lower CPIs and mid/hard-core titles see higher CPIs). A handy calculator to see how many installs you need based on the metrics you want to test can be found here.

User Acquisition

Ad Networks

To run UA campaigns to get installs, you will need to set up accounts with ad networks. Examples include Facebook, Unity and Supersonic. Different networks can bring in traffic of various qualities, with players from Facebook Ads usually yielding the highest KPIs.

Marketing Creatives

Your CPIs will also be affected by the quality of these assets, so it’s important to iterate on these if you are seeing a low CTR and/or CVR. It’s also a good opportunity to test different target audience groups to optimize your campaigns.

App Store Optimization

Test markets are also a good time to run A/B tests on icons and screenshots, as these will affect conversion rates. Google has built-in functionalities to set up these tests, so be sure to take advantage of them.

Platform Considerations

Approval Times

Remember that each time you want to push an update, you will need to wait for platforms to approve your builds.

Ratings

Monitor crashes each time you are releasing an update. Unexpected crashes/severe bugs that prevent gameplay can quickly lead to low ratings from players. Ratings are important, since they show both the platforms and the players the quality of your game. Average ratings are also harder to lift as the number of ratings grows.

And that’s it for now! If you are interested in how Kongregate can guide you through the test market cycle and help you publish your game, please email bd@kongregate.com for more details! For other ways of testing, and the pros and cons of each, check out our CEO Emily Greer’s presentation at this year’s Game Developers Conference.

Follow us on Twitter to keep up-to-date with our weekly blog posts.

More articles you might like:

Hello, my name is David Finseth and I am a Technical Artist at Synapse Games. I work across multiple games to create visuals that require some technical and artistic components. A big part of my job is working on particle effects for our mobile games. I am very passionate about this work and wanted to share my process and some tips for creating these effects with you. Particle effects are a unique tool that can add interactivity and responsiveness to your games. They excel at creating a lot of movement and impact. Particle effects can be used to create magical fireballs, swirling dimensional portals, or for directing the player's attention to a glowing treasure chest. First, I am going to break down the process that I go through when I make particle effects in Unity. Later I will go over some technical tips and tricks. Most of these examples are from work that I did on the games Spellstone and Animation Throwdown. The Process Breaking down the requirements: The first thing I do when I start to make a particle

pictured: Thumper Dark Souls is more likely associated with anguish than euphoria as a series, but it -- and other notoriously difficult games -- are where we can find some of gaming’s most pleasurable moments. If you work through your urges to throw your controller at a wall, you might be lucky enough to experience the joy that keeps us all coming back for more “YOU DIED” screens. We’re going to delve into what leads to these experiences, why they feel so good, and how they can be designed. That feeling? It’s called “flow.” Named by Mihály Csíkszentmihályi in the 1970s in response to interviewees describing the feeling as being carried along by water, flow is a deep, pleasurable immersion in the activity you are engaged in. Most commonly used in reference to creative pursuits like painting or performing music, this experience is absolutely found in games as well. To find yourself in a state of flow, first there are some parameters: You know your goal You know how to accomplish your goal You are receiving feedback

KONG: Hyper Hippo is a Canadian-based independent game developer best known for the hit idle game AdVenture Capitalist. How did you all meet and start making games together? CODY: Hyper Hippo sprang out of the imagination of Lance Priebe, the creator of Club Penguin. He assembled what he felt was a "dream team" for game development, then opened the floodgates and said, "GO!" TRISTAN: We all met while working on the virtual world "Disney’s Club Penguin." We wanted to try our hand at some new game experiments, create new IPs (Mech Mice, AdVenture Capitalist, etc.), develop for leading platforms (mobile, PS4, Xbox), and move away from the processes and politics of larger companies. KONG: For those of us not familiar, what type of game dev culture does Canada have? What sets Canada apart from other development hubs? CODY: Canada is similar to a lot of other "game" or tech hubs, but maybe just more apologetic? We’ve got our big guys like EA, BioWare and WB Montreal. But there’s also