How to Measure the Effectiveness of Your Developer Program

This article was prepared by a guest contributor to ProgrammableWeb. The opinions expressed in this article are the author's own and do not necessarily reflect the view of ProgrammableWeb or its editorial staff.

Metrics can be a murky area for many developer programs. There is no shortage of things to measure, including (but certainly not limited to) API calls, revenue generated by developers, signups, Web visitors and event attendees. The disconnect tends to occur when programs either aren’t measuring the right things or aren’t making effective use of the metrics they do collect.

The key to developer program metrics is alignment--namely, alignment with business goals and objectives. Developer program metrics should be aligned with these objectives. In short, it’s how the developer program will contribute to the success of the overall business.

When properly aligned with business goals, program metrics allow us to:

Communicate priorities to and motivate developer program teams, and show them how they contribute to the overall business

Measure progress toward goals and objectives

Forecast if those goals and objectives can be reached

Make improvements to strategy, tactics and resources

Results vs. Activities

These program metrics are then aligned with activity metrics and community metrics.

A common mistake organizations make is focusing program metrics solely on activities--things like number of events participated in, blog posts written, and so on. Program metrics should focus on results and outcomes that contribute to the overall business, such as:

Number of API calls or SDK downloads

Number of active developers

Number of apps/services successfully integrating or using an API

Developer satisfaction/net promoter score.

Program metrics reflect our developer program strategy: Is it sound? Does it deliver value to our business?

Activity and community metrics lie beneath program metrics. Activity metrics allow you to track the things you are doing to achieve program goals and metrics, while community metrics track progress in building a developer community. Combined, they show the effectiveness of your tactics, and how they are contributing to the program metrics.

Activity metrics take on several different forms, all focused on the things a program and its team does:

Functional metrics around certain job roles, such as promotion or evangelism

Deliverable metrics, such as number of support articles written or hackathons attended

Developer cycle stage metrics, such as Time To First Hello World or the percentage of apps that pass a certification process

Similarly, there are many potential community metrics that illustrate the size and depth of your developer community and how your program interacts with and supports it. These can include:

Metrics that show developer adoption, such as number of mentions on Stack Overflow and number GitHub project starts, or number stars/forks for company-hosted GitHub projects

Metrics that show community support, such as percentage of answered/unanswered questions on Stack Overflow

When applied correctly and taken together, your metrics should help you see how your program strategy is working and contributing to the overall business goals (in the program metrics), and whether your tactics are contributing toward your program’s success (in the activity and community metrics).

Keep in mind there is no one specific correct set of metrics for every program; the best fit differs from program to program based on the overall business, its strategy, program resources and more. But every metric should pass the “So what?” test. Ask “So what?” after every metric. What does the number tell us? What does it mean? Does it tell us something we can act on, or show how what’s being measured contributes to our overall success? If you can’t answer these questions, chances are the metric isn’t relevant.

Metrics in Action

Beyond simply showing that your developer program is doing “something,” metrics can be extremely useful in helping to make adjustments to your strategy and tactics.

Take, for example, the chart below. It shows the number of developers who progress through the various stages of a developer program:
The chart shows the number of developers that enter this process (1,000), and some examples of the metrics that could be used to measure success at each stage. Notice that as the developer travels further through the process, the metrics evolve from activity- or community-driven to higher-level program metrics. This is that alignment we’ve discussed in action.

By having the right metrics in place at each stage, we can also see where we’re losing developers in their journey, or where our program and its tactics need enhancement. The chart below shows some general problems in each stage, and the metrics that can help illustrate them.

Rather than simply shooting in the dark when trying to make adjustments to your program and activities--or just moving to a shotgun approach to maximize the number of developers coming into the funnel and not worrying about how many fall out--this data (and organization of that data) will help you get a better grasp on exactly where and how your program is succeeding and where it needs to improve.

This is part of our series How To Build a Strong Developer Community. In part 4, we will look at how to take control of the onboarding process by minimizing the hurdles developers will face when they decide to move forward with your product.

About the author:Carlo Longino
Carlo has more than a dozen years' experience in the mobile industry, beginning just after the turn of the century when he worked for Nokia at its headquarters in Finland and mobile data services were in their infancy. Before joining WIP in 2010, Carlo worked as a freelance consultant and writer while he completed an MBA; prior to that, he was Senior Analyst for Floor64, a Silicon Valley-based analyst firm, where he covered the mobile and fixed telecom industries. He also helped launch and spent five years running TheFeature.com, a thought leadership site owned by Nokia. Carlo is a well-known mobile blogger at MobHappy.com, and his writing has also been published in The Wall Street Journal, Business 2.0 magazine and on Dow Jones Newswires