Red Hat Open Source CommunityTag: Metricshttp://community.redhat.com/blog/2018-05-22T10:25:00-04:00Red Hat, Inc.On Humane Teams at Home and Around the Worldhttp://community.redhat.com/blog/2017/08/on-humane-teams-at-home/2017-08-17T08:45:00-04:002017-08-17T09:17:42-04:00Amye Scavarda<p><img alt="Pivotal Meetup" width="250" height="204" src="/images/blog/pivotalmeetup.JPG" /> I had the pleasure last week of seeing Dan Young and Emma Jane Hogbin Westby's talk on <a href="https://www.meetup.com/Pivotal-London-Talks/events/242224262/">Humane Teams at Home and Around the World</a> at the Pivotal London Lunch meetup, and came away with a lot to think about: how different cities do meetups, the choices that we make about how we work with teams, and what information informs those choices.</p>
<p>It's always a delightful experience to see how different cities do different meetups. Even though it was the middle of the week and pouring down rain, over 60 people came to see the talk! The structure of the meetup is around topics interesting to tech, not necessarily the most technical deep dive. It's almost like a bite-sized DevOps Day feeling, in a really lovely space in the Pivotal office.</p>
<p></p>
<p>Pivotal was a great place to be able to host this talk as well, focusing on the differences between co-located and distributed teams. Dan took the side of co-located teams, teams that work all together in one space, and Emma Jane advocated for distributed teams where no one works in the same space and has to rely on virtual tools and collaboration software. Part of the focus for each was that Dan's based in a city, where Emma Jane talked about living in rural areas where there was no co-located options, and learning to work with that in order to work with other teams.</p>
<p>Important takeways:</p>
<ul>
<li>What does it mean to be a humane team? What does an 'unhumane' workplace look like, and how can you, as a manager, help resolve this for your teams? They defined this from a Kent Beck quote in XP Explains: "I had begun to notice that the more humanely I treated myself and others, the more productive we all became."</li>
<li>Working as a humane team means living by your values, and for Dan that means using a framework to help define your work. He advocated the Extreme Programming framework as something that was effective for him and his teams, Emma Jane talked about the Agile Manifesto as a way to be able to identify common values between teams.</li>
<li>Most importantly for me, I loved the clarification that you, as an employee, don't get to choose your workstyle between colocated and distributed, you choose companies. Alternatively, as a founder, you get to chose the shared values of how your team will function - but as an employee, your choice is applied in a different way.</li>
</ul>
<p>Finally, they left the audience with a github link for slides, which is probably my favorite new way to leave slides, and so I'll <a href="https://github.com/emmajane/humane-teams">leave it here for you all</a> as well.</p>
<p><em>Image courtesy of Amye Scavarda.</em></p>
<hr/><p>This article originally appeared on
<a href="http://community.redhat.com/">community.redhat.com</a>.
Follow the community on Twitter at
<a href="https://twitter.com/redhatopen">@redhatopen</a>, and find us on
<a href="https://www.facebook.com/redhatopen">Facebook</a> and
<a href="https://plus.google.com/u/0/b/113258037797946990391/113258037797946990391/posts">Google+</a>.</p>
In Search Of... Software Usershttp://community.redhat.com/blog/2017/08/who-are-my-users/2017-08-01T13:41:00-04:002017-08-01T18:42:55-04:00Brian Proffitt<p><img alt="Searching" width="250" height="167" src="/images/blog/binoculars.jpg?1501645116" /> When I was kid, there was a pretty cool documentary show called <a href="https://en.wikipedia.org/wiki/In_Search_of..._(TV_series)"><em>In Search Of…</em></a>, which examined various paranormal (Bigfoot, The Bermuda Triangle) and natural (killer bees, hurricanes) phenomena and mysteries. This may seem dull for a pre-teen kid, but it was narrated by Leonard Nimoy, so I was pretty much all in.</p>
<p>One of the things this show (and others like it) does is avoid any real conclusions. Episodes paint a picture of many possibilities, present some counter arguments, then wrap up with a vague innocuous statement like "Is there really a Bigfoot? Only time may tell."</p>
<p>(Yeah, not really big on the mystery solving.)</p>
<p>As entertaining as this show is, it's not really the vague approach community team members want when trying to solve <em>their</em> greatest mystery: "Who is using our software?"</p>
<p></p>
<h2 id="plain-as-the-nose">Plain as the Nose?</h2>
<p>It may initially seem like a dumb question: why <em>wouldn't</em> you know who your software users are? After all, you know there are downloads, you see people participating on mailing lists and chat channels–what's the problem?</p>
<p>The problem lies in the fact that what you see in your project's community channels is a representative sample at best, and likely not even a good sample at that.</p>
<p>Depending on your community, what you may be seeing on your communication channels (mailing lists, forums, IRC, Slack, StackOverflow, GitHub) are just the really passionate members of your community. There is an old axiom in the journalism community: those who write letters to the editor are either really, really happy about something, or really, really angry. The same notion can be applied to active and vocal community members. I have used some form of Mozilla Firefox since the days when it was known as Netscape Navigator… and to date, I have never posted a single word on any forum or chat channel. I just use it. If it ever breaks, I search online for a solution and consume that solution. None of these things, though, gives Mozilla and Firefox's team any notion of who I am and what are my use cases. This blog, actually, may be the first time they've heard of my use.</p>
<p>Even numbers are a tricky business. Many people in the software industry like the "golden arches" ideal of being able to say "<em>X</em> users served,
just like McDonald's and their hamburgers. But downloads are a very hard thing to even track accurately, especially if your project has mirrors involved in the dissemination process. And, even if you think you have some good numbers, my colleague Michael Scherer has made a <a href="https://community.redhat.com/blog/2015/09/lies-damned-lies-and-downloads/">compelling argument</a> why downloads aren't something you want to, er, hang your hat on.</p>
<h2 id="less-nails-sticking-out">Less Nails Sticking Out</h2>
<p>This wasn't always the case, of course. Before the turn of the century, when all consumer and server software was shiny and new and pretty much broken. Free or commercial, early software that hadn't been born on a mainframe and was being consumed by the general public and a new breed of IT manager was clunky and likely to fly apart in spectacular blue screens of death or little computer icons with x-ed out eyes.</p>
<p>In those wild west days, books, magazines, and an Internet in its public infancy were the places to find solutions, and online communities were intimate affairs where a lot of people would come in for help. Developers were more apt to communicate with the Average User because everyone was too busy figuring out how all this stuff worked to stand on much ceremony.</p>
<p>Buggy software created a lot of nails sticking out, and there was a lot of hammering going on. These days, there are still nails to be hammered, but not as many and (usually) not as potentially catastrophic. Software is easier to install.. you can even install an entire Linux distribution in minutes and not worry about XF86Setup frying your monitor if you chose the wrong refresh rate.</p>
<p>The interoperability and ease of software across all platforms and licenses has, I believe, created less strong <em>user</em>-based communities, because when things are running smoothly, the need to band together decreases. It doesn't disappear completely, of course, but the "Average User" tends to be much quieter than the louder disgruntled this-is-broken-and-I-want-it-fixed-now users and the passionate this-is-the-best-thing-since-sliced-bread advocate.</p>
<p>This leads to the challenge for community members who very much want to know who's using their software: contented users are quiet users, but you want to know what <em>they</em> are getting from your software so they <em>stay</em> contented.</p>
<p>Ask many community participants and leads what their biggest unknown is and they will give you a variant of this same answer:</p>
<p>"Who are my users and where are my gaps against my use cases?," replied Fedora Action and Impact Coordinator Brian Exelbierd. "Specifically, I'd like to figure out how to increase the number of feedback-level contributors who I could then court for greater contribution. I'd also like to figure out how much of the 'strong feelings' that are heard in Fedora are actually felt by the non-vocal members of the community. I worry at times we are trapped by a loud voice who is not representative."</p>
<p>CentOS Community Liaison Rich Bowen agrees. "Who are all of the people that are using the project but are not connecting back with the community in any way? Finding these people, and being able to talk with them about their experiences, would hugely inform everything else that we do."</p>
<h2 id="seeking-answers">Seeking Answers</h2>
<p>Tracking users in a given community is hard, but not impossible. Right now, the best tools for discovering information about your users include:</p>
<ul>
<li>User Surveys</li>
<li>Registration</li>
<li>Forum/mailing list analytics</li>
</ul>
<p>You will not I did not mention more aggressive user tracking, like opt-in reporting from client applications. Historically, apps that "phone home" have been an historical non-starter in the FLOSS community, due to serious privacy concerns. Even the more passive methods can raise privacy issues.</p>
<p>Moving forward, we will investigate each of these methods and try to determine what process might work for your project.</p>
<p><em>Image courtesy of <a href="https://www.flickr.com/photos/edith_soto/7271415680">Edith Soto</a>, under <a href="https://creativecommons.org/licenses/by-sa/2.0/">CC BY-SA 2.0</a> license.</em></p>
<hr/><p>This article originally appeared on
<a href="http://community.redhat.com/">community.redhat.com</a>.
Follow the community on Twitter at
<a href="https://twitter.com/redhatopen">@redhatopen</a>, and find us on
<a href="https://www.facebook.com/redhatopen">Facebook</a> and
<a href="https://plus.google.com/u/0/b/113258037797946990391/113258037797946990391/posts">Google+</a>.</p>
Open Source Stats--But What Do the Numbers Mean?http://community.redhat.com/blog/2017/04/open-source-stats/2017-04-21T10:30:54-04:002017-04-21T10:47:30-04:00Rich Bowen<p><img alt="rdo" width="200" height="200" src="/images/blog/rdo-logo.png?1478100396" /> I recently sent a report to project management containing some numbers that purport to describe the status of the <a href="http://rdoproject.org/">RDO</a> project.</p>
<p>I got a long and thoughtful response from one of the managers—we'll call him Mark—and it seems worthwhile sharing some of his insights. To summarize, what he said was, don't bother collecting stats if they don't tell a story.</p>
<p></p>
<h2 id="focus-on-the-goals">1. Focus on the Goals</h2>
<p>Listing a bunch of numbers without context—even with pretty graphs—doesn't tell us anything unless you relate them to goals that we're trying to achieve.</p>
<p>Several weeks ago I presented a "stakeholder review" to this same audience. Any statistics that I present in the future should be directly related to a goal in that review, or they are just meaningless numbers, and possibly a distraction, and, worse still, might cause people to work towards growing the wrong metric. (Google for "<a href="https://goo.gl/sjxPLc">be careful what you measure</a>" and read any of those articles for more commentary on this point.)</p>
<h2 id="focus-on-the-people">2. Focus on the People</h2>
<p>One of the stats that I provided was about how certain words and phrases feature in the questions on <a href="http://ask.openstack.org/">ask.openstack.org</a>. Mark looked beyond the numbers and saw three people who are very active on that website, two of whom are not obviously engaged in the RDO community itself. Why not? How can we help them? How can they help us? What's their story? Why are we ignoring them?</p>
<h2 id="focus-on-the-blips">3. Focus on the Blips</h2>
<p>In February, our <a href="http://twitter.com/rdocommunity/">Twitter</a> mentions, retweets, visits, and so on, went through the roof. Why? And why didn't we do that same thing again in March?</p>
<p>As it turns out, in February there were two conferences that contributed to this. But, specifically, we captured a lot of video at those events, and the Twitter traffic was all around those videos. So clearly we should be doing more of that kind of content, right?</p>
<h2 id="ignore-the-stuff-that-doesnt-seem-to-mean-anything">4. Ignore the Stuff That Doesn't Seem to Mean Anything</h2>
<p>We track "downloads" of RDO, which roughly speaking means every time someone runs the <a href="https://www.rdoproject.org/tripleo/">quickstart</a> and it grabs the RPM. Except RDO is on a mirror network, so that number is false—or, at best, it reflects what the trends might be across the rest of the mirror network. So we have no idea what this metric means. So why are we bothering to track it? Just stop.</p>
<h2 id="ask-not-the-usual-suspects">5. Ask Not-the-Usual-Suspects</h2>
<p>This last one wasn't one of Mark's observations, but is what I'm taking from this interaction. We tend to ask the same people the same questions year after year, and then are surprised that we get the same answers.</p>
<p>By taking this data to a new audience, I got new answers. Seems obvious, right? But it's the kind of obvious thing we overlook all the time. Mark provided insight that I've been overlooking because I'm staring so hard at the same things every day.</p>
<p>By the way, I've presented Mark's insight very bluntly here, because it's important to be clear and honest about the places where we're not doing our job as well as we can be. Mark's actual response was much kinder and less judgmental, because Mark is always kind and supportive.</p>
<hr/><p>This article originally appeared on
<a href="http://community.redhat.com/">community.redhat.com</a>.
Follow the community on Twitter at
<a href="https://twitter.com/redhatopen">@redhatopen</a>, and find us on
<a href="https://www.facebook.com/redhatopen">Facebook</a> and
<a href="https://plus.google.com/u/0/b/113258037797946990391/113258037797946990391/posts">Google+</a>.</p>
Lies, Damned Lies, And Downloadshttp://community.redhat.com/blog/2015/09/lies-damned-lies-and-downloads/2015-09-02T09:42:00-04:002015-09-02T18:33:31-04:00Michael Scherer<p><img alt="Download Icon" width="200" height="201" src="/images/blog/download-icon.png?1478100396" /> Most people who went to school learned to count, a skill basic enough to be mastered by everyone. Yet, we often forget to use the necessary amount of critical thinking with counting, which leads to some problems–especially in free software.</p>
<p>Working for a company that started as a Linux distribution provider, it will surprise no one by saying that my team is often asked to acquire some statistics about the downloads of packages in order to estimate community engagement and the number of users of the project.</p>
<p>Though my job is not a community liaison, I work with enough of them to understand they need a way to measure the growth of their respective communities and the impact of their work. The professionalization of the community metrics field has even spawned its own <a href="http://flosscommunitymetrics.org/">mini-conference</a>. Similarly, my colleagues working on the commercial side of the company have long since embraced the use of metrics and KPI offered by numerous CRM software platforms that provide reports to the upper management and enabling them to have a synthetic view and much-needed feedback on their initiatives.</p>
<p>But when people ask me for such statistics, I often explain why this is not the best idea. While download statistics are good for trends in community growth, they are not the sole sign of community health. This is due to three points I will now explain in detail.</p>
<p></p>
<h2 id="mirror-mirror">Mirror, Mirror</h2>
<p>The first one should be quite obvious to anyone who worked on the infrastructure side of a Linux distribution. If a project is working well, it get more users and more deliverables, which translates into an exponential growth of the network bandwidth used on the infrastructure side, which in turn causes some scaling challenges.</p>
<p>While academics have proposed for years to distribute packages using peer-to-peer (P2P) methods in various <a href="www.camrdale.org/apt-p2p/presentation.pdf">research</a> <a href="www.prism.uvsq.fr/~preda/papers/edos-demo.pdf">papers</a>, most projects I am aware of haven't adopted that mode of distribution and still rely on volunteers providing mirrors to the public–either with FTP or HTTP as the transfer protocol.</p>
<p>There are various reasons for this mirroring practice, from corporate acceptance (no firewall fiddling, easy to mirror in the LAN), to ease-of-deployment from a community point of view, since this is a battle-tested and well-understood method.</p>
<p>Yet that doesn't mean that the P2P idea is useless, since it has been used by game publishers such as Blizzard for years. Even the newly released Windows 10 is rumored to offer this feature. But if the communities around free software are not implementing P2P, it might just be because P2P is not solving their problems.</p>
<p>Therefore, any kind of accurate statistics would require getting the logs of the mirrors' servers, a operation problematic both from the privacy and execution angles. Most sysadmins I know prefer to set up a periodic task and be done with it, not doing the maintenance of yet another system more complicated than a single rsync command line. Given that mirror admins are doing this on their own free time, no one really wants to add more burden on their shoulders.</p>
<p>Since there is likely a huge variation of use between mirrors due to being located in different countries and offering between 10 Mb to 10 Gb of bandwidth, you can hardly extrapolate from the logs of one single mirror and just multiply by the number of mirrors to get a estimation. Most distributions and big projects also have a adaptive method of directing a user to a download mirror, so a drop or a peak around one mirror could just be the effect of an outage or the creation of another mirror in a nearby network location.</p>
<p>And this does not even take into account the people using download accelerators or similar software promising to accelerate downloads, but mostly just results into more load on the mirror network by opening more network connections, consuming more memory and resources, and skewing download counts in the process.</p>
<p>There is also the issue of not knowing what we want to actually count. An rpm updated every day would have many more downloads than one updated every month, so we would see a difference from 1 to 30, depending on the policy of the project and the downstream users and how we count and present the data. Or, for software projects with a more regular cadence, what about downstream distributions who offer their own packages? Who would then not be counted by your set of mirrors? What about language-specific archives such as CPAN, PyPI, or maven, which would further isolate you from your users?</p>
<p>In theory, this issue could be solved by using something like <a href="http://docs.openstack.org/developer/swift/">Swift</a> or Amazon S3 to distribute the content, provided someone is willing to deal with the infrastructure and the cost. There are even various start-ups offering this as a service, so it's not impossible this issue could be solved in the future, at least for a small project with proper funding.</p>
<h2 id="figuring-out-equivalency">Figuring Out Equivalency</h2>
<p>Let's assume for a moment there is a way to get access to all download logs across all content mirrors, and we are able to count who downloaded a specific package, correlating and correcting errors such as download accelerators or bias due to release policies. This bring me to my second point, which is a quite common error: believing that one hit in the log means something that can be converted to a number of users.</p>
<p>While this seems rather intuitive, in practice some people (or even some countries) have caching proxies, local mirrors, shared Internet connections or just dynamic IP addresses. In the absence of a unique identifier for each system, we cannot really estimate precisely the number of systems associated with each hit. And, since a lot of free software projects actually care about their users' privacy, they aren't likely add any sort of tracking technology within their software.</p>
<p>So, even if one download could be counted as one installed system, not every software installation maps cleanly to one user. For example, for a highly distributed system such as <a href="http://www.ceph.com/">Ceph</a>, one organization will likely have more than one computer within its Ceph cluster, and perhaps more than one Ceph cluster will be present as well.</p>
<p>With this in mind, we cannot guess the number of users based on the download logs only, nor even based on the IPs in the log since this can be a public IP for each server. Quite the contrary, several groups in an organization with a shared Internet access can be hidden. While not all projects suffer from such issues, enough of the ones I help do, so this a problem we can't ignore. We can reduce the impact by applying enough correlation and corrections, but then the numbers stop to be easy to understand, which defeats the original purpose of the operation.</p>
<h2 id="the-delivery-method">The Delivery Method</h2>
<p>If someone did have a way to identify a system for anonymous statistics-gathering purpose (a guarantee you can't really make, but I will skip the ethics discussion around such an approach), we are in a era of continuous integration, continuous delivery, and containers. This brings me to my third point. Like the others caveats I have mentioned, this will be highly dependent on your project and for some of the communities I work with, this will not be a problem. For example, I cannot imagine people endlessly downloading the virtual machine images from <a href="http://www.manageiq.org">ManageIQ</a> to re-upload them in their cloud for CI/CD purposes.</p>
<p>But it would be also likely to have someone somewhere using the project deliverables to build <a href="http://martinfowler.com/bliki/ImmutableServer.html">server immutable images</a> with <a href="https://www.packer.io/">packer</a> or <a href="http://patrick.uiterwijk.org/2014/01/21/rpm-ostree/">rpm-ostree</a>, or just building containers using <a href="https://www.docker.com/">Docker</a>, <a href="http://rocket.readthedocs.org/en/latest/">Rocket</a>, or <a href="https://www.openshift.org/">Openshift Origin</a>, and thus constantly installing the same <a href="http://www.gluster.org">Gluster</a> rpms or <a href="http://www.jboss.org/">Wildfly</a> archives in their images, thus making the download count skyrocket, and dominate the more "regular" usage of someone deploying upon installing a server. Given the reliance by part of the industry on public cloud infrastructure–where Internet addresses are reused and recycled–we wouldn't even be able to deduce any kind of clustering to group them by user to correct the data.</p>
<p>In some cases, it might even be others' community sub-projects who skew the numbers, especially for something like Java middleware, where the culture of code reuse is quite high. Pushing that idea further, a simple architectural change on someone else's continuous delivery pipeline could change from 10 download hits per hour to 50 or 100, because someone suddenly decided to start their build in a fresh cloud instance and therefore no longer caches the artifacts needed for any kind of build, thus creating the need to download artifacts each time.</p>
<p>If people's choices in terms of infrastructure can make a difference in the download numbers by one or two orders of magnitude, this means that you can't really compare numbers between projects, and you can't even know if an increase of downloads is a increase of users, or just a increase of builds by a downstream user.</p>
<p>In the end, we can try to compile numbers we cannot truly get for some projects; numbers that would in all cases require some filtering and domain-specific modifications to be meaningful and sometimes not even reflect anything regarding the realties of community and adoption. In the worst case, it's not even numbers we can compare from one year to another.</p>
<p>This is why we need to have events such as the <a href="http://flosscommunitymetrics.org/">Floss Community Metrics Meeting</a> and why we should stop trying to focus on the download numbers and start to look at what really matters. Discuss what it means to be successful in the community. Should we examine the people in the community and their diversity, or should we look at the events around the community and their frequency Or find out how people speak of the project, perhaps with more formal users' surveys as done on a regular basis by the OpenStack Foundation?</p>
<p>There are a lot of good ways to have insights into a free software project, but most of the time download numbers aren't one of them.</p>
<hr/><p>This article originally appeared on
<a href="http://community.redhat.com/">community.redhat.com</a>.
Follow the community on Twitter at
<a href="https://twitter.com/redhatopen">@redhatopen</a>, and find us on
<a href="https://www.facebook.com/redhatopen">Facebook</a> and
<a href="https://plus.google.com/u/0/b/113258037797946990391/113258037797946990391/posts">Google+</a>.</p>
When Metrics Go Wronghttp://community.redhat.com/blog/2014/07/when-metrics-go-wrong/2014-07-29T11:37:00-04:002014-10-22T09:44:36-04:00Dave Neary<p><img alt="" width="150" height="150" src="/images/blog/metrics-dneary-post.jpg?1478100396" />
Metrics are great. They can give you situational awareness about what's going on in your community, help you identify issues that you need to fix, and prove the effectiveness (or lack thereof) of community initiatives. But sometimes things go wrong.</p>
<p>Good metrics should lead to action, but sometimes, if you're not careful, you can end up with results you didn't intend. The very act of measuring something, and communicating that measurement, creates an incentive in the community. And sometimes the incentives you create do not match the behavior you want to encourage. (This is called <a href="http://freakonomics.com/2012/10/11/the-cobra-effect-a-new-freakonomics-radio-podcast/">The Cobra Effect</a>.)</p>
<p></p>
<h2 id="destructive-incentives">Destructive Incentives</h2>
<p>In April 1902, after the French had renovated Hanoi to install a modern sewer, the city was overrun with rats. The French Governor General of Indochina, Paul Doumer, instituted a "deratisation" scheme using local city workers, who were paid for each rat they captured. In the first week, they captured more than 8,000 rats. By the third week, more than 4,000 rats per day were being killed. By mid-July, up to 20,000 rats per day were being captured. But they barely made a dent in the problem.</p>
<p>In July, the government decided to widen the net and offer a reward to the citizens of Hanoi who offered proof that they had killed rats. To avoid a pile of rodent corpses building up outside the colonial government buildings, the proof requested for payment was just the rat's tail.</p>
<p>Immediately, two phenomena were observed: tailless rats started appearing, and a thriving rat farming industry emerged in the city. (Read the PDF <a href="http://www.freakonomics.com/media/vannrathunt.pdf">"Of Rats, Rice, and Race: The Great Hanoi Rat Massacre, an Episode in French Colonial History"</a> for more on this story.)</p>
<p>There is a possibly apocryphal story about an insurance company's call center manager who noticed that, although customer satisfaction rates were high, some customers had to wait for several minutes to get a human on the line. His idea was to put a display in the call center showing the wait time of the person who had been waiting the longest, dynamically updating all the time, so that everyone was aware that someone was waiting. He also evangelized his desire to bring wait times down so that no-one would wait longer than three minutes for an answer.</p>
<p>The wait times came down, but at a cost. Call center attendants would hang up on customers whose problem had not yet been resolved, to prevent the wait time going over the three-minute mark. They reasoned that the person would call back and their problem would be solved regardless. Of course, this frustrating experience led to customer satisfaction ratings dropping down through the floor, and the counter was swiftly removed.</p>
<p>Both of these stories show that measuring something, and implicitly or explicitly setting expectations for a trend in the measurement, can sometimes have unintended consequences. And typically there are three ways that metrics can cause these unintended results.</p>
<h2 id="measuring-the-wrong-thing">1. Measuring the Wrong Thing</h2>
<p>Like the French consulate who measured rat tails as a means of measuring the reduction in the rat population, we sometimes measure what is easy, rather than working to figure out whether that really correlates with the result we want to achieve.</p>
<p>In the late '90s, the key measurements that were used to measure batter skill were RBI (Runs Batted In) and batting average (how often a player got on base with a hit, divided by the number of at-bats). The Oakland A's were the first major league team to recruit based on a different measurement of player performance, on-base percentage. This measures how often they get to first base, regardless of how it happens.</p>
<p>It turns out that this correlates much better to what you want to produce in a baseball team, runs, rather than hitters. The league undervalued this measurement because a "walk" (where the player gets on base without hitting the ball) was commonly seen as a pitcher error rather than a batter success, but some players ended up having many more walks than others.</p>
<p>By looking at the results they wanted, and measuring the right thing, the Oakland A's got an edge over the competition, which allowed them to have a period of success relative to teams with much bigger budgets, winning the American League West three out of four years between 2000 and 2003.</p>
<h2 id="myopia">2. Myopia</h2>
<p>The other way metrics can go wrong is when the measurements concentrate on part of a system, but don't consider effects on the rest of the system. An actor optimizing for his measurements may actually be doing harm overall.</p>
<p>One example (without names, to protect the innocent) involved the marketing team of a company focusing on new sign-ups to their online service. They designed a marketing campaign in which new customers would get some free credit for the system and some goodies at a conference upon signing up.</p>
<p>The campaign was a great success, and soon the system had thousands of inactive accounts, with a balance sitting at $9.99. The CFO was unhappy because he had to carry this on the books, and the CEO was unhappy because the metrics he cared about, whether people stayed active on the service after 30 or 60 days, were terrible. Whether people wanted or needed the service did not enter into the calculations of the marketing project.</p>
<p>In one open source project (on which I was a release manager), the main metric I cared about was the bugs open against a milestone. As time went on, and the number was not going down fast enough, we regularly would bump bugs to the next milestone, not because they were not important issues, but because we knew that they would not be fixed by the date we had set ourselves. Having participated in a number of projects, I have a pretty good idea that this is a universal tendency as release approaches.</p>
<h2 id="drawing-the-wrong-conclusions">3. Drawing the Wrong Conclusions</h2>
<p>The third way things can go wrong is when you are measuring the right thing, but you draw the wrong conclusions from the data.</p>
<p>One set of metrics that is important is how many bugs are open against your project, and the rate at which they are being created. In one project I worked on, some people were concerned about a recent spike in the number of bug reports coming in because it seemed like the project must have serious quality issues to resolve; however, when we looked at the numbers, it turned out that many of the bugs were coming in because a large company had recently started using the project. The increase in bug reports was actually a proxy for a big influx of new users, which was a good thing.</p>
<p>Similarly, messages on mailing lists is something we generally want to see going up and to the right. More email means more members, right? Except when spikes are cause by community outrage at a recent controversial decision, or are the result of a massive and destructive flame war between two or three individuals. By looking only at the trends, you miss the richness of what is happening underneath, and run the risk of missing an opportunity to improve.</p>
<h2 id="final-advice">Final Advice</h2>
<p>So should we stop measuring things in our communities? Absolutely not.</p>
<p>We do, however, need to be discerning about what we measure, and how we communicate those measurements. Because metrics create incentives whether we like it or not, ensure that your measurements are creating virtuous incentives that encourage good behavior.</p>
<p>Also, constantly draw the metrics you use back to the overall goals of your project. Am I growing the user base, and helping those users to be happy? Am I helping the bottom line? By creating the link between primary and secondary metrics, you help mitigate myopia.</p>
<p>Finally, include both quantitative and qualitative analysis in your metrics reports, to ensure that trends are understood in their context.</p>
<p>Metrics are an awesome tool to help you identify your weaknesses, evaluate progress, and trumpet your successes. Just remember that you will get what you measure.</p>
<p>(<em>This post is derived from a presentation I gave at the inaugural <a href="http://flosscommunitymetrics.org/">FLOSS Community Metrics Meeting</a> on the topic "What you measure is what you get. Stories of metrics gone wrong." You can find the <a href="http://www.slideshare.net/nearyd/metrics-gone-bad">slides on SlideShare</a>.</em>)</p>
<hr/><p>This article originally appeared on
<a href="http://community.redhat.com/">community.redhat.com</a>.
Follow the community on Twitter at
<a href="https://twitter.com/redhatopen">@redhatopen</a>, and find us on
<a href="https://www.facebook.com/redhatopen">Facebook</a> and
<a href="https://plus.google.com/u/0/b/113258037797946990391/113258037797946990391/posts">Google+</a>.</p>