I'm attaching the notes from the metrics discussion we had on Saturday. Thanks a lot for the people that attended. We had some great discussion.

Kudos to Jackson from New Relic that took the notes that you can see. I've edited a bit them to give some format. I've also removed any reference to companies (not open source projects) as this may be sensitive information. If you feel this is valuable information we can add this later.

Comments and extra discussion is really welcome :).

Cheers,Daniel.

Topic about Metrics (Initial discussion and needs and roles introduction)

Measuring community and what does measuring community really mean?

Share some experience - two or three roles around the topic of metrics - what is the purpose of measurement? “I am good at my job, and this is how I’m good at doing my job”

Specific needs like - people are coming, and w need to say hello and “thanks for coming.” “it is great that you provided X”

after a while you might have people who are leaving the community and why they are leaving. Getting them interacting after they have left

Daniel is focused on open source communities - so please share other things you’d like to share

Engineering roles - metrics could be useful so how they are performing - how fast are we –are we talking to specific volunteers, and and are we saying hell and providing them with numbers about time for review about them vs community - which came from the wikimedia foundation that really cares about volunteers - and they are continually doing hackathons, wikimedia meetings, and academia events. They needed metrics for volunteers and.

Metrics for human resources - need for hiring people, but no idea where to look for them, and where is the best place(s) - aggregate the best places to hire, such as stack overflow, glassdoor, etc

Why are you measuring things? (Discussion about the reasons why the attendees were measuring things)

Ben Lloy - interested in seeing how they perform to the rest of the industry. How do they contribute to the rest of his company, and how they compare to the rest of that company, along with the impact of their efforts on open source and that company.

People mostly focused on quantitative metrics, not qualitative

Some discussions and ideas:

Understanding how many companies there are around

Business vs community, and to make sure everything is fair (eg code review process).

Measuring things that don’t matter in order to make a case for things that do.

working with the community and measuring things that don’t

Examples of metrics “how may downloads, replies to blog posts, etc”

Measure in numbers their community impact -

Manrique - Many metrics are based on the amount of people in a community - joining, leaving the community, and and how to see the flow as they flow through the community. How to make the path though a contribution - and how to make the path as short as possible.

Sometimes you are measuring too much, and how to remove the metrics that don’t matter, as well as how to come down to the low-hanging fruit

Some discussion

Davey: community is mostly php, and some do not know about the offerings that we have. Most people that I speak to are long-term '[investments]'

Amanda - we aren’t tied to revenue either, and the second that you get associated with indirect revenue is the minute that you kill organic growth

Ben - we regularly publish technical articles on the items were are working on. We recently had a post about our key technologies we posted a change that is a controversial change - other websites picked it up and that was valuable - and that’s valuable about engagement.

Davey - Do you track negative vs positive connotations on engagement?

Clay - Tying metrics to revenue, especially in B2B companies - I get the impression that measurement will increasing in pressure to track revenue based on engagement

Davey - API calls are maybe a basic metric to evaluate how people use the product

Alexandra - You can use a user funnel and see how people flow through your website and downloads or trials and tracking how people

Jason Yee - Metric about No one is going to click a tweet and decide to blow $1500.

Going from a developer and decision maker.

Doing A/B testing could be based on those who are engagement with the community - are they spending more money that people not engaged with the community?

What sort of metrics? Sometimes there aren’t = correlations can be measured, but correlations cannot be tied to causation.

Rikky

we do a lot of metrics concerning the site, contribution,s page views. documentation about the event, ROI - who we talked to, takeaways from sessions, rather than what you may be able to capture with metrics

Stephen Walli - Conflation between community and product revenue. - trying to tie the cost of community marketing, and tying it to the business value.

Can community mangers articulate the difference between items in the open source arena or

Value must be only monetary -

Churn retention and acquisition - for community, not customers.

Upper management - “one metric that matters’

be able to track lower things so you can get more information.

Reporting on one item, which aligns most with the business goal.

Ale Murray - Apache cassandra

We have a very established community after 2.5 years of working on that.

the more people that use the open source will eventually fall into the pipeline of the revenue streams

growth metrics - how many people attended an event, downloads, etc

once the community has hit scale and is self-sustaining - now trying to measure the success of those who are using the product

ex: how many users stories - how many forks, etc

Jyee - metric around use - looking at top skills on linked, top 10 database skills

“how many interactions happen on stack overflow”

forking of forks become a ripple effect of engagement - which becomes harder to measure

what is the impact of may company on this particular project?

If someone is investing in cassandra that is outside of cassandra - it means that people want to add value to the project

Are people using qualitative vs quantitative data to evaluate their efforts?

foundation efforts - see how contributes to the foundation, and see who asks questions about the project on the mailing list, for example - Company X asks questions, and then joins the foundation 3 months later.

Why do we have this peak of questions - why do I have this peak of of pull requests -

Then you can dig in about KPIs and how metrics are evaluated

For some projects, we need to have 6 months before ”the project is dead.” in some projects, they may be 1 year vs 6 months.

Problems - using metrics to find people to hire

going to the community itself is a non-starter

Analyzing the community around the item to evaluate the environment and how people are using metrics at a given company - using pull requests that are good to be able to see who is accepted in the community

Having real numbers is different from what you might be able to find form an interview

Having a look at the community and how they use for instance the API allow us to improve the documentation.

Some challenges

Challenge - how to get metrics about how people improve their skills around community

how the community is improving he collaborative support if they don’t work around repositories

value of the community, which is hard to track because they don’t have common infrastructure.

it is difficult to connect to some activities with the training of the community, with the usage of the community, and how they support each other in a collaborative way.

Challenge - measuring impact on a very extended timeframe. How do you measure impact of in person interactions and an eventual move to buy?