So I propose, that we should sit down and try to objectivise that fuzzy criteria and turn them in some numbers. In fact, this is what happens in the minds of SE team, it's only not yet written.

For example, the question states the problem with the possible lack of users with moderator privileges after changing from beta to graduate site. That means, that the requirements from Area51 are too loose. We need more people with close privileges (>3000) and at least a few ones with access to moderator tools (>10000).

Another factor is the gross size of the site, which makes the critical mass. We could be inspired by Generalist badge criteria, which require at least 40 tags with at least 200 questions in each in order to grand that badge. Maybe 40 tags with 100 questions is too much as graduate criteria, but how about at least 20 with 100 questions?

Anyway, I think there's enough SE sites to sum up the experiences and write out the common problems with beta sites and with sites that should become graduated. Writing clear criteria for graduation would not only make the whole process more transparent, but what's more important, they would become the guidelines for each beta community, saying what should we work on to help our children grow up.

Formalizing the procedure would also reduce the impression that among beta sites with very similar statisics, some of them are allowed to graduate anyway, some are allowed to live in beta for years, and others about high quality very specialized topics were never given a real chance and have been closed after only 6 months without warning or fair chance to improve. People who know me know what I am talking about ... So yes, the graduating procedure should urgently be formalized and made more objective and transparent for everybody.
–
DilatonJan 9 '14 at 20:20

1 Answer
1

Executive Summary

Creating a "Checklist" of items that guarantee graduation will cause certain members of communities to aim toward achieving the checklist rather than focusing on site quality.

The emphasis should be less on numbers, and more on what defines a healthy community, with numbers as one indicator of whether a community is healthy.

"I know it when I see it" is a very effective tool for handling something as fuzzy as whether a community has the chops to make it.

Gaming the System

How many people review only to the point that they get a badge? How many people will post quick answers and then edit their posts to get the Enlightened badge or a bit more rep? How many people make minor suggested edits for the sole purpose of getting the +2 rep? How many people edited a community wiki post or made a bounty during winterbash just to get a temporary hat?

When you create incentive to game the system, people will game the system. When that's a meaningless hat or a badge, it's not so bad, but when the energy of a new community is focused on achieving arbitrary numbers rather than building the community, the community actually suffers because of it.

This is my #1 complaint about the Area 51 procedure -- the arbitrary cutoffs for getting to the next stage, along with the time limitations, encourage people to game the system just to push the process along to get to the actual community building. This encourages appeals to amateurs rather than experts in the sample questions and 'marketing' for the push toward private beta

The Emphasis Should be on Community Health

No matter how hard you try, numbers will never be able to define who is a good community member, they can only be used as indicators to keep an eye on to see where the community stands so that those community leaders can actually act on improving them.

The users that make a community successful are the ones who focus on the things that actually make a community successful. They define the scope in meta. They help out new users. They curate tags. They make helpful edits. They encourage community behavior. They create relationships with the regular users and make them feel like a part of the community.

Numbers should focus on letting the community know which areas they may want to take a look at and tackle as a community. For instance if meta is like a ghost town, that should be a red flag for the community to act on. On the flipside though, a lot of meta activity doesn't necessarily mean that the meta activity is good. If there are constant discussions of site design, or resolving conflicts between members, or rehashing the same arguments, that doesn't have the same value as meta posts that are defining scope or promoting community moderation. So how do you quantify "Successful Meta"?

"That's where the truth lies, right down here in the gut. Do you know you have more nerve endings in your gut than you have in your head? You can look it up. I know some of you are going to say "I did look it up, and that's not true." That's 'cause you looked it up in a book. Next time, look it up in your gut. I did. My gut tells me that's how our nervous system works."

- Stephen Colbert

Communities should be graduated when they feel right. They could be growing exponentially, have tremendous participation, but have an ideological rift within the community pulling the same site in two different directions. A community could be in beta for 4 years without seeing the growth that the SE team may expect, but show through their participation that the community is healthy enough to step out on its own two feet. There is no magical formula for what succeeds.

If anything, communities should be given more of an ability to speak directly with the SE Community Managers to discuss the "State of the Beta" to give SE a better view of how healthy the community actually is. The Area 51 stats and the regular site self-evaluations only provide numbers, if anything the focus should be more on the community members who are participating daily and what they feel about the health and strength of the community.

This is pretty spot now. However, re: your last paragraph, we do watch the individual communities. We read your meta. We check up on your main site. We don't just look at the numbers - we observe how the entire community works together (if it does). "Show, don't tell" works really well here.
–
Anna Lear♦Jan 10 '14 at 6:46

8

@Anna, I know you guys lurk, but I would love to see more active participation by the community managers with the individual communities. From my experiences on The Workplace, there isn't much feedback about where we stand, what we should work on, or what we're doing well from you guys, and I think that a bit of interaction would go a long way toward making people feel good about what we're building. That's more of a separate point, but I think it would be well-received.
–
jmac♦Jan 10 '14 at 6:50

1

Fine. But then, for [insert deity]'s sake, please get rid of the ridiculous target stats and 90-day timeframe on Area 51.
–
RaphaelOct 9 '14 at 16:06

"Communities should be graduated when they feel right." too subjective. feels right to who? the passive phrasing masks that its almost entirely an se decision behind closed doors without openness/ transparency on the thinking =(
–
vznOct 9 '14 at 18:38

@Raphael, for reference, this was written well before I became an employee. Could you clarify which 90-day timeframe you're talking about? Thanks in advance!
–
jmac♦Oct 9 '14 at 18:41

@vzn, feels right to the people making the decision (SE staff with input from the community in the forms of self-evaluations, etc.). Introspections are hard. Let's say we have a review for graduation, and our concern is that while the quality is awesome, and we are the best resource on the net for that topic, the question volume is incredibly low and shows no signs of growth. What would being transparent about that solve other than making you feel bad about something you can't change? It's the same concept as not notifying about a downvote or having your question closed.
–
jmac♦Oct 9 '14 at 18:48

presumably most sites would be open to constructive feedback from mgt about why a site isnt graduating. you use the word "solve", am not personally saying/ viewing nongraduation entirely as a problem, more of a challenge. something more informative other than "keep on doing what you're doing" (eg the case with shog9/ Computer Science) or "our queue is long & you still arent at the top of it"
–
vznOct 9 '14 at 18:54

@vzn, if the issue is that there aren't enough people with privilege levels for a graduated site, that's something fixed easily and non-destructively (vote early, vote often!). But if the issue is that the questions are saturated already, and the site isn't likely to grow to be big enough moving forward, what could you do about that information? Even if a site isn't graduating, having motivated users is still helping make the web better. Do you really think saying, "Sorry, your site will probably never make it for reasons X, Y, and Z, good luck!" will help?
–
jmac♦Oct 9 '14 at 18:58

@jmac Noted. This timeframe: "This site is in Public Beta. Anyone can participate! Sites remain in beta for at least 90 days to build up a critical mass of users, questions, and participation." It's right there on any Area 51 page of a proposal that is in Beta.
–
RaphaelOct 9 '14 at 19:23

honesty is the best policy! maybe not harsh, better diplomatic... if se mgt thinks that some beta sites will never achieve full fledged graduation, have never heard about that case. what is se's opinion on that? just leave em be & push the decision indefinitely into the future? doesnt the clock eventually run out with moderate/ decent performance? apparently not!
–
vznOct 9 '14 at 20:03

@vzn, apologies, I don't think I communicated well. A site could be (a) the best site in that space on the internet (b) full of quality posts and an active energetic community, but (c) occupying a space that is small and isn't likely to grow. Let's say we had a 'fingernail clipping' SE -- it could be awesome, and could have all the info, but there likely won't be much growth in that topic in the future (there is only to much to ask about nail clipping). The problem isn't the site (the site is awesome!) it's the lack of growth in the topic. The community can do nothing about that.
–
jmac♦Oct 10 '14 at 2:18

So in that case, what do we do? Do we tell the community, "It's not you, it's your topic" which they can do nothing about and may discourage some currently energetic and excited people? Do we just hope that some miracle in nail clipping is released and the nail clipping site gains traction in the world as a whole as people clamor for information on this new type of nail clipping? Telling people something negative about something they have no control about when they are doing everything right doesn't seem fair to anyone involved. Will that make anyone happy and/or help site quality?
–
jmac♦Oct 10 '14 at 2:21

and do you/se mgt think what you just described is actually the case with any existing se or that merely a highly hypothetical/ contrived scenario? looking at it from other angle, if the se mgt "heart is not in it" about the potential/ merit of a site is it really fully fair/ ethical not to disclose those doubts in any way whatsoever to community participants?
–
vznOct 10 '14 at 14:54

@vzn, the situation is not hypothetical or contrived. We have many smaller technologies represented in our network. And those sites are awesome. But technologies come and go. If one of those technologies becomes virtually obsolete (think having a lycos.SE, for instance), even if the community, quality of Q&A, and the activity level of the site were great, could you not imagine that this would have an impact on graduation status? If so, why do you think this is a 'doubt' in the community on the SE side? Sometimes stuff happens. Nobody is to blame. It still doesn't feel nice though.
–
jmac♦Oct 10 '14 at 16:43

ok see a possible gap here (trying to "find way to agree"). the area51 stats merely track "good enough". you here insist on growth. so a beta site that has been good enough for many months may still not achieve a continual growth target. so revealing what this continual growth target is that se is interested in, how it is tracked, making it objective, would be helpful to the communities interested in the collective goal of graduating.
–
vznOct 10 '14 at 16:49