Bizarre Trend: Journalism Professors Using Klout Scores As Part Of Students' Grades

from the bad-metrics dept

We've raised questions in the past about the relevance of "Klout" scores. If you don't know, Klout is one of a few companies that try to measure "influence" online by looking at your social media activity. The whole process seems kind of silly, but for whatever reason, once you put a number on things, people take it seriously, no matter how bogus the number might be. Lots of companies now use Klout scores to determine who they should give special perks to, leading to plenty of people just trying to game their scores. However, should Klout scores count towards your grade as a student? Adam Singer sent over examples of two separate journalism professors who think so.

In both cases, the professors admit that Klout scores may not be particularly accurate, but both have interesting defenses for including the score as part of the grades that students get. Todd Bacile, from Florida State University, argues that even if people think Klout is inaccurate or silly, since others are taking it seriously, students need to take it seriously too. Even if it's a problem-perpetuating situation, he argues that he's being something of a realist and merely preparing his students for the real world, where they'll be judged by their Klout scores whether they like it or not -- and thus one project (but not the entire grade for the course) will be graded based on Klout scores:

Most people seem to either love or hate Klout, so the notion of assigning a portion of a student’s grade to their respective Klout score may cause some to react … what’s a good word to use here … fretfully. Yet, as an educator teaching electronic marketing at the collegiate level I owe it to my students to introduce them to every and any concept that will help them land an internship or fulltime job.
Klout matters to employers

And here is an inescapable fact. Many firms are sizing up college student’s Klout scores as a quantitative metric to use for job applicant screening. Therefore, I decided to create a class project in which the final grade earned is solely determined by a student’s Klout score.

For whatever it's worth, he notes that his students have massively increased their Klout scores because of this. He also claims that by having them focus on Klout scores -- something that could constantly be measured -- it actually kept students more engaged:

An experiential project like this proved to be enjoyable for the students and maintained their attention and enthusiasm throughout the semester. Many students would compare scores and discuss different techniques used to engage with powerful opinion leaders within the social world. Which students had higher scores became a friendly competition causing students to work even harder at engaging others. Imagine that: students wanting to work more to develop skills that current marketing employers are searching for!

The second story Singer sent over is from Ryan Thornburg, who teaches journalism at UNC, and who is also using Klout as part of the grade (in this case, counting as 20% of it). Thornburg also admits, quite directly, that Klout is probably "overly simplistic." So why is he doing this? Because he thinks that it forces students to do actual experiments in which they can all learn how Klout works, and see if they can improve it.

How is that possibly fair to students who are struggling to raise this arbitrary number that's contrived inside a black box? It's fair because it transforms the class from a workshop on button-pushing to an exercise in hypothesis testing, strategy and critical thinking. Students -- who often approach grades with calculating economy of effort -- don't know what they have to do to boost their Klout scores, so they are forced to design simple experiments, isolate variables, and generalize their findings.

In the end, he notes that perhaps his best students will actually be able to build a better Klout.

I'll admit that my initial reaction to these stories was horror. The idea of basing grades on a silly system like Klout certainly feels very, very wrong. However, the explanations and defenses from both professors have me rethinking that stance somewhat. Is it really all that different from "teaching to the test", as some teachers do for standardized testing? An SAT score may not really tell us much of anything, but it is important for many colleges, so is it a surprise that teachers help their students optimize for it? While we can quite reasonably worry that focusing on Klout has students optimizing less useful skills, from an experimental standpoint, perhaps it's not such a crazy idea.

Just because it's like "teaching to the test" doesn't mean it's a good idea. As a college journalism professor, this scares me. Even if Klout were a perfect tool, what they're telling their students is its more important to use social media to promote themselves than it is to use it to find sources, story ideas and better understand the issues people are discussing.

All of my classes have a heavy dose of social media in them these days, but Klout is among the five or six big Internet trends I dismiss outright early on. I'd much rather have them increase their clout by learning how to write and report stories that people actually care about.

Re:

"what they're telling their students is its more important to use social media to promote themselves than it is to use it to find sources, story ideas and better understand the issues people are discussing."

That can be applied to wider society as well which is scary. It is no longer sufficient to be intelligent and hard working in order to succeed in life. Nowadays, it is all about marketing themselves in the right way which, let's be honest, not every one is good at. This is why you see so many arse kissers promoted to high level jobs while those who would actually be good at the job get left behind.

Re:

I appreciate your insight, but why do you believe that the two are mutually exclusive? Your last sentence appears to indicate that you can either focus on Klout or learn to write, which from my perspective is nonsensical.

What I read from this article and these professors, is that they are trying to teach students both how to write/etc. and how to understand and influence measures of their impact online.

Re: Re:

Your last sentence appears to indicate that you can either focus on Klout or learn to write, which from my perspective is nonsensical.

I think his point is that any time spent working on one's Klout score is time not spent on other things, such as learning how to research or write. Obviously it's possible to do both, but not an unlimited amount of both, so it's a tradeoff.

Re: Re: Re:

Well, I get that, but then I would expect some analysis of how time spent working on the Klout score detrimentally cannibalizes writing time and therefore skill.

It is my opinion that conclusion does not logically follow even if time is scarce. People are capable of promoting themselves and honing their craft at the same time and I would presume that the more you learn and experiment with promotion the better you get and the less time it may take.

In fact, I think the fundamental premise behind what Mike tries to teach is that most professions require that individual promotion and outreach to become a core part of becoming successful.

Re:

While being able to write and report stories that people care about is important, it's completely irrelevant if no one reads it.

At least this system is impartial, repeatable, and known. Typically when being graded on how well you write, you get a heavy heavy dose of bias based on the teacher. You're not writing for the "public", you're writing for your teacher. If you do something unconventional or something they don't agree with, you'll do badly despite it perhaps being an idea the public (your future audience) would love.

I think the idea has merit. Perhaps in the future many classes that teach so to speak public service positions will be judged and graded by that very same public. The internet is a wonderful thing... embrace it

Re: Re:

The entire point of the fucking internet is that outside Onionland, if something is interesting, it WILL find its way into the focus of the masses -- you are essentially arguing that something which does not NEED to be optimised should be prioritised over something which can never be optimised enough.

Re:

At least with Klout, none of these professors are deluding themselves. They recognize that Klout is more or less arbitrary, and they go into it with that recognition. With standardized tests, everyone is convinced that this is the best (and perhaps only) tool to predict performance.

The problem with this, and the general trend to use measurement of human abilities to make decisions, is that it maximizes whatever behavior increases the measured value. Unless their is a strong correlation between the desired behavior/ability and the measured score, this approach is ultimately damaging. This is the same problem that led to the financial crisis, using short term profit to measure long term performance, and rewarding people on the short term profit.

Not Quite Like the SAT

Is emphasizing Klout scores like a standardized test? No. For one thing, you don't have to cough up a bunch of access to your personal social media accounts to write the SAT.

Klout's secret formula for determining influence is only part of the problem with using it for decisions like employment and grades. The much bigger problem is that it requires giving up lots of access to personal data to a private entity that will do who knows what with it. Want a higher Klout score? Give access to Facebook and other social media channels. This will be problematic, particularly down the road as people get jobs or grades that they want, then forget about their Klout accounts, which still have access to personal and otherwise private information.

Enough, Already!

When are we gonna get rid of this Liberal "touchy-feely" crap and get back to EDUCATION, ya know, readin' ritin' and 'rithmetic!!! The rest of the world is passing us by, and laughing their collective asses off at us.

Re: sounds like the 'Ebonics' of the Internet

I agree. The first point is the general "dumbing down" of the curricula to allow more students to "pass" and therefore boost the schools Federal scores and their funding. The second is that more and more courses are becoming available on the internet, usually free, that can be granted credit. Eventually, the formal schools WON'T be necessary. Couldn't happen to a nicer bunch!

The "economist" concludes: "bad-metrics" are better than none.

You say first: "from the bad-metrics dept".

But you don't grasp that's a mis-applied buzz-word; here's the entire definition from my plain old text dictionary of the early 90's:
metrics: n.pl. [with sing. v.] the science or art of writing in meter

... and now the cynical viewpoint...

So these kids grew up playing mmorpgs, where they leave their dude swimming in the ocean to up his stamina while studying. Gaming the system. What are they studying? Whatever the standardized test requires of them, because the professor is ... gaming the system. And then they start or work for a company constantly trying to rise above the rest: upping their SEO; creative marketing; new approaches, etc. ... essentially gaming the system. Meanwhile these kids try to present a public social media persona that will help them later in business and life ... i.e., gaming the system.

So, sure, gaming Klout scores can be a great way to learn valuable real life skills of ... gaming the system.

(Or, less cynically, they are learning to work within the construct that has been given them, and finding creative ways to gain some advantage over all the other folks who are trying to do the same thing. Isn't that pretty much business and life?)

Re: ... and now the cynical viewpoint...

(Or, less cynically, they are learning to work within the construct that has been given them, and finding creative ways to gain some advantage over all the other folks who are trying to do the same thing. Isn't that pretty much business and life?)

If life is zero-sum, yes. If not, then there's much more to it than gaining advantage over those around you.

The problem here

Is that the courses are NOT measuring the student's knowledge, retention, abilities, etc. They are measuring a 3rd party over which the student has only minimal impact.

Those professors might as well grade a student on how many dogs walk past their house. Certainly, a student could game the system by leaving dog food outside, but what is really being measured here?

Also, TL:DR but are any companies REALLY using Klout scores as part of their hiring decisions? As a hiring manager I have taken a number of training courses and it seems to me that Klout would fall in the *really don't do this* section of the manual.

Metrics are better than being judged subjectively by a human. How many people here got a bad grade because their professor didn't like their subject matter? How many people here watched someone get a promotion even though they are a terrible employee just because that person was a kiss ass? Metrics give people something objective to work towards and even though they can be gamed, at least when that happens the metrics can be updated to prevent that. How do you prevent someone from being biased when giving out grades/promotions without metrics?

I think these J-school professors need to be turfed out to the public relations or advertising departments. Journalists should be reporting what is rather than trying to influence trends. Gaming Klout scores is perfect training for someone trying to become influential and sell products or ideas, but really doesn't have much impact on journalism.

Re: Grade Buying

How honest can we depend on Klout to be? If school grades are to be based, in part, on Klout scores, Klout has to be absolutely honest, with no possibility of corruption, and with perfect judgement. "Not possible", says Auto. It's run by humans, subject to greed and a desire for power and sometimes revenge. I suppose if it were run by a good AI (even Auto failed, you know), maybe, but not under ordinary people. Bad idea.

Whether it is a good idea or not depends on how the requirement is formulated, and how large a portion of the grade it is. (My opinion as college professor.)

If it is formulated as the professor using Klout as an arbiter of the quality of student work, that is bad. He should be the arbiter. His opinion is more direct, more responsive to your work, conveys more information, is more interesting, teaches you more.

If it is formulated as a specific "Klout project", an assignment where you maximize your Klout score and get a grade for doing this, then it looks like a fun competition to me.

Implicit in the "Klout project" is that the fraction of the grade it represents will not be too big. Say 10% at most. So you get 10% of your grade for your ability to manipulate the net in a certain way. That is fine as a kind of "practical assignment".

Also implicit in the "Klout project" is that the percentage is subject to critique. If it is too much, then the students would quite naturally ask, why are we spending so much time on this aspect of things.

It reminds me of Qualitative Analysis (a chemistry lab course in high school) where all that counted was identifying the chemical. You write the chemical name on a card after several weeks of work. A cool challenge, but please not too important percentagewise because there is too much luck involved.

I suppose you could even have a class purely in "manipulating the net" where the whole class is one big project to get hits or raise a google rank or hack something. But that's a pretty *specialized* class and it's not journalism per se.

Or to put it succinctly: you can't use Klout as a metric, because it does not have integrity. You can post it as an external goal, a contest -- subject to critique. The difference in conceptualization is important to the integrity of the class.

I think it's a good idea. I also think some comments are missing two key factors:
a) the subject is journalism.
b) the Klout score is only a fraction of the grade.

Look at modern journalism - how many journalists function well in their role without a plethora of social media? Not just for socializing, but for putting out some of their work (albeit in link form), garnering interest, communicating with regular readers and garnering new ones - all well recognized uses for social media in business here on Techdirt! The Klout score projects sound like they are designed to focus students on using social media in this manner, instead of solely with a social focus.

Secondly, a key aspect of journalism is not only getting your work to the public, but gathering resources. Human resources are mainly collected through networking. Developing social media skills, and online networking, is a valuable skill for journalism students, surely? I would likely feel different if the course was calculus.

Finally, as someone else pointed out, this develops their writing skills - not the traditional skills they are already learning in class, and have spent several years through out primary and secondary school developing, but a specific skill-set used online. It is quite a different task to write a compelling argument in 2000 words as opposed to writing an interesting or informative statement in under 160 characters.

These are all skills that they will need in their profession. Academia is often faulted for being too esoteric - at least these two professors are focusing on real-world skills that will assist these students in the workforce. And there is no indication they are doing it at the expense of traditional core subjects for the unit.

Preposterous

The business model is less about your score and more about delivering you to companies to make money. That's fine - I'm a capitalist pig small business guy - but purporting to measure influence when there is little independent research establishing that one can determine influence from social media activity... Doesn't belong in the curriculum.

The Klout Controversy...

Great article, Mike! There are also several terrific comments regarding the use of this metric in a higher education course. For the record, I teach an Electronic Marketing course in a business college. The course has a heavy emphasis on how brands use social media to create engaging content. I'm not sure how I was cast as a journalism professor! Just a few brief comments I would like to make, then I will head on out of here. I have no interest in a sparring match with those who dislike the metric or dislike me. It is controversial and I respect each person's opinion.

The grade for this assignment is a mere 10%, which is similar to a standard participation grade or attendance grade in many courses. That being the case, I am surprised how much uproar it has created! I see too many benefits to use this in my class to immediately discount it due to the argument that it is an imperfect metric. First, I have spoken directly with hiring managers who use Klout to screen applicants seeking social media jobs. These are the types of jobs my marketing students are hoping to land upon graduation. Do all companies use this? No. Do some? Yes. Second, I am teaching students how to create content other people want to engage with (comment on, retweet, 'Like", +1, re-share, etc...), similar to successful brands doing the same. I see a strong correlation with Klout and these types of activities. Can people cheat? Sure, but anything online - and in life - can be cheated or manipulated. For example, some people estimate as many as 30% of online product reviews are fake; yet, online reviews are still useful and serve a purpose. Third, as for the argument of 'it is not fair to base a grade on an unknown algorithm', such a stance would cause thousands of schools to immediately cease the use of marketing and managerial simulation software. These software programs provide a good exercise for students, yet the grading is based on an unknown algorithm analyzing several criteria. The point is we live in a world of uncertainty with useful, yet somewhat questionable metrics.

I have to run. Feel free to contact me via Twitter if you want to discuss this more. Take care everyone! @toddbacile

The clout of Klout

Nothing wrong with teaching students about Klout, but what's important is that they are aware if its weaknesses, in addition to its strengths.

Klout is not the most accurate metric. Nonetheless, it doesn't stop companies from trying to grasp at any type of metric that's available. For example, genesys has started prioritizing customers according to their Klout score. I write about this here: http://www.blurban.org/2012/11/08/whats-your-number/