Should we bother measuring sentiment?

Nearly all social media monitoring tools offer sentiment analysis, but is it actually useful?

The short answer is yes, but only if you get smart with the data.

During our recent social media monitoring webinar, Matt Rhodes from Fresh Networks argued that in order to gain actionable insights from sentiment you need to look beyond the basic level. Citing the number of positive and negative mentions isn’t good enough. Instead, you should break the data down into topics and then keep on drilling.

For example, if you manage a restaurant chain you will want to know a range of different things such as; What are people saying about the menu? Did people enjoy their food? Was the service friendly? Were people served promptly?

Once you have drilled into this data, you can break it down further still. What do people think about the breakfast menu? What do people think about your spaghetti carbonara (or anything else you happen to serve)? What did people say about the service in one part of the country and how does this compare to others?

The problem is that the more you break down the data, the less likely it is that automated analysis will get it right. Automated sentiment works best with large amounts of data and can’t be relied upon for smaller samples. As Nathan Gilliatt pointed out, a lot of humans struggle with sarcasm and irony, so how can we expect computers to cope? This is particularly problematic when looking at Twitter.

There are also examples of posts which contain both positive and negative sentiment. For instance: “The food was fantastic but the service was terrible”. In this case a computer won’t know which way to turn.

“The plus side with automated analysis is that it is fast and close to real-time. It is very good at is spotting high level trends and extreme swings in sentiment such as swearing and complaints which makes it great for social customer services.”

Another benefit is that automation is less costly than having human analysts manually coding the sentiment, but the ideal solution is probably a mixture of both.

A decent monitoring tool will let you manually override sentiment. This means you can be instantly alerted to high-level trends and can quickly respond to particularly positive or negative mentions, but later human analysts can look over the details and check the accuracy when producing insights reports.

Undoubtedly automated sentiment analysis will continue to improve over time and Leon even suggested that soon we will be looking at intention analysis rather than sentiment. It will never be 100% accurate, but as long as you think carefully about how you use it you will see the benefits.

I’d be interested to hear other people’s views on this. Do you trust automated sentiment? Is it useful? And do you use human analysts to check it?

I think it depends, significantly, in what context the sentiment outcome is being applied to… it’s one thing to analyse the sentiment of a piece of text, but then depending on the context, that text may then be required to be categorised – something which in the case of stock market prices would be harder than deriving sentiment itself…

In the context of a restaurant, sure, most people are likely to use a hashtag or at tag (a hook) – the restaurant could chose to promote to their diners the tweeting of reviews, with their specific hashtag, even.

However, in the context of the stock market, anybody other than official market news Tweeters are unlikely to supply a consistent ‘hook’ – making the tweet-to-stock association particularly difficult.

Then – on another note entirely, there’s the question of whether providing stock market sentiment in the context of a trading platform should ever be taken with anything more than a pinch of salt – after all, it is in the interest of the trading platform provider for people to lose.

http://twitter.com/GenevieveWalsh3 Genevieve Walsh

Good natural language processing software uncovers more than just likes and dislikes. @behaviormatrix we’ve innovated basic sentiment analysis to the next level. We’ll kick off custom studies, on any topic(s), measuring more than 50 complex emotions in digital conversations found in blogs, social media, and news commentary. Basically, we can measure emotion in any digital conversation or anything that can be converted to digital text.

Nikolai Manek

We use sentiment analysis on blog and Tweet content at http://www.apphera.com and we are working hard to get it more accurate. Unfortunately, nothing really replaces a human monitoring it, therefore we added a feature to adjust the sentiment. The most difficult to correctly determine is sarcasm, the system is really lost on that. “The food was so delicious that I felt like throwing up” comes out as kind of a neutral sentiment but every person would most likely have a different opinion. We are working hard on those issues but you basically have to build an index for every vertical (like restaurants, hotels, car dealerships etc.) For example, moist cake is positive while a moist hotel bedding is not very pleasing. Good thing we are an open source project for sentiment analysis so the public can help to improve as improvement is really needed! Computers are a tremendous help for analysis, however, don’t really transcend 80% accuracy. I would go so far as to claim we are 2 years away from “perfect” results.