« The new threat: I’ll put/read/do it on the Internet
AOC3, speaking engagements, and other great stuff »

Become intimate with the conversation to overcome the flaws of sentiment analysis in social media monitoring

6 April 2010, 3:43 pm

Ever have something come out of your mouth one way and become surprised to learn that the person you’re talking to heard something completely different? You can’t change what the person believes they heard, even if it’s not what you said or meant. And though the intention of your own words seemed clear, the other person’s perception is their truth.

Now imagine writing a thought down and sending it to someone. Without the benefit of body language, it’s hard to know the tone of that message, whether it’s meant to be serious or sarcastic, positive or negative. And if you don’t know the person well, the task is even harder. Just think of all the emails you write that fall into this category.

Writing is tricky, full of nuance and unspoken communication. And in the sentiment analysis game – the practice of determining whether an online conversation is either positive, negative or neutral (or mixed) in tone – that’s a sticky issue. For social media monitoring service providers, it’s a hurdle they’re trying really hard to overcome.

Here’s where things stand right now with sentiment analysis:

• Human filtering: Companies like RepuTrack use people on their team to help determine sentiment.

• Automated: Companies like Alterian’s SM2 use a complex algorithm that looks for “trigger words” to help classify conversations.

• Analyst driven: Individuals, either on the client side or a consultant being paid by the client, positioned between the monitoring platform and the end-user to verify and/or reclassify the information.

However, according to Maria Ogneva of Biz360, you can only get so close to a perfect result. “If two humans were rating an article for sentiment, they would agree 79% of the time (source of this statistic escapes me now), so automated sentiment simply can not be higher than that.”

So if that’s the case, how do you make the analysis as close to perfect as possible?

Over lunch recently with Alexandre Gravel of Toast Studio and Martin Perron of Bloom Search Marketing, we reasoned that crowdsourcing could hold the key. Over on Biz360’s blog, MarketIQ, they describe an experiment they conducted using Mechanical Turk to crowdsource sentiment analysis to verify if their own classifications were accurate. (It’s a fascinating read, take some time to digest it.)

But the question remains, how do you overcome the flaws of sentiment analysis if you’re responsible for tracking conversation related to a brand?

Become intimate with the conversation.

You read that right.

Before you even set up a single keyword search, make sure you have a thorough understanding of the brand and its environment. By getting to know the brand beforehand, you’ll understand the tone associated to it, the issues it faces and you’ll be able to outline the things you expect to hear.

Then, once you start collecting results, you’ll see and understand the trends, key influencers, lingo – and the discussions taking place will just make sense to you. Like any relationship, you’ll know what the hidden body language is within these conversations. Monitor on a daily basis and you’ll find the flow and be able to judge for yourself how a conversation should be classified.

Of course, this isn’t the perfect solution for all projects and search profiles, but no matter how you approach your social media monitoring program, if you’re not reading the results, you’re wasting your time anyway. So become intimate with the conversation and you’ll be in a better position to classify sentiment for yourself.

Tags: , , , , , , , , , , , , , , , , ,

3 Responses
  • Maria Ogneva

    Great discussion, and thanks for the Biz360 shoutout and the kind words about our blogpost on automated sentiment.

    How to deal with limitations of automated sentiment? Well, it’s never going to be perfect, and shouldn’t be used by itself without considering other metrics. Also, it’s most useful when used looking at several thousand mentions - this way you can at least get an idea of what’s going on. Otherwise, if you just have 1-10 articles in front of you, it’s probably better to just read it.

    Your recommendation is spot on about getting familiar with 1. the community, 2. the influencers, 3. the industry, 4. the jargon and type of conversation.

    We are doing something fun with our sentiment data - we are looking at mentions and sentiment around each contestant on American Idol, in order to predict who is going home each week. Check out http://idolstats.com and let us know what you think.

    Maria Ogneva
    @themaria @biz360

  • Martin Perron

    Mark, great continuation of our lunch discussion a few weeks back! Sentiment analysis, at least to me, seems to be such a complex problem with lots of nuances and as Maria points out, the larger the data set, the more reliable it can be.

    I guess the same challenges apply with automatic translation. Google Translate does a pretty bad job at capturing the nuances between french and english translations for example. But as with automated sentiment analysis, over time, the reliability should improve if the human input continues to validate the results.

    @maria - cool project for idolstats!

    Martin

  • Mark Goren | Transmission Content + Creative

    Maria, you’re 100% right in that it’s all about scale and finding the right balance. That said, if you are monitoring a profile on a daily basis, you do get a feel for the conversation, regardless of how many results come in. Sure, 50,000 at once is daunting and likely impossible to go through at once, but how many profiles will pick up that kind of conversation in a single 24-hour period? (And if one does, look out!)

    Martin, really like the comparison between sentiment analysis and Google Translate, another service that requires human verification to get things right.

    Thanks!

  • Leave a Reply




     
    Is social media a fad? Check out this great video by Socialnomics, I think you'll see that the question is well answered.