Become intimate with the conversation to overcome the flaws of sentiment analysis in social media monitoring6 April 2010, 3:43 pm
Ever have something come out of your mouth one way and become surprised to learn that the person you’re talking to heard something completely different? You can’t change what the person believes they heard, even if it’s not what you said or meant. And though the intention of your own words seemed clear, the other person’s perception is their truth.
Now imagine writing a thought down and sending it to someone. Without the benefit of body language, it’s hard to know the tone of that message, whether it’s meant to be serious or sarcastic, positive or negative. And if you don’t know the person well, the task is even harder. Just think of all the emails you write that fall into this category.
Writing is tricky, full of nuance and unspoken communication. And in the sentiment analysis game – the practice of determining whether an online conversation is either positive, negative or neutral (or mixed) in tone – that’s a sticky issue. For social media monitoring service providers, it’s a hurdle they’re trying really hard to overcome.
Here’s where things stand right now with sentiment analysis:
• Human filtering: Companies like RepuTrack use people on their team to help determine sentiment.
• Automated: Companies like Alterian’s SM2 use a complex algorithm that looks for “trigger words” to help classify conversations.
• Analyst driven: Individuals, either on the client side or a consultant being paid by the client, positioned between the monitoring platform and the end-user to verify and/or reclassify the information.
However, according to Maria Ogneva of Biz360, you can only get so close to a perfect result. “If two humans were rating an article for sentiment, they would agree 79% of the time (source of this statistic escapes me now), so automated sentiment simply can not be higher than that.”
So if that’s the case, how do you make the analysis as close to perfect as possible?
Over lunch recently with Alexandre Gravel of Toast Studio and Martin Perron of Bloom Search Marketing, we reasoned that crowdsourcing could hold the key. Over on Biz360’s blog, MarketIQ, they describe an experiment they conducted using Mechanical Turk to crowdsource sentiment analysis to verify if their own classifications were accurate. (It’s a fascinating read, take some time to digest it.)
But the question remains, how do you overcome the flaws of sentiment analysis if you’re responsible for tracking conversation related to a brand?
Become intimate with the conversation.
You read that right.
Before you even set up a single keyword search, make sure you have a thorough understanding of the brand and its environment. By getting to know the brand beforehand, you’ll understand the tone associated to it, the issues it faces and you’ll be able to outline the things you expect to hear.
Then, once you start collecting results, you’ll see and understand the trends, key influencers, lingo – and the discussions taking place will just make sense to you. Like any relationship, you’ll know what the hidden body language is within these conversations. Monitor on a daily basis and you’ll find the flow and be able to judge for yourself how a conversation should be classified.
Of course, this isn’t the perfect solution for all projects and search profiles, but no matter how you approach your social media monitoring program, if you’re not reading the results, you’re wasting your time anyway. So become intimate with the conversation and you’ll be in a better position to classify sentiment for yourself.
Tags: Alexandre Gravel, Alterian SM2, Biz360, Bloom Search Marketing, Jim Reynolds, Joseph Fiore, Maria Ogneva, MarketIQ Blog, Martin Perrson, Mechanical Turk, RepuTrack, Sentiment Analysis, SM2, Social Media Monitoring, SocialMediaMonitoring.ca, Techrigy SM2, Toast Studio, Transmission Content + Creative