Blog: How can AI help you listen to the open web?

How can AI help you listen to the open web?

Some interesting (and recent) Machine Learning based studies on Social Media and other

Internet websites Internet -a powerful medium to check the world’s pulse

The Internet or open web has become an essential tool for expressing ideas, marketing and connecting with people. It’s a market, town hall and postage department packed in the language of 0’s and 1’s that computers understand. But there is another (rather unappreciated) use social media and internet has, and that is getting the world’s pulse. This tweet by Jack Dorsey after the 2017 Joint-session speech is an excellent example of using Twitter as a listening tool.

With this blog post, we summarize some studies on social media and other internet websites that demonstrate how the internet or open web can be a powerful tool when looking for meaningful insights about the real world. There is a vast ocean of data on the open web and we need the recent advances in AI to summarize the signal in a form humans can consume. Please note that a lot of the studies are pre-prints and might not be accepted in the current form, but the idea is to talk about the possibilities rather than doing a review.


You can predict referendums using Twitter Sentiment. The balance of twitter sentiment (the difference between positive/negative sentiment) could have predicted whether UK was to leave the EU. All other predictors (opinion polls/political pundits) were pointing to a different conclusion. At Karna-AI, we gauge the trends in the mood of the masses through plots where you can visualize sentiment distributions on different topics determined using Deep Learning algorithms. We can even go a level deeper by breaking sentiment into emotions (sadness and anger both count as a negative sentiment but evoke different feelings) as we did in our latest study of Oscars 2017. In this study, we noticed a clear spike in ‘sad’ tweets when the in-memoriam was going on and a noticeable jump in ‘angry’ tweets when the best picture was incorrectly announced.

Similarly, an analysis of Twitter activity during Irish marriage referendum shows some very interesting results. They have an imperfect sentiment scoring (not using Machine Learning or Deep Learning, but just based on occurrence of positive/negative words), hence they work at a bigger picture, dividing the entire Twitterati around the referendum into multiple communities and looking at the aggregate sentiment, which strongly points at “Yes” becoming the likely outcome of the referendum.


There are other interesting results that show very less interaction between politically different communities and people from ‘ghettos’ on twitter according to their political ideologies. Karna-AI has tools to segment online users on the open web and find out the influencers in an open web network, so as to figure out the important topics that define that network.

Political ideologies tend to form large sub-networks within the Twitter network

In this study, the authors show that just modeling what the twitter connections of a user are talking about, his/her stance towards any issue can be determined, even if the user is dormant. It is a proven fact now that computers can analyze humans better than other humans do. So AI algorithms can either profile the user themselves or their neighborhood into categories and we can easily deduce their stance from these categories. We are very soon launching a product at KarnaAI to model social media users’ profiles and the ability to group and filter users.

Profiling a user and/or their network is a good metric to measure their stance


While these studies work on the big picture, there is another important aspect we might want to listen to. These are minority opinions, rare events or anomalies. They are all effectively texting classification problems in Machine Learning. One requires special text classifiers to work on datasets where one has to classify these rare data points from common ones, as searching for “needle in the haystack” gives a lot of noise (irrelevant data). These researchers discover positivity on Twitter (which they find to be very rare, <5%) using cascaded Machine Learning algorithms. Cascaded algorithms are a series of algorithms which are trained to correct errors of the preceding ones. It’s not just limited to positivity, we can listen to people talking about their pregnancies and can be used to look for depression symptoms from among the population.

Mental forums can be analyzed by AI warning users of potential episodes

Combining the network information and text information and training AI can even help generate automatic contextual responses like the one mentioned in this YikYak based study. Emojis, another component of social media and open web can also be predicted using AI on text interactions.


Another area of interesting research is the spread of propaganda and fake news. This analysis of ISIS accounts on twitter before getting suspended gives some very interesting insights. It seems small sub-networks of 100–1000 accounts follow each-other and tweet/re-tweet common generated content to increase their influence. Using bots for re-tweets is another possible conclusion. This paper also suggests a fast algorithm to block out people who are major nodes of such propaganda networks to stop the spread of what it calls “cyber-epidemics”. This paper gives a simple method to dig out communities that are not as obvious.

While we are talking about social media, fake news (and biased non-fake news) is also a problem generally on the open web. This paper suggests a method to group all coverage of a news story by different sources and let social activists compare the different versions and share whenever they see a bias. The authors used Google to get different coverage of the same news story, but that can be better achieved by Karna-AI’s Machine Learning based news analytics tools as well to automate this process. Talking of social unrest, there is work on automatically listening to agitations being planned and discussed on Twitter. It kind of works in a similar fashion as “needle in haystack” problems above.


I have been just talking about what can twitter in the form of the open web be used to deduce about the real world, but that is not the only Internet medium that gives us signals. Another important medium is discussion forums. The first question is what discussion forums to consider ? A study on Facebook discussions gives us some signals for this. It turns out that high-quality comments attract other high-quality comments. While this conclusion looks pretty obvious, the study proves that by displaying comments based on social feedback (likes in case of Facebook) instead of the most recent comment, better quality comments are attracted to the study, hence making the quality better gradually and avoiding the spam, troll or hateful comments. Also, the dynamics for posts with too many comments are different as spam or hurtful comments does not stop people from coming in and expressing their opinion. Extensive use of Machine Learning capabilities is needed for such studies where comments need to be graded for quality constantly in an automated fashion. At Karna-AI, we have our own intent classifier which filters user comments into opinion/feedback/marketing, etc to grade comments and filter irrelevant comments.

You can make your forum better just by showing high voted comments

Machine Learning algorithms can also be used as a triage for mental health forums giving alerts about the deteriorating condition of users of these forums. Citation networks of scientific publications can be analyzed using Machine Learning to show inherent bias and help counter such disadvantages of minorities and women. Data Mining on the same networks can help us trace citation cartels, which cause trivial publications to come out citing other trivial publications.

Paper citation networks can be analyzed for some great insights


With the advances in image Deep Learning methods, image-based Internet media like Pinterest, Instagram, and E-Commerce websites can also be mined for some brilliant insights. For example, this study from FeiFei Li’s group uses Image AI capabilities to read Google Streetview images and perform a survey of cars in the United States. It can count what models of cars it sees and can make a report without any human work. Any type of survey based on observations can hence be automated such as race distribution of a locality, amount of traffic on a street, a ratio of kids to adult and similar stuff which till now requires a lot of manual effort. There is another study on Social Media images that can help identify events like festivals and concerts and other such things directly from images shared on social media. Our team has done a similar study that identifies broad trends regarding how people interact with #gopro on Instagram.

This very interesting study takes food photos from Instagram, uses AI to classify these images and then compare the food articles seen from images of a county to common food-related problems in that same county (like alcohol overdose and diabetes).

This study similarly uses image classification ability to classify clothes shown on e-commerce websites into categories and then mine fashion trends over time.

Clothing trends can be analyzed using AI


In a nutshell, the open web Internet gives us extensive opportunity to feel the pulse of the world around us. AI algorithms have matured enough to act as an assistant to make sense of the insane amount of unstructured data generated on the internet every day.

Karna-AI is a product that helps you derive such insights from sources like news, blogs, forums, and social media which can be used for market research, reputation management or competitive benchmarking. It generates automated AI based reports that can be used to get quick feedback on your social media strategy.

Want to improve your social presence? Click here to schedule a free demo.

Source: Artificial Intelligence on Medium

Leave a Reply

Your email address will not be published. Required fields are marked *

Back To Top

Display your work in a bold & confident manner. Sometimes it’s easy for your creativity to stand out from the crowd.