Emote-Controlled

Imagine you are a famous Twitch user and you stream your videos on your channel for thousands of viewers. The audience can express their feelings regarding events in your stream by commenting in real-time. Because comments arrive at a fast pace, you are not able to read them all and understand how certain events are perceived by your audience.

In our paper Emote-Controlled, we explore the use of sentiment analysis on Twitch comments to automate the feedback gathering process for these situations. Given enough comments over time, we can calculate the mean sentiment of user’s comments and correlate these with events in the video.

We analyzed approximately 3 Billion public Twitch comments and found that Twitch’s language is different from common English:

• Twitch comments are very short. The median comment has just 3 words.
• Twitch provides so called emotes, which are extraordinary popular. Emotes are small images showing faces, symbols, or other meme-like pictures.
• Many ALL-CAPS comments, which indicates that people are “shouting”.
• Many comments only contained few words that were repeated multiple times. This results in longer/larger comments that may be better seen by other users.

Due to these characteristics, we found that lexicon based methods that were designed for common English did not perform well on Twitch comments. Emotes, however, capture some kind of sentiment. That is why we extracted the most frequent 100 emotes from the comments and created an emote lexicon.

We combined this lexicon with other lexica, such as an emoji and an English word lexicon, and tested the performance of two lexicon based methods and one neural network based approach on test comments we labeled using crowd-sourcing.

We found that especially the emote lexicon was very helpful in determining the sentiment of Twitch comments. To apply our method in the real world, we analyzed the sentiment trajectory of two popular Twitch streams.

One of them was the BlizzCon 2018 Opening Keynote, where a new Diablo game was announced (at around 2:31:00 in the linked video). However, the game was mobile-only, so it was not perceived well by the computer game focused audience. With our method, we also found this drop in sentiment in the Twitch comments, which you can see in the following image.

For more information, take a look at the paper. You can find the code and the data, including the novel emote lexicon, on GitHub. In addition to the publication of the paper in the TSC journal, we will also present this work at CHI 2020.