"Having good, high-quality discussions between our users on our stories has always been at the center of what we do. To achieve this goal, we had to invest a lot of our editors' time, time they could use for writing and researching instead. Working with Utopia has allowed us to overcome these challenges with less effort, while keeping the heart and quality of our discussions alive.”

Megan Silberbauer
Chief Product Officer at Watson News

‍

1. Challenges & earlier approaches

Before teaming up with Utopia, Watson had challenges with moderating the large volume of reader comments they get for their news articles. Most concerning ones being having inappropriate comments and off topic comments being published. At the same time, holding comments that were totally fine waiting in queues, was clearly decreasing reader engagement.

Their earlier reader comment moderation system depended on two things:
      - A bot that automatically filtered comments based on specific words
      - A team of editors and freelancers manually checking comments

As the bot could only automate about 40% of the moderation, and because its accuracy rate wasn't particularly high, it didn’t make a significant reduction in the moderation team’s workload. This also forced Watson to keep bringing in new freelancers to manage the growing comment volume, causing rising costs.

Watson had also tried to make the moderation process smoother by creating a "trusted users" program. Here, the users were scored based on whether their comments were appropriate or not. Upon reaching a certain threshold, users could have their comments published straight away without additional checks. However, this approach had a few disadvantages. For example, new users could not have their comments published in real time, even if they were appropriate, and conversely, trusted users could eventually misbehave and get their inappropriate comments published regardless.

Additionally, sensitive articles were also treated differently, where every comment needed a thorough review before going live, adding manual loadwork for the team. Watson realised their process presented some major challenges and that they needed a new approach.

2. Objective & solution

Watson had heard that Utopia Analytics has been working with another Swiss news media company and decided to find out if Utopia could also help them improve the effectiveness and efficiency of their comment moderation.

Watson decided to try the Utopia AI Moderator solution through Utopia’s standardised pilot process. First, Utopia provided Watson with instructions on how to collect the training data set together with instructions on how to integrate from Watson commenting platform to Utopia API.

Next, Utopia trained a tailor-made AI moderation model based on Watson’s historical human-moderated comments. During the training, Utopia’s team spotted that Watson's team didn't have a uniform approach on moderation. It seemed some moderators were very lenient, approving the majority of comments, while others were quite strict, rejecting most. This was very surprising to Watson’s team as they did not expect that the current setup allowed for such levels of inconsistency. This was not only an eye-opening situation but also a turning point where they knew something needed to change for the better.

On the other hand, to Utopia’s team this wasn't surprising given that it is a very common challenge with a team of moderators in any company. Even less surprising for Watson who was juggling a mixed team of in-house staff and a group of external helpers, making it even harder to keep their comment publishing decisions consistent.
Once Watson had finalised the quick and easy integration to Utopia API, they proceeded to test their new AI-based moderation approach with new comments that were coming in volumes everyday.

3. Results

The pilot showed promising results right from the start. Watson was pleasantly surprised to see that integrating Utopia's AI moderation technology didn't compromise the quality of the published comments, a major concern they had had initially. On the contrary, they noticed it helped improve the quality of discussions as Utopia AI now spots 2 to 2.5 times more improper comments before they get published than their earlier approach. The new approach has also significantly increased their reader engagement as acceptable & relevant comments are now being published in real time.

Based on the pilot results, it was an easy decision for Watson to start using the Utopia AI Moderator solution in their daily operations. In just a few months, in addition to the improved quality, Watson has also reduced their comment publishing delay and has managed to cut down the manual effort by 50%, a huge improvement on operational efficiency. As the moderation workload has shrunk to a half of what it used to be, this also represents a real cost saving for Watson.

About Watson

Watson News is an online portal for news and entertainment that focuses on Switzerland, especially the German-speaking part. It was founded in 2013 and it is one of the top 3 online news media in Switzerland.