Content filtering

Overview

Content filtering is technology that automatically analyzes text posted by users and detects harmful content such as abuse and defamation, threats, sexual content, and hate speech. It has become an important feature for services that handle user-generated content (UGC), such as social media, message boards, and chat services.

Filtering methods

There are several methods of content filtering. The basic method is keyword matching, which detects words on a list of banned words. It's easy to implement, but because it can't take context into account, false positives tend to happen. If you block every post containing the word "die," harmless sentences like "This cake is to die for" get caught too.

Machine-learning-based methods use models trained on large amounts of labeled data to make judgments that take context into account. From combinations of words and sentence structure, they can estimate the probability that a post has harmful intent. Filtering with large language models (LLMs) is said to make judgments based on a wider context, and it can sometimes detect clever harmful posts that include sarcasm or metaphors.

Design choices: automatic deletion vs. warnings

How to handle the results of filtering is an important decision tied to a service's design philosophy. With automatic deletion, posts judged harmful are deleted immediately. The operational burden is low, but there's a risk that legitimate posts disappear because of false positives.

With a warning approach, posts judged harmful get a warning mark, and the final decision is left to a person. This prevents legitimate posts from disappearing because of false positives, but it takes effort for the owner to check the warnings. In services like question boxes, where the recipient is an individual, the recipient is the one who understands the background of a question, so leaving the final decision to a person becomes an option.

The limits of filtering

No matter how advanced the filtering technology, it's impossible to detect 100% of harmful content. Language keeps changing, and new code words and slang keep appearing. Also, because the same expression can be harmful or harmless depending on context, fully automatic judgment is difficult in principle.

That's why it's appropriate to treat content filtering not as "a perfect wall" but as "a tool that reduces the burden." A realistic approach is a layered one: posts detected by filtering are handled automatically, and posts that slip through are dealt with by the owner deleting them manually.