Lesson 26: Beyond Keywords — Context-Aware Toxicity Detection
What We’re Building Today
A multi-label toxicity classifier backed by Ollama that reasons before it labels — producing a chain-of-thought trace alongside each verdict
A 50-tweet verification suite that asserts ≥ 90% classification accuracy against a deterministic ground-truth set, with per-label breakdown
A reasoning trace logger that captures the model’s justification for every decision, forming the audit record for moderation appeals
Why This Matters
In 2022, Discord’s Trust & Safety team published a retrospective on their AutoMod system. Their keyword-regex pipeline achieved 94% recall on slurs — and generated a 34% false positive rate that banned users for quoting song lyrics and news headlines. The cost: 1.4 million appeals in Q3 alone, each requiring a human reviewer. The problem wasn’t the word list. It was the architecture’s inability to model context. “I hate Mondays” and “I hate [ethnic group]” are syntactically identical to a regex. They are categorically different statements to anyone with a theory of language.
NEXUS without this lesson would inherit the same failure mode. The Day 25 classification pipeline streams tweets through Redpanda consumer groups — fast, parallel, bounded. But if the classifier at the center of that pipeline is pattern-matching on tokens, you will ban your most engaged users for quoting themselves.
Core Concepts
Chain-of-Thought as Audit Infrastructure
The instinct when building a classifier is to prompt for the label directly: “Is this tweet toxic? Respond: yes/no.” That instinct is wrong for two reasons. First, large language models produce better classifications when they reason before they commit — a phenomenon quantified by Wei et al. (2022) as chain-of-thought prompting, which improved PaLM’s accuracy on multi-step reasoning tasks by 41 percentage points over direct answer prompting. Second, a bare label is legally and operationally useless. When a user appeals a suspension, you need a paper trail explaining the machine’s reasoning, not just its verdict.
The mechanism: your prompt instructs the model to emit a structured block — REASONING: ... followed by LABELS: [...] — before outputting the final classification. The parser splits on these delimiters. The reasoning block lands in your audit log. The label block drives the enforcement action. You get both signal quality and explainability from a single inference pass.
In NEXUS, this trace feeds directly into the moderation_log table in SurrealDB, timestamped and linked to the originating tweet ID. The Day 27 vector search integration will embed these traces to surface precedents when edge cases recur.
Multi-Label Classification via Constrained Output
Binary toxicity detection (toxic/not-toxic) is architecturally insufficient. A single tweet can simultaneously be spam and misinformation. A targeted harassment campaign often involves content that is individually borderline — only the pattern is toxic. You need a label set: harassment, spam, misinformation, hate_speech, safe.
The enforcement mechanism is grammar-constrained sampling — the same technique used in Day 25’s classification pipeline. Rather than post-processing free text, you constrain the model’s output to valid JSON matching a schema: { "labels": ["harassment", "spam"] }. This eliminates hallucinated label names and removes the need for fuzzy string matching in your parser.
The tradeoff: constrained sampling requires a model that supports JSON mode or a grammar parameter. Ollama exposes this via the format: "json" field in its /api/generate endpoint. For models that don’t support grammar constraints, you fall back to a regex parser on the LABELS: delimiter — slower to build, brittler at the edges, but functional.
In NEXUS, the label set maps directly to enforcement tiers: safe → no action; spam → rate-limit; harassment → shadow-restrict; hate_speech → queue for human review; misinformation → attach warning label.
Prompt Engineering as System Architecture
A toxicity prompt is not a prompt. It is a specification document that encodes your platform’s moderation policy in executable form. The structure matters:
Role assignment: establishes the model’s perspective and constraints
Label definitions with examples: prevents label drift across model versions
Explicit edge cases: “quoting harmful content to critique it is not hate_speech”
Output format specification: drives the constrained parser
Reasoning instruction: placed before the output spec so the model reasons before it formats
The production tradeoff is prompt length vs. inference cost. A 400-token system prompt adds ~18ms to every classification at Phi-3 Mini’s throughput. On a pipeline processing 50,000 tweets/hour, that is 250 CPU-hours per day in prompt evaluation alone. You amortize this by batching: classify 8 tweets per inference call rather than 1, reducing prompt overhead by 87%.
NEXUS implements single-call classification in this lesson. Day 28 introduces batched inference with the same engine interface.
False Positive Budget
Every moderation system has a false positive rate. The question is whether you measure it. A 5% false positive rate on a platform with 100,000 daily active users means 5,000 legitimate posts incorrectly flagged — per day. If 2% of those users churn after a false suspension, you lose 100 users daily to classifier error.
The 90% accuracy target in this lesson’s test suite is not an accuracy floor — it is a budget constraint. You are committing to a false positive rate ≤ 10% across the 50-tweet ground-truth set. The per-label breakdown in the test output lets you identify which category is leaking: misinformation is typically the noisiest label because its boundary with safe is genuinely ambiguous at the level of a single tweet without external grounding.
Component Architecture
Tweets enter the ToxicityClassifier via its classify(text) method. The classifier constructs a prompt from the system template (loaded once at initialization) and the input text. It calls the Ollama /api/generate endpoint with format: "json" and stream: false. The response body contains a response field with the model’s output.
The ResponseParser splits the output on the REASONING: and LABELS: delimiters. If either delimiter is absent — network timeout, model refusal, malformed output — the parser returns { labels: ["safe"], reasoning: "parse_failure", confidence: 0 }. This fail-safe default is deliberate: a classification failure must never result in an incorrect enforcement action. You accept false negatives over false positives on parse error.
The ReasoningTraceLogger writes the full trace — input text hash, model response, parsed labels, latency — to an in-memory append-only log. In ExternalEngine mode, this log is flushed to SurrealDB via the /sql endpoint in 10-record batches.
State that persists between requests: the system prompt string (compiled once) and the trace log (append-only). No model state is held in the engine — Ollama manages its own KV cache. On failure, the engine retries once with a 500ms backoff, then returns the parse-failure safe default.
