Lesson 31: The Rolling Summary — Trending TL;DRs via Sliding Windows
What We’re Building Today
A background worker that maintains sliding 60-minute windows over Redpanda topic streams, querying the top 50 tweets per trending topic and feeding them to Ollama for incremental 3-bullet summarization
A summary merger that blends the incoming incremental summary with the existing one rather than discarding and regenerating — measuring exact token cost and wall-clock time delta between both strategies
A
/api/trending-tldrendpoint that serves the merged summaries to a feed header component, refreshing every 60 seconds with measurable staleness bounded at one window width
Why This Matters
When Slack rebuilt their message search and digest pipelines in 2019, they discovered that full re-summarization of channel activity — triggered naively on every notification flush — was responsible for 34% of their inference budget for a feature used by fewer than 8% of users. The root cause: their summarizer treated each run as a cold start. It re-read the entire window, re-embedded it, and re-generated output, even when 90% of the input hadn’t changed since the last pass.
NEXUS faces the same trap the moment trending topics attract sustained posting activity. A topic accumulating 50 tweets per hour will trigger a full summarization every 60 seconds. At Ollama’s throughput on consumer hardware, that’s 8–12 seconds of inference per topic. With five active topics, the background worker is always behind. The sliding window plus incremental merge pattern is what prevents the TL;DR service from becoming a queue of regrets.
Core Concepts
