Lesson 23: The Model Selection Matrix — Phi-3 vs Gemma vs Llama 3.2 on CPU
What We’re Building Today
A multi-model benchmark harness that runs Phi-3 Mini, Gemma 2B, and Llama 3.2 3B through four social media tasks via Ollama’s HTTP API
A labeled 50-tweet test set with ground-truth annotations for sentiment, toxicity, summarization quality, and QA correctness
A decision matrix that assigns the best-performing model to each pipeline slot in NEXUS, with per-task P99 latency and accuracy recorded
Why This Matters
In 2023, Roblox shipped a moderation system that used a single general-purpose LLM for every classification task — spam detection, tone analysis, and policy violation scoring all routed through the same model endpoint. When traffic spiked during a major game launch, that model became a throughput bottleneck at precisely the moment accuracy mattered most. The system had no fallback because nobody had measured whether a smaller model could handle any of the subtasks adequately.
NEXUS faces the same trap at the design stage. Without a measured model selection policy, every inference call goes to whatever model was last pulled. The result: 4× the latency you need for sentiment classification (a task where Phi-3 Mini is competitive with models three times its size), and a pipeline that will buckle under load because you never asked which tool fits which job.
Core Concepts
Task-Model Fit as a First-Class Engineering Decision
The intuition: a 3B model that’s right for the job outperforms a 7B model that’s wrong for it, and does so in half the memory. The mechanism is instruction fine-tuning specialization — Phi-3 Mini was tuned on reasoning-dense synthetic data, which makes it disproportionately strong on classification tasks with short context. Gemma 2B’s training mix included more conversational data, giving it better coherence on summarization. Llama 3.2 3B has the widest general capability distribution, making it the natural choice for open-ended QA where the query shape is unpredictable.
In NEXUS, this matters at every stage of the inference pipeline. When a tweet enters the CDC stream from Redpanda, it may need sentiment tagging before routing to a Qdrant collection, toxicity scoring before storage, and summary generation before embedding. Running all three through Llama 3.2 when Phi-3 Mini handles sentiment at 2.1× the throughput is waste that compounds at scale.
The tradeoff: task-specific routing adds a dispatch layer. You gain per-task optimization; you give up the simplicity of a single model endpoint.
Tokens Per Second as the Wrong Primary Metric
Engineers default to tokens/sec because it’s easy to measure. The production metric is task completion latency: time from HTTP request to parsed, validated output. A model that emits 80 tokens/sec but requires 200 tokens to answer a sentiment question is slower than one emitting 45 tokens/sec that answers in 40 tokens.
The mechanism behind this: instruction-tuned models learn task-specific answer lengths during fine-tuning. Phi-3 Mini on a binary classification task terminates after 3–5 tokens (”Positive” or “Negative.\n”). Llama 3.2 3B tends toward verbose justification before the label. Both behaviors are baked into the model weights, not controllable by temperature alone.
In NEXUS, benchmark each task with time_to_first_token and total_completion_time captured separately. The former predicts perceived responsiveness; the latter determines pipeline throughput.
The tradeoff: measuring correctly requires instrumenting around the HTTP call, not inside Ollama. You get accurate numbers; you give up the simplicity of using Ollama’s reported stats directly.
Accuracy Calibration on Domain-Specific Data
The intuition: a model that achieves 91% on the SST-2 benchmark may achieve 74% on tweets about cryptocurrency because the fine-tuning distribution didn’t include that vocabulary register.
The mechanism: tokenization mismatch between benchmark text and your target domain degrades attention head activation patterns in the early layers. For short social media text, this effect is amplified — there are fewer tokens available for the model to recover from a poor initial representation.
The 50-tweet labeled test set in this lesson is annotated specifically for NEXUS content: posts mentioning AI agents, decentralized identity, data streams, and technical communities. Using SST-2 or IMDb-derived benchmarks would overestimate accuracy for the actual inference workload.
The tradeoff: domain-specific annotation costs time upfront. You get calibrated accuracy numbers that predict production behavior; generic benchmarks would be faster to produce but misleading.
Decision Matrix Construction
A decision matrix is not a leaderboard. A leaderboard picks one winner globally. A decision matrix assigns each task slot independently based on the Pareto frontier of accuracy and latency for that specific task.
The mechanism: for each of the four tasks, compute a composite score S = accuracy_pct / p99_latency_ms. The model maximizing S for task T is the default assignment for T. Where two models are within 5% accuracy of each other, the faster one wins — in production, the latency budget is a harder constraint than an incremental accuracy improvement.
In NEXUS, the decision matrix output feeds directly into engine.mjs as a static routing table. When a tweet arrives requiring classification, the engine dispatches to the pre-selected model for that task type, not to a general-purpose endpoint.
