Lesson 22: The Local Runtime — Serving LLMs Without a Cloud Bill
What We’re Building Today
Ollama running Phi-3 Mini locally at
localhost:11434, serving token streams over HTTP with zero GPU requirementA Server-Sent Events consumer in Node.js that reads the NDJSON stream token-by-token using built-in
fetchandReadableStreamAn OpenAI-compatible wrapper that routes
chat.completions.create()calls to local Ollama with no SDK changes, verified against the standard response schema
Why This Matters
In November 2023, OpenAI’s API went dark for several hours during their developer conference. Every application that had hardwired api.openai.com as its sole inference path stopped working. The cascading failures weren’t caused by bad architecture — they were caused by a single-point dependency on a remote service that no team controlled. NEXUS runs offline by design: every component, from the graph database to the vector store, operates without an internet connection. Leaving inference as a cloud-only operation would break that contract. More concretely, Microsoft’s Phi-3 Mini paper (April 2024) established that a 3.8-billion-parameter model trained on high-quality synthetic data can match GPT-3.5 on reasoning benchmarks while fitting in 2.3GB of RAM after Q4_K_M quantization. NEXUS adopts that model because it runs on the same hardware as the rest of the stack.
Preparing for a distributed systems interview?
→Download the free Interview Pack
→ Subscribe now to access source code repository - 200 + coding lessons
