Skip to main content

Posts

The Throughput Trap: Benchmarking vLLM on OpenXLA and the Reality of Production LLM Serving

vLLM Systems · DevLab 2026, Deep Dive I was recently invited by the Google TPU team to speak at the OpenXLA Summer DevLab 2026 . This post breaks down our deep-dive evaluation of the matured vLLM + OpenXLA stack, the fundamental engineering mismatches between CUDA and XLA serving paths, and why traditional capacity metrics are lying to you. If you are operating large language models at enterprise scale right now, your platform architecture team is likely staring at a massive infrastructure crossroads: Should we migrate our core serving workloads from GPUs to TPUs? Historically, NVIDIA's CUDA ecosystem was the only serious option for user-facing, low-latency LLM generation. But here in 2026, the economics and infrastructure options have transformed. Google TPUs are highly available, cheaper per chip, and the open-source serving stack built around vLLM and OpenXLA has officially achieved absolute production readiness. Yet, when our infrast...
Post a Comment
Read more
Recent posts

Splitting the Brain to Beat the Clock

Racecraft · Part 3 of 5 · ← Prologue Splitting the Brain to Beat the Clock How a "brake!" lands in 5 milliseconds while a cloud model thinks for five seconds — in the same app, on the same frame, without ever colliding. Two posts in, we have a coach that knows who's driving and what to say. This post is about the only thing that lets it say anything useful: structure. Specifically, the decision to give the system not one brain but three, each on its own clock, with an ironclad rule about which one is allowed to make the driver wait. I call it the Split-Brain engine , and the whole design collapses out of one observation. The three jobs a coach does — react, strategize, prepare — have wildly different deadlines. Trying to serve all three from one code path means the fastest job inherits the latency of the slowest. That's the original sin of every cloud-first coaching app. So I refused to let them share a path. The Spli...
Post a Comment
Read more

Teaching the Coach to Read the Driver

Racecraft · Part 2 of 5 · ← Prologue Teaching the Coach to Read the Driver The best instructors don't coach the car. They coach you , and they figure out who you are in about two laps. Here's how we taught software to do the same. In Part 1 I argued that trust is the only metric that matters, and that it's mostly a latency problem. That's true , but there's a second half I glossed over. The same sentence, delivered at the exact same millisecond, can build trust or destroy it depending on who's listening. Tell a nervous first-timer "brake spike detected, modulate your input" and you've just handed them a stack trace mid-corner. Tell a fast amateur "squeeze the brakes, don't stab" for the tenth time and they'll mute you out of sheer irritation. The words have to match the driver. So before Racecraft can say anything, it has to answer a question a human coach answers instinctively: how good is this person, rig...
Post a Comment
Read more

The 800-Millisecond Problem

Racecraft · Part 1 of 5 · ← Prologue The 800-Millisecond Problem Why I stopped trusting every " AI race coach " I'd ever tried , and what it would take to build one I'd actually listen to at 130 km/h. The first time an app yelled "Brake!" at me on a track day, I'd already braked — not by a hair, but by a full corner. I was unwinding the wheel and feeding in throttle on the exit of Sonoma's Turn 7, one of the fastest corners on the circuit, when the phone told me — with great confidence — to brake. It wasn't merely lagging by a beat; for that specific high-speed corner the cue was flat wrong, because braking mid-exit at that speed is how you put the car into a spin. And I couldn't even mute it. I was a beginner on a mandatory-instructor day, both hands on the wheel, eyes up. It was the instructor in my passenger seat — there to call flags and lines, allowed to say anything but never to touch a single control — who reached...
Post a Comment
Read more

Racecraft (Project Koru) · Prologue — The Origin Story

Racecraft · Prologue , The Origin Story It Started With a Wine List and a Question About Racing How a happy-hour conversation in the Bay Area turned into a trustable AI race coach , and then into a second version that runs entirely on a phone, on the NPU. This is the prologue to a five-part series. Two years ago(1st November, 2024) I was in the Bay Area for a GDE Summit. If you've never been: it's a couple of days of talks among Google Developer Experts, the kind of people who get unreasonably excited about a new on-device runtime, and then , mercifully , a happy hour where everyone stops performing and just eats. We ended up at a restaurant(Puesto Santa Clara), a long table of GDEs, and I was doing the most important engineering of the evening: trying to decide which wine to order. Across the table was Ajeet Mirwani . I don't even remember how the wine talk turned into racing talk , these things drift , but the moment the word "racing" ...
Post a Comment
Read more

The Blind Spot Horizon: Why Your AI Benchmarks Are Lying to You

"If you measure the wrong thing, you optimize for the wrong thing. And when dealing with frontier, multimodal models, the wrong thing might just break your entire system without throwing a single error." We are remarkably comfortable evaluating the AI models we’ve already built. We have our standard suites, our automated leaderboards, and our hard-coded unit tests. We feel in control. But let me drop a truth bomb that’s been brewing across frontier labs: we are profoundly, staggeringly bad at evaluating the models we are about to build . Most benchmarks, safety evaluations, and red-teaming protocols operate on a comforting but incredibly lazy assumption. They treat the next iteration of an LLM or a large multimodal model like a linear upgrade,like turning a dial from 8 to 10. But if you’ve spent any time hacking away at deep learning architectures or building agentic frameworks, you know that complex networks don’t scale smoothly. They undergo massive, silent phase transi...
Post a Comment
Read more