Racecraft: Tale of Car Sensors

Racecraft · Technical Deep Dive · Sensor Fusion & Gemma Generalization

Fusing 100+ On-Device Sensors at 130 MPH , And Why Gemma Didn't Need Fine-Tuning

Inside Racecraft's real-time telemetry engine: multiplexing AiM CAN bus, RaceBox BLE, OBDLink, and 6-DOF IMU streams on Android, zero-shot corner doctrine generalization, and benchmarking Gemma 4 E2B on Tensor G5 silicon.

A few days ago after sharing early results from Racecraft (Project Koru), a question popped up in one of the comments that immediately caught my attention:

"Curious how your team handled real-time fusion of 100+ sensors on-device. Did Gemma need race-specific fine-tuning, or did general coaching logic generalize?"

It's the exact right question to ask. When people hear "AI race coach running on a phone in real time at Sonoma Raceway," they usually picture one of two extremes: either a brittle set of hardcoded if (speed < 40) statements, or a massive fine-tuned LLM that ate 10,000 hours of track telemetry during training and burns through your phone's battery in three laps.

The reality of how we built Racecraft is much more interesting , and much more reusable for anyone building high-frequency, real-time edge AI applications. In this post, I'm opening up the blueprint: how we fuse 100+ high-frequency physical sensor channels on Android without dropping frames, why we didn't fine-tune Gemma 4 (and why zero-shot generalization actually worked better), and the hard benchmark numbers across Tensor G5 NPU, GPU, and CPU execution lanes.

---

1. The 100+ Sensor Challenge: Ingesting the Vehicle Firehose

When Ajeet Mirwani and I strapped our Pixel setup into a Toyota GR86 at Sonoma Raceway, we weren't dealing with a clean, polite Web API emitting tidy JSON packages every second. We were tapping into a violent firehose of raw physical telemetry operating across fundamentally different interfaces, baud rates, and sample frequencies:

• AiM CAN Bus (USB SLCAN @ 1 Mbps / 500 kbps): Streaming 8 core CAN Frame IDs (0x420 CORE, 0x421 PRESSURE_RATES, 0x422 CONTROLS, 0x423 MOTION, 0x424 AUX, 0x450 WHEEL_SPEEDS, 0x451 ECU, 0x452 GPS_POSITION). This exposes individual 4-wheel speeds, raw/calibrated brake pressures (PSI), steering angle, engine RPM, ECU speed, DSC regulation activity, oil filter temps, water pressure, and 3-axis acceleration.
• RaceBox Mini (BLE @ 25Hz GPS + 10Hz IMU): High-precision 25 Hz latitude, longitude, Doppler speed, and heading vector. We request CONNECTION_PRIORITY_HIGH from Android OS to lock down Bluetooth LE packet jitter.
• OBDLink (Bluetooth / USB @ 10-20Hz): Polling engine diagnostics including Short/Long Term Fuel Trims (STFT/LTFT Bank 1 & 2), MAF g/s, Intake Temp, O2 Sensor Voltages (B1S1 & B2S1), and Timing Advance.
• Phone IMU & Vision (100Hz 6-DOF Gyro/Accel + Camera Store): Phone rotation vector, linear acceleration, and optional visual horizon/lane alignment snapshots from CameraX.

When you sum up every individual decoded channel across wheel speeds, fluid pressures, 6-axis motion, fuel trims, thermal sensors, and high-cadence GPS, you are continuously processing over 100 raw sensor variables simultaneously on the device.

Figure 1: High-cadence sensor fusion pipeline multiplexing CAN, BLE, and OBD inputs into a unified frame payload for Split-Brain routing.

How we keep 100+ channels synced without melting the thread loop

If you naively allocate objects or run reflection over 100+ channels on every 10ms frame, Android's garbage collector will stall your main looper within 30 seconds. To guarantee sub-50ms deterministic execution, we engineered three key mechanisms into TelemetrySources.kt and TelemetryFusionEngine.kt:

1. Bounded Bitrate Maps & Zero-Allocation Parsing: High-frequency CAN frames are parsed directly into primitive float arrays and cached in bounded LRU maps (boundedCanMap), preventing GC allocations on the live thread.
2. Multi-Tier Fallback Cascade: Hardware disconnects in a racing cockpit are inevitable (e.g. vibration loosening a USB-C CAN cable or OBD PID timeouts). The fusion engine classifies telemetry into 8 distinct fallback stages:

   aim_can_full ➔ aim_can_racebox_motion ➔ aim_can_phone_motion ➔ full ➔ racebox_only ➔ phone_obd_fusion ➔ phone_only ➔ no_live_data

   If CAN drops, the system seamlessly falls back to RaceBox BLE motion; if BLE drops, it falls back to Phone IMU + GPS, keeping the driver safe regardless of hardware status.
3. Sensor Trust Rating: The engine computes a live sensorTrust metric (0.0 to 1.0) based on GPS fix status, OBD sample age, and CAN frame freshness. If hardware health degrades, downstream AI coaching automatically suppresses hardware-dependent advice (e.g. brake pressure cues) and sticks to motion-proven cues.

---

2. Did Gemma Need Fine-Tuning? (The Generalization Breakthrough)

Now for the second half of the question: Did Gemma 4 need race-specific fine-tuning, or did general coaching logic generalize?

The short answer is: We did NOT fine-tune Gemma's weights for racing. General coaching logic generalized zero-shot across tracks, cars, and skill levels.

In fact, attempting to fine-tune an LLM directly on raw telemetry numbers is a trap. Here is why:

Fine-tuning a model on raw sensor arrays creates brittle numerical over-fitting. If you train a model that 38.4 MPH at Sonoma Turn 3 means "overbraking", the moment you put it on Turn 1 at Thunderhill at 72 MPH, or change tire compound, the fine-tuned weights hallucinate useless numbers.

Instead of forcing the LLM to do physics math in its weights, we separated Physical Evidence Computation (deterministic math) from Linguistic Strategy & Nuance (Gemma 4 E2B reasoning).

The Architecture of Zero-Shot Generalization

We achieved zero-shot generalization through three decoupled layers in our codebase:

A. Reference-Trace Delta Evidence (DEL)
At corner exit, our deterministic CornerPhaseDetector and del/ engine compares the driver's live apex frame against a gold-standard reference trace for that track. It calculates exact delta evidence: speed_delta: -14.2 mph, coasting_ratio: 0.18, input_smoothness: 0.62.

B. Property-Keyed Corner Doctrine (Zero-Shot Track Generalization)
Instead of hardcoding rules like "Sonoma Turn 4 is a right hander," corners are tagged with abstract doctrine properties: brakeZone, exitPriority, maintenance, sacrifice, doubleApex, trailBraking. Any track on earth with doctrine-tagged corners instantly works without writing a single line of track-specific code!

C. Grounded Prompt Ingestion into Gemma 4 E2B
We pass this pre-digested physical payload and corner doctrine hint to Gemma via a tightly constrained EdgeReasoningWindow. Here is what Gemma actually sees at the edge:

// Compact On-Device Prompt generated by LiteRtPromptFactory
Koru EDGE coach. JSON only: {"speak":true,"action":"THROTTLE","priority":2,"text":"<=14 words","confidence":0.0}
trigger=exit_hesitation; phase=EXIT; corner=T4; speed=41mph; decel=-0.8g
skill=INTERMEDIATE; doctrine=exitPriority; causeHint=late_throttle
hint=Commit on exit throttle.

Gemma's pretrained general intelligence , trained on millions of examples of human reasoning, cause-and-effect relationships, and communication styles , effortlessly translates this structured delta evidence into natural, punchy, contextual voice cues like:

{"speak": true, "action": "THROTTLE", "priority": 2, "text": "Unwind steering wheel before committing to full exit throttle.", "confidence": 0.88}

Confidence Routing & Fallback Safety

What if Gemma hallucinates or outputs low confidence? The engine enforces a strict confidence gate (confidence >= 0.6). If Gemma's output fails validation or times out beyond 750ms, the system discards the LLM response and the deterministic delta cue stands instead. The driver never hears a bad cue.

---

3. Benefits of Gemma & On-Device Silicon Benchmarks

By bringing Gemma 4 E2B directly onto the Pixel device via MediaPipe LiteRT-LM, we unlocked three game-changing benefits for automotive telemetry:

• 100% Offline Autonomy: Race tracks like Thunderhill or Laguna Seca are often tucked behind hills with zero cellular reception. Cloud LLM calls fail or latency spikes to 4,000ms. Gemma runs entirely locally on the phone's chip.
• Zero Latency Jitter: No network round-trips, socket renegotiations, or server queue delays.
• Non-Blocking Single-Flight Thread Model: The LLM runs in an isolated single-flight background coroutine. It never blocks the 5ms P0 safety path or telemetry ingestion.

On-device execution layout: Multiplexed CAN, BLE, and OBD sensor streams route through Google Tensor G5 NPU for <424ms TTFT zero-shot voice cues.

Empirical Benchmarks: Tensor G5 NPU vs. GPU vs. CPU

Using our instrumentation suite (OfficialLiteRtLmBenchmarkRunner and AcceleratorComparisonInstrumentedTest), we benchmarked Gemma 4 E2B across different hardware accelerators on the Pixel 10 (Tensor G5). Here are the empirical results:

Execution Lane	Model Artifact	Time to First Token (TTFT)	Decode Speed	Status in Racecraft
Tensor G5 NPU	gemma-4-E2B-it_Tensor_G5.litertlm	424 ms	36.2 tok/s	PRODUCTION PREFERRED
OpenCL GPU	gemma-4-E2B-it.litertlm	648 ms	22.4 tok/s	ACTIVE FALLBACK
CPU (4 Cores)	gemma-4-E2B-it.litertlm	1,820 ms	6.1 tok/s	OFFLINE TEST ONLY

Figure 2: Empirical TTFT benchmark showing Tensor G5 NPU comfortably staying inside the 750ms edge reasoning budget.

Prompt Optimization: The COMPACT Strategy

In our tests, we also benchmarked three prompt formatting styles in LiteRtPromptFactory:

• FULL prompt (includes entire feature map key-values): 780ms TTFT
• COMPACT prompt (pre-formatted key metrics + cause hint): 424ms TTFT (35% speedup)
• MINIMAL prompt (raw trigger string): 390ms TTFT (lacks sufficient context for voice naturalness)

By picking the COMPACT prompt style, setting maxTokens = 32, topK = 4, and temperature = 0.10, we achieved rock-solid JSON output structure with zero syntax errors across 157 automated test suites.

---

4. Key Takeaways for Edge AI Engineers

Building Racecraft taught us a few fundamental lessons about real-world on-device AI that extend far beyond motorsport:

1. Don't waste fine-tuning on raw signal math: Let deterministic algorithms compute physical deltas and domain evidence. Let the LLM handle contextual reasoning and human interaction.
2. Decouple safety from intelligence: The HOT path (<50ms deterministic rules) must run independently of model inference. If the LLM stutters or drops, safety and baseline feedback must never fail.
3. NPU is the true edge accelerator: On Tensor G5, NPU execution provided a ~35% TTFT speedup over GPU and over 4x over CPU, making live 400ms conversational loops possible on smartphone hardware.

Racecraft · An on-device, real-time AI driving coach built around Gemma 4 and LiteRT-LM. Open-source code & full architecture: github.com/rabimba/speedracer-AI.