Jev is an honest game changer.
On 48 real game questions, Jev plus a Gemini fallback matched Grok's accuracy with 84.0% lower mean latency and 88.5% lower mean cost.
The best feature of Jev is not the speed or low cost but the way it's honest when it doesn't know something whereas an LLM will often confidently provide an incorrect answer. This, combined with the high efficiency at classification problems makes it a great gatekeeper in front of an LLM.
Jev with a Gemini fallback made my evaluation 6.24x faster and 8.7x cheaper than Grok 4.6, matching its 89.6% accuracy. That's an 84.0% reduction in mean latency and an 88.5% reduction in mean cost.
The part I find interesting is how little Gemini had to do. Jev is a constrained 'decision' model that handles common questions and exposes uncertainty. It scored 81.3% on its own. Sending low-confidence questions to Gemini added 8.3 percentage points, with Gemini invoked for just 14.6% of questions. I wanted to understand when a smaller model could do enough, and when it should ask for help.
That is why Jev feels like an honest game changer. It does not pretend to replace a general-purpose LLM. It handles the common path, admits uncertainty, and makes the expensive model optional.
- Frontier among LLM-only policies (Jev excluded)
- Jev-enabled frontier
- Jev only
- GPT-OSS-120B only
- Gemini 3.8 only
- Grok 4.6 only
- Jev → GPT-OSS low
- Jev → GPT-OSS low+medium
- Jev → Gemini low
- Jev → Gemini low+medium
- Jev → Grok low
- Jev → Grok low+medium
Smaller hollow points use the low+medium confidence threshold.
- Frontier among LLM-only policies (Jev excluded)
- Jev-enabled frontier
- Jev only
- GPT-OSS-120B only
- Gemini 3.8 only
- Grok 4.6 only
- Jev → GPT-OSS low
- Jev → GPT-OSS low+medium
- Jev → Gemini low
- Jev → Gemini low+medium
- Jev → Grok low
- Jev → Grok low+medium
Jev → Gemini on low confidence matches Grok's accuracy with much lower mean cost and latency. Standalone Gemini is the most accurate policy.
| Policy | Matched label | Valid responses | Valid-only | Questions sent to LLM | Mean | Median | P90 | Mean cost |
|---|---|---|---|---|---|---|---|---|
| TypeSafe Jev 1.13 (Decisions; effort N/A) only | 39/48 (81.3%) | 48/48 | 81.3% | 0.0% | 420 ms | 360 ms | 515 ms | $0.00005001 |
| GPT-OSS-120B Nitro (medium; Cerebras) only | 34/48 (70.8%) | 46/48 | 73.9% | 100.0% | 530 ms | 468 ms | 774 ms | $0.00044833 |
| Gemini 3.8 Flash (medium) only | 46/48 (95.8%) | 48/48 | 95.8% | 100.0% | 5830 ms | 3592 ms | 13568 ms | $0.00198019 |
| Grok 4.6 (medium) only | 43/48 (89.6%) | 48/48 | 89.6% | 100.0% | 11369 ms | 8384 ms | 22127 ms | $0.00552654 |
| Jev, then GPT-OSS-120B Nitro (medium) on low confidence | 39/48 (81.3%) | 48/48 | 81.3% | 14.6% | 492 ms | 379 ms | 876 ms | $0.00011548 |
| Jev, then GPT-OSS-120B Nitro (medium) on low or medium confidence | 38/48 (79.2%) | 48/48 | 79.2% | 22.9% | 530 ms | 392 ms | 876 ms | $0.00015609 |
| Jev, then Gemini 3.8 Flash (medium) on low confidence | 43/48 (89.6%) | 48/48 | 89.6% | 14.6% | 1823 ms | 379 ms | 8583 ms | $0.00063487 |
| Jev, then Gemini 3.8 Flash (medium) on low or medium confidence | 43/48 (89.6%) | 48/48 | 89.6% | 22.9% | 2567 ms | 392 ms | 10376 ms | $0.00095304 |
| Jev, then Grok 4.6 (medium) on low confidence | 41/48 (85.4%) | 48/48 | 85.4% | 14.6% | 2975 ms | 379 ms | 10838 ms | $0.00101659 |
| Jev, then Grok 4.6 (medium) on low or medium confidence | 41/48 (85.4%) | 48/48 | 85.4% | 22.9% | 4940 ms | 392 ms | 22438 ms | $0.00180184 |
A response matches when its label is in the manually defined acceptable set. Cost and latency use full caller-to-response wall time.
The test
I sampled 48 privacy-scrubbed questions from 48 production sessions. Thirty-six follow the normal traffic mix. Twelve are deliberately difficult, including questions that received substantive negative feedback and questions with genuinely ambiguous wording.
I labelled the questions before seeing the model outputs. Some questions accept more than one label where the wording supports it. This measures agreement with a small human-authored answer set, not general intelligence.
The replay starts at Jev and does not exercise the deterministic routes that run before it in production.
Jev matched 39 of 48 labels on its own. It marked seven questions as low confidence. Sending those seven to Gemini raised the result to 43 of 48, while keeping the median response time at 379 ms.
Sending medium-confidence answers to Gemini as well did not improve accuracy. It only added latency and cost. Jev's confidence was useful enough to identify the right boundary.
What changed
Jev is now the semantic middle layer in production. It is fast enough for the common path and honest enough about uncertainty to know when it needs help.
The result is not that a small constrained model beat every general-purpose LLM. Standalone Gemini scored higher. The useful result is that Jev avoided most generative calls while matching Grok's accuracy at a fraction of its latency and cost.
That is the game changer: not replacing the expensive model, but making it optional.