I spent two days trying to get TypeSafe's Jev to do impressive things. I fed it a live news firehose and asked it to predict market moves. I gave it 500 windows of BTC price data and asked it to trade. Both experiments produced coin-flip accuracy or worse.
But buried in the data from these failures was something that actually works — and it's so boring that I almost didn't write about it.
It sorts piles
Across the prediction market experiment, Jev classified 19,740 news-market pairs into three buckets: none, weak, strong. I ran a control arm — same requests, same model, same everything — to measure whether Jev gives the same answer to the same question.
90% self-consistency on a three-way classification at 175ms and $0.00007 per call. That's not flashy, but it's genuinely useful. It means you can use Jev as a first-pass filter on a high-volume stream and trust that the sorting is stable.
The fact that removing the market price barely changed the results tells you something important about what Jev is actually doing. It's not anchoring on the price. It's reading the news text against the resolution criteria. It's a text classifier.
The confidence score is just math
TypeSafe's docs say Jev returns a "confidence" score alongside each answer. The confidence cookbook is marked "coming soon." In the meantime, the confidence feels like a separate signal — a meta-judgment about how sure the model is.
It's not. I ran the formula (p_max - 1/n) / (1 - 1/n) against 87,273 answers across all question types and option counts. p_max is the top probability, n is the number of options.
Mean absolute error: 0.0048. The confidence score is a deterministic rescaling of the top probability. It carries zero additional information. If you're building threshold gates on Jev's confidence, you can use the top probability directly and skip the mystery.
What it costs vs what it replaces
The economics are the real story. At $0.00007 per classification, Jev is two orders of magnitude cheaper than a frontier model for the same job.
The speed difference matters too. Jev at 175ms vs a frontier model at 1-3 seconds changes what you can build. At 1,200 requests per minute, you can classify a firehose in real time. Content moderation, ticket routing, relevance filtering, document triage — anywhere you need a fast yes/no on unstructured text, these economics make sense.
Three lessons from testing this
Test self-consistency before you test accuracy. If the model gives different answers to the same question 10% of the time, your accuracy measurement has a 10% noise floor built in. I found this by adding a control arm — running the same questions twice. Without that, I would have attributed random variation to a deliberate schema change and rewritten my entire question set for nothing.
An honest "I don't know" is a feature. When I asked Jev a yes/no question about BTC direction, it answered 0.5 — "I don't know" — 82% of the time. That felt like a failure. But compare it to the forced-choice version that predicted "down" 63% of the time while BTC was going up. The calibrated uncertainty was the correct response. A model that admits ignorance is more useful than one that makes confident wrong calls.
Use it for classification, not prediction. Jev reads text against criteria and sorts it into buckets. It does this reliably, fast, and for almost nothing. Don't give it numbers and ask what comes next. It's trained on language calibration, not time series patterns. The hype is "AI decisions at the speed of thought." The reality is "cheap, consistent text classifier." That's less exciting and more useful.
All code and data are at github.com/noreff/jev-signal.