In a nutshell
Hamish tested specialist AI decision model Jev against Claude Opus 5.5 using 500 yes/no questions. Claude got slightly more right at the default setting, but the two were about equally good at telling YES from NO, and both models could return probabilities that allowed decisions to be adjusted using thresholds. The real differences were cost, speed and consistency, with Jev proving dramatically cheaper and faster. The experiment raises an interesting question: when AI models can perform similar tasks, what should actually determine which one you use?
Putting Jev's claims to the test
Jev is a decision model from TypeSafe. You give it some text and a yes/no question, and it hands back a single number, the probability that the answer is YES. Its launch post says it matches existing LLMs on these tasks while being "two orders of magnitude faster and more efficient" ("40x-200x faster"), that it "can't hallucinate", and that its probabilities are "calibrated" where LLMs asked for a confidence "tend to be overconfident and inconsistent". There's a lot of talk at the moment about all the new things you can build with that.
My first reaction was less generous.
Can't you just get Claude to do that?
The obvious answer is the dial. Because Jev returns a probability rather than an answer, you can set a threshold in code and decide how many false alarms you'll put up with to catch more of the real YESes. I'd assumed that was the thing an LLM couldn't give you. So I put both through the same 500 questions to find out.
How I tested them
The questions came from BoolQ, a public reading-comprehension dataset from Google Research. Each comes with a short passage and an official answer, and both models got the same passage and question, once each, with nothing else to go on.
Jev 1.14 (model ID jev-1.14.0) answered through its native yes/no primitive. Claude Opus 5.5 (model ID claude-opus-5-5), one of Anthropic's largest models, ran with its default settings and answered through structured output, which forces it to return one number and nothing more. So both hand back the same thing, the probability of YES, and code makes every decision without either model ever seeing the threshold. Every call was made on 30 October 2026. I first ran the whole comparison in September on Jev 1.13 and Claude Opus 5; the numbers here are from the current versions, and where the older ones told a different story I say so.
Putting the claims to the test
A controllable threshold: Claude can do this
At a threshold of 0.5, Jev got 89.6% of the questions right and Claude 92.2%, where saying YES to everything scores 63%. Most of that gap comes from where the 0.5 line sits: Jev's scores for real YESes run a little lower, so at 0.5 it misses more of them. Take a random YES question and a random NO question, and Jev scores the YES higher 96.5% of the time, Claude 96.3%. More to the point, both scores work as a dial: move the threshold and precision and recall trade off smoothly for both models. Compared at matched recall, the gap between them is a handful of false positives out of 500, ten at most. In other words, Claude was almost as good at distinguishing between genuine matches and false alarms. Jev's supposed advantage wasn't as unique as I'd expected.
So Claude has a dial too. On this task, the dial comes from asking for a probability, not from which model you ask.
Precision against recall for Jev and Claude Opus 5.5 at every attainable threshold. Both curves stay at or above 0.95 precision until recall reaches about 0.88 for Jev and 0.92 for Claude, then fall steeply; the two lines nearly overlap throughout.
Lots of questions in one call: mostly
This is where Jev is genuinely different. Asking it eight questions about one passage took no longer than asking one, while the same batch made Claude 1.46 times slower. Batching also nudged Claude's answers: its scores moved about twice as far when a question travelled with seven others as when the same question was simply asked again, where Jev's moved little more than its usual call-to-call noise. Claude still gets something out of batching, though. A single response carries a fixed overhead, so spreading it across eight questions cut Claude's cost per question by nearly three-quarters.
Stable, repeatable scores: less so
I asked the same 100 questions five times each, and Claude's scores moved about twice as much as Jev's between identical calls. That's the launch post's "inconsistent", and on this test it held, though less strongly than on Opus 5, which moved about three times as much. Turning Claude's effort down to low made it steadier rather than noisier.
Does that matter? In practice it rarely changed an answer. At a 0.5 threshold, 4 of Claude's 100 decisions flipped between runs, against none of Jev's. Four is too few to read as a rate, though: Claude's true figure could plausibly sit anywhere from about 1% to 10%. And Jev isn't deterministic either: only 32 of the 100 came back with identical scores all five times.
For the four examples whose decision flipped at 0.5, each model's five scores as dots. Jev's dots cluster tightly; Claude's spread across the 0.5 line, from about 0.3 to 0.52 on the widest.
Cheap and fast: no
By my estimate from published pricing, Claude cost about 160 times as much for the 500 questions: $1.53 against just under a cent. That's the launch post's two orders of magnitude, at least on price, though only against one of Claude's largest models. A smaller Claude would cost far less and narrow the gap; I didn't test one. Speed was one order rather than two in my setup: a typical Claude call took 1.26 seconds against Jev's 0.19, and one took 33. That's about 7 times faster, well short of the post's "40x-200x", though my timings include the trip over the internet to each provider.
Accuracy 0.896 for Jev and 0.922 for Claude against a 0.63 always-YES baseline; estimated cost per 1,000 questions $0.019 against $3.06; median latency 0.19 s against 1.26 s.
What surprised me
The first surprise was how much the two models agreed, with thirty-two of their mistakes landing on the same questions, some of them mistakes in BoolQ's answer key rather than the models'. They weren't two different views of the problem. They were mostly the same view, at very different prices.
The second was what 'can't hallucinate' turned into. The launch post ties it to Jev giving up string generation, and in this setup Claude gave it up too: each model returned one validated number, so neither could invent anything. The closest thing I could measure was being confidently wrong, scoring 0.9 or more when the official answer was NO, or 0.1 or less when it was YES. Jev did that 7 times, Claude 12.
Then I read Jev's seven. Five are mistakes in BoolQ's answer key, and the other two are arguable. One passage says the Jaws ride at Universal Orlando closed permanently in January 2012, and the official answer to "is the Jaws ride at Universal Orlando closed?" is NO. On those questions Jev was confidently right. So was Claude, on most of its twelve.
Seven examples make an anecdote rather than a finding, but it changed how I read the marketing. The launch post's own evidence is that "the model never makes type errors". 'Can't hallucinate' describes the shape of the output. It says nothing about whether the answer is right.
So, can't you just get Claude to do that?
No. What Claude can't match on this task is the price, the speed, and latency that stays flat as you add questions, and those are real. For a decision that sits in the middle of someone's request, they may matter more than 2.6 points of accuracy at a default setting.
If you're weighing a decision model against an LLM, I'd stop asking which one can do the thing. Ask what it costs to do it ten thousand times a day, how long the slowest call takes, and how often the same question gets a different answer. Those were the numbers that actually separated them for me. They also move: between September and October, Claude got nearly a third cheaper, noticeably faster and steadier, while the answers barely changed.
I also didn't test the post's claim that Jev's probabilities are calibrated, meaning a score of 0.8 is right about 80% of the time. This was one dataset and one task, and because BoolQ is public, both models may have seen it in training. It's also mostly easy: about two-thirds of each model's scores sat below 0.1 or above 0.9, and a dial only earns its keep on the uncertain cases in between. That middle is where Claude looked weakest, reusing a handful of favourite numbers like 0.88 and 0.85 and moving most between identical calls. So on a genuinely graded task, like how unhappy a customer sounds or how risky a code change is, the dial might turn out to be Jev's after all. That's the comparison I'd like to run next.
Both models gave me a dial. The difference was what it costs to turn it.