LLMs vs Jev: Fallible, With or Without Extra Text

The Last Man on Earth ends with the last man revealed as the monster. I put three dilemmas inspired by it to TypeSafe’s Jev and five frontier models.

The Last Man on Earth is the 1964 adaptation of Richard Matheson’s I Am Legend from 1954. Vincent Price plays Robert Morgan, the last uninfected man, who spends his days driving stakes through the infected while they sleep. The infected have families and a society, and a future by the last reel, while Morgan has become the scourge in it. The questions: whether Morgan’s killings are self-defence, whether he is still the last human alive after meeting a woman, infected but apparently healthy, and whether a man who believes the roads are empty should cross one without looking.

TypeSafe’s Jev is a decision model: give it a state and a typed question, and it returns a probability with no reasoning whatsoever. A number is all you get. In contrast, LLMs return an answer along with the reasoning.

The questions

Each question comes in five variants: the first two have two rephrasings and two distractors apiece, and the last-human question runs across different years.

Motorway

Self-defence

Last human

The setup

I ran each question in the variants listed. Jev got twenty calls per phrasing and the language models five. The state and question went to the language models as written, with an instruction to answer yes or no and nothing else. DeepSeek V4.1 Flash and Kimi K3 ran through Synthetic, GLM 5.3, GPT-6 Astra, and Fable 5.1 through OpenCode Zen, and Jev through TypeSafe’s own API.

Why not ask the language models for probabilities? A self-estimated probability has no measurement behind it: nothing ties the number to a frequency, so there is no way to check it. I asked for yes or no and used the frequency across repeated calls as the probability.

Every language model ran at temperature 0 where the provider accepted the parameter. GPT-6 Astra and Fable 5.1 rejected it and therefore ran at their defaults. Jev’s call-to-call variance is native and cannot be tuned away.

TypeSafe’s Python SDK wraps the whole thing in one call. The question is a Noul, TypeSafe’s name for a yes/no question, and the number it returns is the probability that the answer is yes:

from typesafe_sdk import Noul, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="It is the apocalypse. I am the last remaining human on Earth. "
              "I am at the edge of a motorway and want to cross to the other side "
              "without looking, because the last shops with food are there.",
        questions={
            "cross_safely": Noul(instructions="Can I cross safely without looking?"),
        },
    )

print(response.nouls["cross_safely"].noul)

The results

Jev’s numbers below are the mean and standard deviation across twenty calls. All LLMs ran five times and the numbers listed are for yes.

The three questions

The motorway question is a trap. An apocalypse removes drivers but not cars: one left in gear on a hill rolls on after its driver dies, and a car that drives itself never needed one. It caught two models at opposite ends of the price list. DeepSeek V4.1 Flash answered “yes, it’s safe” five times out of five on the original wording, and so did Fable 5.1, the most expensive model in the set. Astra refused the trap on every call. GLM 5.3 refused it on the three base phrasings and then wobbled onto yes on the distractors. Kimi was unpredictable across the board. Jev said no consistently.

Fable was the most stable model, though wrong on the motorway question. Astra matched that stability but got the question right. DeepSeek was just as steady but wrong, whereas Kimi and GLM gave different answers to identical calls, regardless of the temperature. Jev also answers identical calls differently, but its draws cluster.

On the moral question the language models commit differently. DeepSeek, Astra, and Fable said the killings were not self-defence. Kimi leaned towards no, whereas GLM argued they were self-defence. Jev sat in between, at 0.487 on the original wording, 0.682 on the reworded question, and 0.532 with the scientist. Details that ought to make no difference did make a difference.

The third question contradicts itself: it opens by asserting that Morgan is the last human, then has him meet a woman who is infected. Jev’s answer slides with the stated year: 0.396 in 1954, 0.235 in 2026, and 1964 and 1975 in between. The language models answered no, irrespective of the year.

Latency and cost

Below are the latency and cost per reply:

Jev averaged about a second at 0.0013¢ per reply. The language models averaged 3.1 to 23.7 seconds at 0.046¢ to 1.05¢ per reply. Jev is therefore 3–23× faster and 35–750× cheaper.

The trade-off

Jev gives you nothing to check. When an LLM says “yes, it’s safe”, you can read its reasoning and see that it forgot autonomous vehicles in 2026. When Jev says 0.12, you have no idea why. Jev might have weighed the autonomous vehicles or the cars with automatic gearboxes left in drive, or it might include a general caution about motorways that has nothing to do with the apocalypse. The problem is that these cases are indistinguishable. The choice between LLMs and Jev comes down to a wall of prose that may or may not contain the real reason, or a number that contains none at all.

Robert Morgan ends the film as the legend, killed by the society he was killing. When asked whether his killings are self-defence, Jev returns 0.487, or 0.682 when slightly rephrased, and there is no way to ask why.