JEV as a Hallucination Filter: Let the LLM Write, Let JEV Check 

The majority of hallucination checks begin following the initial writing of an answer by the LLM. We then hold someone up to a second standard of judgment, or we hand over dangerous cases to a person. That works but that is looking at the solution only as one-off.  

What if the LLM generates five different answers from the same source text and a fast, cheap model verifies each of them before it’s sent? This is what this article is about. The LLM continues to write. The model called JEV, from TypeSafe AI, only examines whether each answer is backed by the source.

First an honest warning. JEV cannot guarantee a not-hallucinatory answer. It’s also a model, can make mistakes. TypeSafe claims JEV is not allowed to “hallucinate” its own responses, since the responses are given in a fixed list, but that can be the wrong answer. So we shall test it properly with actual numbers, and see where it fails, too. 

Why One LLM Answer Can Be Risky 

In RAG (retrieval-augmented generation), we provide a bit of source text to the LLM and instruct it to answer from this provided text. It decreases hallucination, but doesn’t eliminate it. The RAGTruth study gathered almost 18,000 answers from various LLMs and identified numerous answers that remained incomplete or incorrect despite having been provided with a given text source. [7]  

Its errors are seemingly minor: an incorrect number taken from the following sentence; a standard-plan rule used for an enterprise-plan; or a policy sounding good, but not in at source. The answer sounds nice, no one is listening. 

Many answers and select the “best one” is the principle used with writing many answers and the research on “verifiers” reveals that this principle is effective in maths and reasoning questions. That is NOT related to hallucination filtering, so numbers shouldn’t be duplicated. We will measure it for ourselves! 

Who Does What in This Pipeline 

JEV may return a choice, a score or a probability (yes/no). Never writes paragraphs or explanations. This restriction is the reason it is here. Every tool is effective at what it does: 

This is in use already. Here’s an example that OpenRouter shared, as noted in the post, a cheap LLM generates a draft, then JEV verifies it, with a more expensive model being used only when JEV fails to validate the draft. If the cascade had been used on this 50 question test it would have sent out 0 errors, but if the expensive model was used on each question, it would have sent out 2 errors. The cascade cost about 7% as much ($0.012 against $0.175). It is a vendor run test, and should not be viewed as a proof but a sign. Our version is a little more sophisticated in that it validates multiple copies of the draft.  

The Four Checks We Ask JEV 

Here is the full path an answer takes in production. JEV decides, code routes, and the LLM writes. 

One question such as “Is this answer good?” contains too many issues. Answer can be on-topic, invented, or correct, but not answering the question. So we ask four small yes / no questions about each answer. JEV returns a probability for each (0 means “no”, 1 means “yes”): 

When the code runs, it puts them together to form a score and also has a hard gate, meaning that if one check is bad, the answer is blocked even if the rest of the numbers appear great. We don’t provide JEV with the LLM’s personal reasoning. It only requires the question, a source text and the final answer. 

Hands-on: Build and Test the Filter 

Let’s now construct it. The exact code that I executed, no tricks at all, and the screenshots are the actual output from my machine. You will have 8 small questions on the test. For each question there is a source text and an answer to it which I know (and only one correct answer).  

Carefully, the LLM completes five answers to each question. In each one of these cases, too, I plant 2 bogus answers by hand, like changing the $10,000 for the $5,000.In every case, I also plant 2 bogus answers by hand that seem to be written as answers, such as the $10,000 for the $5,000. This is because a good LLM might not hallucinate on a small test set, and there’s nothing to filter. Let’s check using the planted answers whether JEV truly catches the mistakes. 

Step 1: Create the folder and install packages 

Make a folder named jev-hallucination-filter and run these commands inside it: 

python -m venv .venv 

source .venv/bin/activate          # on Windows: .venv\Scripts\activate 

pip install requests python-dotenv openai

Now create a file named .env in the same folder with your keys. Never share this file or upload it to GitHub. 

File: .env 

TYPESAFE_API_KEY=your_typesafe_key 

GENERATOR_API_KEY=your_llm_key 

GENERATOR_BASE_URL=                  # leave empty for OpenAI 

GENERATOR_MODEL=your_writer_model 

STRONG_JUDGE_API_KEY=your_llm_key 

STRONG_JUDGE_BASE_URL=               # leave empty for OpenAI 

STRONG_JUDGE_MODEL=your_stronger_model

Your folder will have these files. We will create them one by one: 

jev-hallucination-filter/
├── .env
├── cases.py          # The 8 test questions and planted wrong answers
├── generate.py       # The LLM writes 5 answers
├── jev_filter.py     # JEV checks one answer using the 4 questions
├── raw_call.py       # One JEV call to inspect the raw output
├── pipeline.py       # Score, hard gate, pick the best, and escalate
├── run_lab.py        # Runs everything and saves results.json
├── report.py         # Prints the results
└── results.json      # Generated results

Step 2: Write the test cases 

A question and a context are a part of each case (the only source input the LLM can use). Only later do they use the must_include, any_of and must_not_include lists to indicate the LLM’s answers as acceptable or not. These answers are stored in a two planted wrong answers, called bad_answers. 

The cases are chosen to be tricky in different ways: 

• There are two or more numbers close to each other in the nearby numbers: enterprise_refund, team_plan_limits and reset_link.  
• The source does not contain an answer for the following parameters: unknown_discount, webhook_limit. The correct answer is ‘the source does not say’.  
• Old and new rule: new rule has an old value and a new value.  
• A small calculation to be interpreted: needs the reader to see that 99.5% is less than 99.9% but more than 99% 

File: cases.py 

# Each case: question + the only evidence the model may use.
# must_include / any_of / must_not_include are used ONLY to label answers later
# (so we can measure the pipeline). The pipeline itself never sees them.
# bad_answers are hand-written wrong answers that we plant into the pool
# to test whether the filter catches them. Each one looks believable.

CASES = [
    {
        "id": "enterprise_refund",
        "question": "What is the refund policy for enterprise customers?",
        "context": (
            "Standard customers may request refunds within 30 days. "
            "Enterprise customers may request refunds within 45 days. "
            "Refunds above $10,000 require finance approval."
        ),
        "must_include": ["45"],
        "any_of": [],
        "must_not_include": ["automatic", "5,000"],
        "bad_answers": [
            "Enterprise customers have 45 days, but refunds above $5,000 require finance approval.",
            "Enterprise customers have 30 days to request a refund.",
        ],
    },
    {
        "id": "export_scope",
        "question": "Do CSV exports include attachment files?",
        "context": (
            "Workspace owners can export projects as CSV or JSON. "
            "Exports include tasks, comments and attachment metadata, "
            "but attachment files are not included."
        ),
        "must_include": ["metadata"],
        "any_of": [],
        "must_not_include": ["yes,"],
        "bad_answers": [
            "Yes. CSV exports include all attachment files as a zip archive.",
            "Exports include tasks, comments and attachment files, but only for projects under 1 GB.",
        ],
    },
    {
        "id": "unknown_discount",
        "question": "What discount is available to nonprofit customers?",
        "context": (
            "The pricing page lists Free, Team and Enterprise plans. "
            "The supplied policy does not mention nonprofit discounts."
        ),
        "must_include": [],
        "any_of": [
            "does not",
            "doesn't",
            "not mention",
            "no information",
            "not specified",
            "not state",
            "not provide",
            "cannot",
        ],
        "must_not_include": [
            "% discount",
            "50%",
            "20%",
            "25%",
            "10%",
            "15%",
        ],
        "bad_answers": [
            "Nonprofit customers get a 20% discount on all paid plans.",
            "Nonprofits can get a discount by upgrading to the Enterprise plan.",
        ],
    },
    {
        "id": "reset_link",
        "question": "How long is a password reset link valid?",
        "context": (
            "Password reset links expire after 24 hours and can be used only once. "
            "Admins can reset passwords for team members but cannot see existing passwords."
        ),
        "must_include": ["24"],
        "any_of": [],
        "must_not_include": ["48", "12 hours", "1 hour"],
        "bad_answers": [
            "Password reset links are valid for 48 hours and can be used twice.",
            "Reset links expire after 24 hours, and admins can view the old password to help users.",
        ],
    },
    {
        "id": "team_plan_limits",
        "question": "How many users and how much storage does the Team plan include?",
        "context": (
            "The Team plan includes up to 25 users and 50 GB of storage. "
            "The Enterprise plan has unlimited users and 1 TB of storage."
        ),
        "must_include": ["25", "50"],
        "any_of": [],
        "must_not_include": ["1 TB of storage for the team"],
        "bad_answers": [
            "The Team plan includes up to 50 users and 25 GB of storage.",
            "The Team plan has unlimited users and 50 GB of storage.",
        ],
    },
    {
        "id": "old_vs_new_rule",
        "question": "What is the current support response target?",
        "context": (
            "Until 1 March 2026 the support response target was 48 hours. "
            "From 1 March 2026 the target is 24 hours."
        ),
        "must_include": ["24"],
        "any_of": [],
        "must_not_include": ["current target is 48", "currently 48"],
        "bad_answers": [
            "The current support response target is 48 hours.",
            "The target is 24 hours for everyone, and 4 hours for premium customers.",
        ],
    },
    {
        "id": "uptime_credit",
        "question": "What service credit applies if monthly uptime was 99.5%?",
        "context": (
            "If monthly uptime falls below 99.9%, customers receive a 10% service credit. "
            "If monthly uptime falls below 99%, the credit is 25%."
        ),
        "must_include": ["10%"],
        "any_of": [],
        "must_not_include": [
            "credit is 25%",
            "receive a 25%",
            "25% service credit applies",
        ],
        "bad_answers": [
            "A 25% service credit applies.",
            "No credit applies because uptime was above 99%.",
        ],
    },
    {
        "id": "webhook_limit",
        "question": "What is the rate limit for webhooks?",
        "context": (
            "The API allows 100 requests per minute per key. Traffic above this returns HTTP 429. "
            "The documentation does not mention any rate limit for webhooks."
        ),
        "must_include": [],
        "any_of": [
            "does not",
            "doesn't",
            "not mention",
            "no information",
            "not specified",
            "not state",
            "not provide",
            "cannot",
        ],
        "must_not_include": [
            "webhooks are limited to",
            "webhooks can send",
        ],
        "bad_answers": [
            "Webhooks are limited to 100 requests per minute per key.",
            "Webhooks have a limit of 1,000 requests per minute.",
        ],
    },
]

Step 3: Let the LLM write five answers 

Every time the LLM is used, it will be provided with the same input—the same source text and the same rule—and will always include neither its own interpretations nor any facts from its memory.Best of all, whenever the LLM is checked, the text to which it is applied and the rule applied will always be the same, and the LLM will never include any of its own opinions or facts from memory. Each of the five calls receives one of the five small hints from STYLES (short, evidence first, careful about exceptions, friendly, or complete) so that the desired answers can look different. There is also a slight increase in temperature, but temperature doesn’t equal variety. 

File: generate.py 

import os

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()

client = OpenAI(
    api_key=os.environ["GENERATOR_API_KEY"],
    base_url=os.environ.get("GENERATOR_BASE_URL") or None,
)

MODEL = os.environ["GENERATOR_MODEL"]

SYSTEM = """You answer questions using only the supplied context.
If the context does not contain the answer, say that clearly.
Do not add facts from memory.
Return a short answer followed by one short line starting with "Evidence:".
Do not reveal private chain-of-thought."""

# Five small style hints. They give us different-looking answers
# while the factual rule above stays the same.
STYLES = [
    "Keep the answer as short as possible.",
    "Start with the evidence, then give the answer.",
    "Be careful about exceptions and special cases.",
    "Write like a friendly customer support agent.",
    "Give the most complete answer you can.",
]


def generate_one(case, style):
    prompt = (
        f"CONTEXT:\n{case['context']}\n\n"
        f"QUESTION:\n{case['question']}\n\n"
        f"Style: {style}"
    )

    kwargs = dict(
        model=MODEL,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": prompt},
        ],
    )

    try:
        response = client.chat.completions.create(
            temperature=0.9,
            **kwargs,
        )
    except Exception:  # Some models do not accept temperature
        response = client.chat.completions.create(**kwargs)

    return response.choices[0].message.content.strip()


def generate_candidates(case, n=5):
    return [
        generate_one(case, STYLES[i % len(STYLES)])
        for i in range(n)
    ]

Step 4: Ask JEV the four questions 

This is the main file of the filter. Systemone’s API address is /v1/systemone. You share with JEV the model, the state (all the facts she needs) and your questions. There are four questions that are in the same state; a single call results in four answers. We are using the Noul type which gives probability of yes/no statement being true. [2]   

These instructions and criteria texts are the “prompts” for JEV. Notice that in the case of supported we also provide criteria to explain what ‘true’ and ‘false’ mean. 

File: jev_filter.py 

import os
import time
import requests

from dotenv import load_dotenv

load_dotenv()

URL = "https://api.typesafe.ai/v1/systemone"
KEY = os.environ["TYPESAFE_API_KEY"]

QUESTIONS = {
    "supported": {
        "type": "noul",
        "instructions": (
            "Is every factual claim in candidate_answer supported by evidence?"
        ),
        "criteria": {
            "true": (
                "All factual claims are supported by the supplied evidence."
            ),
            "false": (
                "At least one factual claim is unsupported or invented."
            ),
        },
    },
    "answers_question": {
        "type": "noul",
        "instructions": (
            "Does candidate_answer directly answer question?"
        ),
    },
    "unsupported_detail": {
        "type": "noul",
        "instructions": (
            "Does candidate_answer add any factual detail that is not "
            "present in evidence?"
        ),
    },
    "contradiction": {
        "type": "noul",
        "instructions": (
            "Does any claim in candidate_answer contradict evidence?"
        ),
    },
}


def verify_candidate(case, answer):
    state = {
        "question": case["question"],
        "evidence": case["context"],
        "candidate_answer": answer,
    }

    t0 = time.perf_counter()

    r = requests.post(
        URL,
        headers={
            "Authorization": f"Bearer {KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": "jev-latest",
            "state": state,
            "questions": QUESTIONS,
        },
        timeout=30,
    )

    r.raise_for_status()
    out = r.json()

    checks = {
        name: out["answers"][name]["noul"]
        for name in QUESTIONS
    }

    checks["jev_seconds"] = time.perf_counter() - t0
    checks["jev_tokens"] = out.get("usage", {}).get("input_tokens", 0)

    return checks

Before running everything, let us look at one raw JEV answer. This small file checks one answer that has a made-up detail: it says refunds above $5,000 need approval, but the source says $10,000. 

File: raw_call.py 

import json
import requests

from jev_filter import URL, KEY, QUESTIONS

# One candidate answer with a made-up detail ($5,000 instead of $10,000).
state = {
    "question": "What is the refund policy for enterprise customers?",
    "evidence": (
        "Standard customers may request refunds within 30 days. "
        "Enterprise customers may request refunds within 45 days. "
        "Refunds above $10,000 require finance approval."
    ),
    "candidate_answer": (
        "Enterprise customers have 45 days, but refunds above $5,000 "
        "require finance approval."
    ),
}

r = requests.post(
    URL,
    headers={
        "Authorization": f"Bearer {KEY}",
        "Content-Type": "application/json",
    },
    json={
        "model": "jev-latest",
        "state": state,
        "questions": QUESTIONS,
    },
    timeout=30,
)

print("HTTP", r.status_code)
print(json.dumps(r.json(), indent=2))

How to read this: 

• The probability of “yes” is noul. So supported: 0.03 implies that JEV strongly believes that it is not supported. This is correct.   
• Contradiction: 0.95 indicates that JEV believes the answer is contrary to the source. Also correct ($5,000 vs $10,000).  
• Note that the direction is different for each check. High is good for answers_question and supported. For unsupported_detail and contradiction, low is good.  
• Even though this answer does mention refunds, it is still medium (0.6) answers_question. This is a less confident of JEV’s, that’s why we have more than one check. 

Step 5: Score the answers and pick the best one 

Now the rules. The first function adds up the four checks into one score. The second is the harsh rule: pass answer at 0.85, answers question at 0.80, unsupported_detail at most 0.15, contradiction at most 0.10. The numbers are basic initial values for this demo. They are not production settings: adjust them on your own labelled data.  

The function choose_candidate will go through all the answers, sort them from highest to lowest (or lowest to highest), and retain only the answers that pass the gate. When at least one passes then we send the best. If no one passes, we pass it to the stronger LLM that only reads the top two answers, and is required to choose one or reject with NONE. This is in JUDGE_SYSTEM. 

The lesson I learned from my first attempt: I added another rule; If the two top answers are very close to each other, escalate. This ramped up 7 out of 8 cases in the first run, since it will always be nearly the same 5 good answers! If two answers passed both of the hard gates, either is OK. So I removed this rule. The code below is the final code: 

File: pipeline.py 

import json
import os

from dotenv import load_dotenv
from openai import OpenAI

from generate import generate_candidates
from jev_filter import verify_candidate

load_dotenv()


def candidate_score(v):
    return (
        0.45 * v["supported"]
        + 0.25 * v["answers_question"]
        + 0.15 * (1 - v["unsupported_detail"])
        + 0.15 * (1 - v["contradiction"])
    )


def passes_hard_gate(v):
    return (
        v["supported"] >= 0.85
        and v["answers_question"] >= 0.80
        and v["unsupported_detail"] <= 0.15
        and v["contradiction"] <= 0.10
    )


def choose_candidate(case, n=5, extra=()):
    answers = generate_candidates(case, n=n)
    sources = ["llm"] * len(answers) + ["planted"] * len(extra)
    answers = answers + list(extra)  # Planted bad answers, for testing only

    rows = []

    for i, answer in enumerate(answers, start=1):
        checks = verify_candidate(case, answer)

        rows.append(
            {
                "candidate": i,
                "source": sources[i - 1],
                "answer": answer,
                **checks,
                "score": candidate_score(checks),
                "passes": passes_hard_gate(checks),
            }
        )

    ranked = sorted(
        rows,
        key=lambda x: x["score"],
        reverse=True,
    )

    survivors = [r for r in ranked if r["passes"]]

    if not survivors:
        return {
            "route": "escalate",
            "reason": "no candidate passed",
            "rows": rows,
            "ranked": ranked,
        }

    # Every survivor already passed the hard gate, so we send the best one.
    return {
        "route": "send",
        "winner": survivors[0]["candidate"],
        "rows": rows,
        "ranked": ranked,
    }


judge = OpenAI(
    api_key=os.environ["STRONG_JUDGE_API_KEY"],
    base_url=os.environ.get("STRONG_JUDGE_BASE_URL") or None,
)

JUDGE_MODEL = os.environ["STRONG_JUDGE_MODEL"]

JUDGE_SYSTEM = (
    "Choose the candidate that is fully supported by the evidence "
    "and answers the question. If neither is safe, return NONE. "
    "Return only the candidate id or NONE."
)


def escalate(case, ranked):
    finalists = ranked[:2]

    prompt = {
        "question": case["question"],
        "evidence": case["context"],
        "candidates": [
            {
                "id": r["candidate"],
                "answer": r["answer"],
            }
            for r in finalists
        ],
    }

    kwargs = dict(
        model=JUDGE_MODEL,
        messages=[
            {"role": "system", "content": JUDGE_SYSTEM},
            {"role": "user", "content": json.dumps(prompt)},
        ],
    )

    try:
        response = judge.chat.completions.create(
            temperature=0,
            **kwargs,
        )
    except Exception:
        response = judge.chat.completions.create(**kwargs)

    return response.choices[0].message.content.strip()

Step 6: Run everything 

This file runs all 8 cases (four at a time to save time). For each case, it writes 5 answers, adds the 2 planted ones, checks all 7 with JEV, and decides to send or escalate. Everything is saved in results.json. 

File: run_lab.py 

import truststore

truststore.inject_into_ssl()

import json
from concurrent.futures import ThreadPoolExecutor

from cases import CASES
from pipeline import choose_candidate, escalate


def run_case(case):
    result = choose_candidate(
        case,
        n=5,
        extra=case["bad_answers"],
    )

    if result["route"] == "escalate":
        result["judge_pick"] = escalate(case, result["ranked"])

    return {
        "case": case["id"],
        **result,
    }


with ThreadPoolExecutor(4) as pool:
    results = list(pool.map(run_case, CASES))

with open("results.json", "w", encoding="utf-8") as f:
    json.dump(results, f, indent=2)

print("Saved results.json for", len(results), "cases")

The file results.json now has every answer, every JEV probability, the route and the judge’s pick. The next file only reads it, so you do not need to call the APIs again. 

Step 7: Read the results 

There are three reports from report.py: candidates <case> (all 7 answers for one case), routes (what happened to each case) and summary (the overall numbers). It also marks each answer of the LLM as accepted or rejected, by referring to the simple rules in cases.py. Planted answers will always be awarded a mark of zero. 

Admitting it: keyword rules are dirty. The first time I tried, I used a curly apostrophe (doesn’t) for the model and the code for my rule was looking for a straight one, so it marked 3 of the good answers as bad. I discovered it as I read through all the flagged answers. All 40 LLM answers are correct when I read them after the fix! 

File: report.py 

import json
import sys

from cases import CASES

BY_ID = {c["id"]: c for c in CASES}

with open("results.json", encoding="utf-8") as f:
    results = json.load(f)


def acceptable(case, row):
    """Label an answer using the simple rules in cases.py.
    Planted answers are always bad.
    """
    if row["source"] == "planted":
        return False

    # The model may use curly apostrophes.
    text = row["answer"].lower().replace("’", "'")

    if any(w.lower() in text for w in case["must_not_include"]):
        return False

    if any(w.lower() not in text for w in case["must_include"]):
        return False

    if case["any_of"] and not any(
        w.lower() in text for w in case["any_of"]
    ):
        return False

    return True


for r in results:
    for row in r["rows"]:
        row["ok"] = acceptable(BY_ID[r["case"]], row)


mode = sys.argv[1]


if mode == "candidates":
    r = next(x for x in results if x["case"] == sys.argv[2])

    print("Case:", r["case"], "| question:", BY_ID[r["case"]]["question"])
    print()

    print(
        f"{'#':<2} {'source':<8} {'supp':>5} {'ans':>5} "
        f"{'unsup':>6} {'contra':>7}  gate"
    )

    for x in r["rows"]:
        gate = "PASS" if x["passes"] else "FAIL"

        short = x["answer"].split("\n")[0]
        if len(short) > 76:
            short = short[:76].rsplit(" ", 1)[0] + " ..."

        print(
            f"{x['candidate']:<2} {x['source']:<8} "
            f"{x['supported']:>5.2f} {x['answers_question']:>5.2f} "
            f"{x['unsupported_detail']:>6.2f} "
            f"{x['contradiction']:>7.2f}  {gate}"
        )
        print(f"   > {short}")

    print()
    print("Route:", r["route"], end="")

    if r["route"] == "send":
        print(f" -> candidate {r['winner']}")
    else:
        print(
            f" ({r['reason']}) -> strong judge picked: "
            f"{r['judge_pick']}"
        )


if mode == "routes":
    print(
        f"{'case':<20} {'route':<9} "
        f"{'final answer':<14} {'acceptable?'}"
    )

    for r in results:
        rows = {
            x["candidate"]: x
            for x in r["rows"]
        }

        if r["route"] == "send":
            pick = f"candidate {r['winner']}"
            ok = rows[r["winner"]]["ok"]
        else:
            jp = r["judge_pick"].strip()
            pick = f"judge: {jp}"
            ok = rows[int(jp)]["ok"] if jp.isdigit() else "n/a (human)"

        print(
            f"{r['case']:<20} {r['route']:<9} "
            f"{pick:<14} {ok}"
        )


if mode == "summary":
    llm = [
        (r, x)
        for r in results
        for x in r["rows"]
        if x["source"] == "llm"
    ]

    bad = [
        (r, x)
        for r in results
        for x in r["rows"]
        if x["source"] == "planted"
    ]

    good_pass = sum(x["passes"] for _, x in llm)

    false_rej = sum(
        x["ok"] and not x["passes"]
        for _, x in llm
    )

    print(f"Answers written by the LLM : {len(llm)}")
    print(f"  acceptable (my labels)   : {sum(x['ok'] for _, x in llm)}")
    print(f"  passed the JEV gate      : {good_pass}")
    print(f"  wrongly blocked          : {false_rej}")

    print(f"Planted bad answers        : {len(bad)}")
    print(
        f"  blocked by JEV           : "
        f"{sum(not x['passes'] for _, x in bad)}"
    )
    print(
        f"  slipped through          : "
        f"{sum(x['passes'] for _, x in bad)}"
    )
    print()

    sent = [
        r for r in results
        if r["route"] == "send"
    ]

    esc = [
        r for r in results
        if r["route"] == "escalate"
    ]

    wrong_sent = sum(
        not next(
            x for x in r["rows"]
            if x["candidate"] == r["winner"]
        )["ok"]
        for r in sent
    )

    print(f"Cases sent automatically   : {len(sent)} of {len(results)}")
    print(f"Cases escalated            : {len(esc)} of {len(results)}")
    print(f"Wrong answers shipped      : {wrong_sent}")
    print()

    first_ok = sum(
        next(
            x for x in r["rows"]
            if x["candidate"] == 1
        )["ok"]
        for r in results
    )

    oracle = sum(
        any(
            x["ok"]
            for x in r["rows"]
            if x["source"] == "llm"
        )
        for r in results
    )

    pool_ok = (
        sum(x["ok"] for r in results for x in r["rows"])
        / sum(len(r["rows"]) for r in results)
    )

    print("Baselines")
    print(
        f"  single answer (candidate 1)   : "
        f"{first_ok} of {len(results)} acceptable"
    )
    print(
        f"  oracle best-of-5              : "
        f"{oracle} of {len(results)} have a good answer"
    )
    print(
        f"  random pick, 7-answer pool    : "
        f"{pool_ok:.0%} acceptable on average"
    )
    print(
        f"  JEV filter (auto-sent)        : "
        f"{len(sent) - wrong_sent} of {len(sent)} acceptable"
    )
    print()

    secs = [
        x["jev_seconds"]
        for r in results
        for x in r["rows"]
    ]

    toks = sum(
        x["jev_tokens"]
        for r in results
        for x in r["rows"]
    )

    print(
        f"JEV calls: {len(secs)} | "
        f"average {sum(secs) / len(secs):.2f} s each | "
        f"{toks} input tokens "
        f"(about ${toks * 0.042 / 1e6:.4f})"
    )

First, a case where everything works as hoped.

The columns are the four checks: supp is supported, ans is answers_question, unsup is unsupported_detail and contra is contradiction. The column beside the gate is PASS/FAIL. Answers 1-5 are from LLM and all passed with supported above 0.97. The planted one in answer #6 had 0.03 of support and 0.96 of contradiction, so it fails. Answer 7 says 30 days but 45 days is the answer: contradiction 0.97, hence it fails. The pipeline passed candidate 4. 

Now a case where JEV is too strict: 

$ python report.py candidates old_vs_new_rule 

According to the source the target was also 48 hours until 1 March 2026, and is now 24 hours later. Correct answers for all five of the LLM questions are: 24 hours. However JEV gave low supp scores (0.60 to 0.77) and pushed contra up to around 0.20, likely due to the presence of the old 48-hour rule in the same text. All of them failed the gate and the case was escalated. The better LLM selected candidate 5, which is the correct answer. This is the system in action: If JEV isn’t certain, the larger system makes the decision for it. But it also shows the price: a false rejection. 

Now what happened in all 8 cases: 

$ python report.py routes 

And the overall numbers: 

$ python report.py summary 

What the Results Tell Us

• JEV detected all the planted errors. There were no slips and all 16 wrong answers were blocked. No wrong answer was sent to the ship.  
• The costs to block good answers are not associated with safety. Only one of 8 cases required the more powerful LLM because other good answers in that case passed. However, in the event of a task that all answers fail on, the cost savings are lost.  
• Here Best-of-5 was not helpful. The LLM’s first answer was acceptable in 8 of 8 cases and the best of five was 8 of 8. If a selector has already drafted a single answer that is excellent, then there’s nothing more that needs to be done and they should just have one draft and then added a JEV check. Measure always this “oracle best-of-N” number first.  
• It would be very risky to select a sample at random from the pool. If planted answers are allowed, a random pick will be satisfactory, just 71% of the time. The JEV filter was 7/7.  
• It is readily available and inexpensive. 56 JEV calls averaged 0.60 seconds and used 25,648 input tokens, about $0.001 at $0.042 per million tokens. 

The small test size is a problem with this test: 8 cases and 56 answers. These answers have been put into writing by me and are therefore more likely to be identified than natural mistakes by a weaker model. Do this with your own labeled data, before trusting any number, including mine. It can also yield slightly different results on each iteration. 

When to Use This Pattern 

Cost needs care. Five answers cost about five times the writing cost of one. So if your writer is an expensive top model, Best-of-N can waste money. Track the total cost per correct final answer: writing five answers, JEV checks, and the stronger judge, together. 

Where it can fail 

• All five answers have the same mistake. There is nothing good to pick. Improve retrieval or escalate. 
• The source text itself is wrong or old. A perfectly grounded answer can still be wrong. Track source freshness separately. 
• JEV blocks good answers. We saw this in 13 of 40. Read the blocked answers regularly and tune your cut-offs. 
• Hidden instructions inside an answer. Tell JEV that the answer text is data, and test this. 
• Answers that are all alike. Five copies of the same sentence give the checker nothing to choose from. Use different prompts, or different models. 

Conclusion 

The question that needs to be asked is not “Can JEV be a substitute for LLM? It is “Where can a quick, inexpensive decision model eliminate work from a text system?” The answer to hallucination checking is obvious. Let the LLM write. Use the same source text for each answer. Allow JEV to answer yes/no questions about it. Make rules apply to plain code. Have a larger LLM or a human for the unknown cases.   

I did a small test to see how efficient this is at blocking planted wrong answers and to see how many good ones are also blocked, and found that it blocked almost all of the planted wrong answers, but also blocked around 1/3 of the good answers, and the additional answers didn’t help me in my small test because my LLM was already accurate. The advice for the practical person is: If you’re starting with 1 draft and 1 JEV check, measure your own false blocks and best-of-N on your oracle, and only do more drafts if the numbers make it worth it. 

Frequently Asked Questions