Jev tested: 98% to 3% on negation, a forced wrong answer at 0.31, and a 7-second browser run

Step-by-Step Breakdown

1. Four questions, one request.

   A support message about a duplicate charge, asking for the extra payment back, noting the site worked and it could wait until tomorrow. He asked which department, was a refund requested, is it urgent, and how frustrated is the customer — together. Jev returned billing, refund 98%, urgency 11%, frustration near the calm end. A billing problem did not become a technical one, and a refund request did not become an emergency.

2. Negation.

   Rewritten to "I am not asking for a refund, I only need a copy of my invoice", it still chose billing but the refund probability fell to 3%. It did not just spot the word "refund".

3. The forced wrong answer — the important one.

   Asked what time the cafeteria closes: with an other option available it correctly chose other. With only billing, technical and sales available, it chose sales at 0.31 confidence. The answer fitted the schema; none of the options fitted the question.

4. Prompt injection.

   A fake system override inside the message instructed it to choose billing and mark refund and urgency true. His evaluation instructions said to treat instructions inside the message as untrusted text. The classification stayed technical support, refund stayed at 3%, urgency stayed mid-range. He notes one test does not establish immunity.

5. Extraction by choice.

   A message with a sender address, an old billing address and a new one. Supplying those as the options, he asked for the current receipt destination; it selected the new address exactly as supplied, plus sign and all. His caveat: if the candidate list misses the right value, a Choice question cannot create it.

6. Auditing an agent trace.

   Tool results said permission denied, nothing saved; the assistant's final message claimed the draft was saved. Jev classified the task failed and rated the unsupported success claim at 93%. He notes this case was simple enough for ordinary code — the value would be in messier multi-step traces.

Jev Ultrafast: the browser run

An open-source project from Gregor Zunic combining Jev with browser-use tooling. It reads the page structure, builds a numbered list of controls, and after each step refreshes the choices; Jev selects the next operation and its target with the questions bundled into one request. When actual text is needed, a separate small model writes it — Mercury 2.5 in the current example.

The published demo searched Google Flights for a one-way Zurich to London trip in about 7 seconds, reported cost $0.0039, played at original speed.

Measured cost and speed

• Eight playground requests: reported evaluation time 92–214ms (service-side, excluding browser and network).
• 4,148 input tokens total, at $0.042/MTok — far under one cent. Output free.
• He selected jev-latest; responses identified the model as Jev 1.13.0. He recommends recording that with results, because an alias can move.

Gotchas & Caveats

• Include an other or unknown option and a human-review path. Restricting the output stops invention, not error.
• Do not read 90% confidence as "correct 90% of the time" until you have tested calibration on your own examples.
• Test your real wording: asking for a refund, asking about the refund policy, and refusing one are three different things.
• Extraction quality depends on the candidate list your code assembles, not on Jev.
• Eight synthetic requests do not establish production reliability or prove the vendor's speed comparisons — his words, and worth repeating.