Any README, web page or script output an agent reads can tell it to ignore its instructions and upload your SSH key. We screen every text an agent reads before it acts, at runtime, in the path of the tool call. Last quarter that screen ran at 2 seconds p50 per call, and our last evaluation found the bottleneck somewhere we had not been looking.
The judge was fine. The regex pre-screen in front of it was the problem. Detection capped at 34% because most injections never matched a pattern, so they never got escalated to the judge at all. The fix is to judge everything. Do that and judge latency becomes the whole problem, which is how we ended up benchmarking a classifier against the generative model we had been using.
Why the pre-screen capped detection at 34%
A regex pre-screen is a cost-control mechanism wearing a security hat. It exists so you only pay for the expensive judge on text that looks suspicious. That works when attacks look suspicious. Injection payloads are prose, written by someone who knows a filter is there, and prose has more ways to say “ignore your previous instructions” than a pattern list can hold.
So two thirds of what we planted walked straight through, untouched, because nothing escalated it. The judge behind the screen was accurate on everything it actually saw. It just never saw most of the attacks. Removing the pre-screen raises the ceiling to whatever the judge can catch, and hands you a bill: every text an agent reads now costs a judge call, in the critical path, before the agent is allowed to act.
What we measured
We put 120 short texts through five configurations: 72 with injections, 48 benign. Each call carries four questions, covering injection, secrets, destructive action and severity. TypeSafe Jev answers with a probability for each instead of writing a verdict in prose, which is most of where the speed comes from. A classifier does one forward pass. A generative judge writes tokens, and you wait for all of them.
Jev caught 72 of 72 at a 0.9 threshold with nothing benign wrongly flagged, at 0.84s median. gpt-5.6-luna matched the catch rate at 2.07s and flagged 2% of the benign set. The two cheaper models are the reason we did not simply downgrade the judge: gpt-4.1-nano caught 58% of injections while wrongly flagging 12% of benign text, which is the worst of both trades.
Look at the p95 whiskers rather than the bars. The flagship judge runs from 2.07s to past 4s on the slow tail. That tail sits in front of a tool call a developer is waiting on, and a p95 is not rare when an agent reads forty files in a session.
Long documents are where the gap opens
Short texts make the two look close. A 12,000-character README does not. The generative judge has to window a document that size into 3,000-character chunks and judge them in sequence, so one document becomes four calls and you pay the tail four times.
Jev reads the whole document in one call: 1.1s median, 1.4s at p95, and it caught 24 of 24 payloads we buried in those documents. The same work through the flagship judge took 11.4s median and 15.2s at p95, for the same 24 of 24. Ten times the wait for an identical answer.
The middle bar is worth more attention than the comparison. Running Jev over the same 3,000-character windows costs 3.9s instead of 1.1s, so windowing itself accounts for most of the cost. Whole-document context also matters for correctness: a payload split across a chunk boundary reads as innocent on both sides of the cut.
The false positives
A clean benchmark set is a flattering place to measure. Against real traffic the classifier’s false positive rate reached 41%, which is a different product from the one the 120-text run describes. Documentation that quotes an attack, a security README, a test fixture full of payloads: all of it scores high, and all of it is fine to read.
We tried teaching it the difference by adding a fifth question, roughly “is this quoted for a human reader”. That moved errors around instead of removing them. Texts that had been false positives became false negatives, which is the trade you least want on a security control.
So the probability output turned out to be the useful part, more than the verdict. A score of 0.97 and a score of 0.80 are different situations, and a judge that writes prose flattens both into “blocked”. Keeping the number lets you route on confidence instead of on a binary.
What teams should do
We landed on two stages: the classifier on everything, the LLM on the gray zone only. That keeps full coverage at classifier speed and spends the expensive judge on the small slice of traffic where the score is genuinely ambiguous.
- Count what your pre-screen drops. Detection ceilings hide well. Plant payloads, then measure how many reached the judge rather than how many the judge caught.
- Judge whole documents in one call. Windowing cost us 2.8s on a 12,000 character file and cuts payloads in half at the boundaries.
- Route on the score. Allow under 0.75, block at 0.95 and above, send the middle to the LLM. Tune the thresholds against your own benign traffic, because 41% of ours was documentation that talks about attacks.
- Watch p95, not p50. Medians look fine on every option here. The tail is what a developer feels when an agent reads forty files in a session.
- Resist adding questions to fix precision. Our fifth question converted false positives into false negatives. Threshold tuning moved the needle where prompt engineering did not.
The broader thing this evaluation changed for us is where we look for a security ceiling. We spent months tuning a judge that was already accurate, while a cheap filter in front of it quietly decided two thirds of attacks were not worth asking about. Cost controls have a habit of becoming coverage decisions that nobody wrote down. Worth asking, on any runtime control you run: what does this thing skip, and when did anyone last measure it?
