An analyst types a question in plain English and the product has to decide which of the tenant's dozens of connected data sources to search. A connector delivers a log type nobody has seen before and something has to say which of our known types it is. A vendor field called actor.alternateId arrives and has to land in the right slot of a schema. An alert fires and someone has to decide whether it deserves an investigation or is the same backup job we dismissed 400 times this week.
Each of these is "look at this, pick one of these", and the options are known before the question is asked. Today each is an LLM call, some of them an agent holding tools, and every one costs seconds and cents.
Then a model came out that answers exactly this shape of question in one call, with a probability instead of a paragraph. We got access and tried it on a few of the places in our product where it would fit. This post is about one, alert triage, because every SecOps team has that problem and will recognize it.
TLDR (but stay for the side by side): Jev fits two kinds of tasks: where someone waits on the answer, and where there are too many items to pay an LLM for each. We put it in front of our alert triage agent on two tenants. As a gate it closed 15% of the agent's alerts on the busy tenant and a third on the quiet one, right about 98% of the time by three LLM judges whose labels our security analysts then verified and corrected by hand, in one cheap call each. We also checked whether it could decide outright, which was never the goal. It cannot, and the tenant's own history is what made the gate work.
What Jev Actually Is
An agent runs in a loop. The model decides what to do, a tool runs, the model reads the result, and around it goes. Every one of those decisions is another LLM round trip, and most of them are small. You pay LLM latency and LLM prices for a decision that would fit on an index card.
Jev, from TypeSafe AI, is built for those decisions. It is not an LLM and it does not generate text. You hand it a state, your data as text or JSON, and a fixed set of typed questions, and it answers all of them in one pass with a probability on each. TypeSafe calls this a System One model, after the fast intuitive half of Kahneman's two systems, and trains it with reinforcement learning for calibrated decisions, so the probabilities are meant to be real probabilities.
The analogy I came up with when trying to explain it to the boomers at a family dinner. You have a question for a very smart person. With an LLM you ask it and hand them a blank page to answer on, and they write you a paragraph. With Jev you ask the same person the same question, but you hand them a bubble sheet. Back comes the sheet with one bubble much darker than the rest, and how dark it is tells you how sure they are.
Three question types exist:
- Choice: pick one of up to 255 options, and get a probability for every option.
- Noul: is this statement true, and get P(yes).
- Score: rate this against an ordered rubric, and get a probability per level.
resp = client.system_one(
state={"email": "...", "sender_history": "..."},
questions={
"is_phishing": Noul(
instructions="Is this email a phishing attempt?"),
"category": Choice(
instructions="What kind of email is this?",
criteria={"invoice": None, "password_reset": None,
"newsletter": None, "other": None}),
},
)
resp.answers["is_phishing"].noul # 0.93
resp.answers["category"].probabilities
# {"invoice": 0.71, "password_reset": 0.2, ...}Questions in one request are evaluated in parallel, so asking twenty takes about as long as asking one, and TypeSafe quotes 70 to 500 ms per call. On price, our arithmetic says about 70 times cheaper per input token than a frontier model, and the output is free.
LangChain already has a post with two pieces of agent plumbing built on it, a router that picks a cheap or an expensive model for the run and an "auto mode" guardrail that checks a tool call before it executes, the LLM doing the reasoning and Jev the small typed decisions along the way.
It has real limits. It can't count, it's bad at comparing numbers, negations confuse it, it reads literally, and the state tops out around 32k tokens, so you can't feed it a novel. Where it fits is the step where a smart model looks at some data and picks from a list.
The Good Old Days
Each of these questions used to have a classic answer: a regex on field names put anything with user in it in the user slot, a keyword index turned "PowerShell" into the right technique, and where the heuristics ran out a small classifier took over, which meant somebody labeled a few thousand emails, and labeled again for every new tenant. Fast and cheap, and they broke on anything they hadn't seen. The regex had no idea actor.alternateId was a user.
Then LLMs arrived, and here like everywhere else, these steps became LLM calls one by one. The regex became a prompt, the keyword index an agent with a search tool, and a few examples in the prompt replaced the labeling round. Generalization came for free. It works, and for a step that runs a few hundred times a day it is the right call. The trouble starts at a few hundred thousand times a month, when seconds and cents per decision stop being a rounding error and using a text generator to answer "which one of these" starts to look like abuse of the tool.
The fix isn't new either. Any LLM can be read as a decision model: put the options in the prompt and look at the next-token probabilities over the option labels instead of decoding an answer. Jev is that idea done on purpose: a model trained for the readout, served in one round trip, priced per input token. Whether the probabilities mean anything on our alerts is what the rest of this post checks.
Where Jev Fits in SecOps
Before you read on, pause for a second and think what in your own product is "look at this, pick one of these". Where does a model read a blob of text and return an id? How often does that step run, and who is waiting while it does?
Some space for you to think :)
When I did this for Vega the decisions fell into two groups. In the first, the cost of a decision is measured in seconds, because someone is waiting for it. In the second it is measured in dollars, because nobody is waiting but there are hundreds of thousands of them.
When Someone Is Waiting
An analyst types "failed logins for this user from outside the country" and the product has to pick which of the tenant's dozens of connected data sources the query runs on, while the analyst watches a spinner. That pick is a Choice with the source catalog as options, and the answer is a distribution: run on the top one, or on the top three when the probability is spread.
The same shape lives inside agents: is this tool call safe to run, does this request need the expensive model, the guardrail and the router from the LangChain post, one Noul and one Choice answered once per run.
There is a background version too, where nobody is staring at a spinner but the clock still matters. A tenant connects their SIEM and a couple of thousand detections need a MITRE tag before the coverage view means anything, and at 81 seconds a rule that takes about two days. Before any of that, its log types need a name and its fields need a home in the schema, one data source at a time.
When There Are Too Many
An alert is not one decision. Before anyone reads it, the pipeline has already asked a handful of small questions about it. Which of its fields are identifiers, a user, a host, an IP, so correlation can match on them: an LLM call per alert today, with a cache in front of it. Does it belong to the open incident that shares an identifier with it: a yes or no per candidate incident, also an LLM call. Are these three matched alerts one incident or three: one more. None of them is hard. All of them run on every alert that gets that far, and an enterprise tenant may easily fire tens of thousands of alerts a month.
Then comes the expensive question, does this alert deserve an investigation, and for most alerts the product answers it by counting, without a model. A detection that fires more than twenty times a day gets its alerts marked noisy and skipped for the next 24 hours, unread. A KQL-native rule gets a hundred, and both are defaults a tenant can change. An alert that arrives already resolved is never evaluated. In the month we replayed, tens of thousands of alerts fired for one enterprise tenant. Six in ten were skipped as noisy, unread. A third were never evaluated. About one in twenty-five was escalated, most of those by correlation rules on shared identifiers. The agent itself got to decide about one in seventy. Every one of those skipped and unevaluated alerts, most of the stream, is a "look at this, pick one of these" that nobody could afford to ask.
A Gate in Front of the Agent
We have a triage agent. An alert fires, the agent reads the detection and the rows that triggered it, loads the playbook for that alert class, runs a round of queries against the tenant's data, correlates what comes back with organizational context and its memory of earlier alerts, checks a few IPs against a reputation service, and decides: escalate, don't, or missing data. We measured ours in production: about two minutes per alert at the median and $0.34 on average, four to five model calls and one round of online queries, and the slow runs go well past that.
Here is the Jev-shaped question. Before the agent's first query it already has the detection, the trigger rows and the playbook. Hand those to Jev with the playbook's checklist as questions, one call: the decision as a Choice, each check on the list as a Noul. Is the actor an automation identity. Did the action complete. Is there enough evidence to decide.
The contract makes it a gate and not a replacement. A sharp probability on "not escalated", take it and skip the agent. Anything softer, run the agent as today. Jev never has to be right about everything. It has to know when it doesn't know.
The part I find more interesting than the model is the context, because an LLM answers from what is in front of it too. The agent's trick is that when the alert alone is not enough it goes and fetches more, a query at a time, and that loop is where its two minutes go. Jev gets no loop. So the whole game is what goes into the state before the call, and most of what an analyst wants to know about an alert can be gathered the moment it fires with fixed lookups, no reasoning needed to decide what to pull. This is where being a query analytics platform pays off: the tenant's logs and alert history are already indexed and queryable, so each lookup is a fixed query that runs the moment the alert fires, and the same store answers the agent's queries when it does run.
How this detection was triaged before, what else fired on the same user or host this month, which countries this user signs in from, when this host was first seen, whether the IPs sit on one of the tenant's blocklists, whether the account is a service account. For this replay we built the first two, the detection's own history and what else fired on the same entities. The rest are the same shape of lookup and we have not tried them. All of it built only from data that existed before the alert fired, or we are grading on hindsight.
Jev in the Agent's Chair
The Setup
Two tenants this time, and their agents live very different lives. The busy one is the tenant from the funnel above: hundreds of alerts decided over three weeks, about half escalated, most of them vendor alerts passed through. The quiet one: about half as many alerts, and the agent escalated one in six. Going in, I expected the quiet tenant to be the easy one for a gate.
Every transcript records what the agent saw, queried and decided, so on both tenants we could put Jev in the agent's chair at the moment before its first query. Three inputs on each, same questions, same threshold, nothing tuned between them.
- Alert only: the detection, the trigger rows and the playbook, exactly what the agent has before its first query. A few thousand tokens per call, most of them the questions.
- Alert plus the tenant context: the same, plus the two lookups we built from the section above, built only from records older than the alert: how this detection was decided before and what else fired on the same user, host or IP in the month before. A few hundred tokens of short phrases, and it says so when there is nothing to show, because "first time we see this" is a signal too.
- With the agent's evidence: everything the agent's own queries brought back, as an upper bound on what the online round buys.
In no input did Jev see the agent's queries, reasoning or decision on the alert itself.
The Playbook as Questions
The playbook's checklist became the questions: one Choice for the decision, with the agent's three options, one for the best benign explanation, one for the threat shape, and fifteen Nouls straight from the playbook, each a box the analyst would tick. One call, and the questions are most of its tokens, billed as input like everything else.
questions = {
"decision": Choice(
instructions="Based only on the state, which "
"escalation decision does the "
"evidence support?",
criteria={"ESCALATED": "...",
"NOT_ESCALATED": "...",
"MISSING_DATA_SIGNALS": "..."}),
"benign_explanation": Choice(
instructions="Which benign explanation best "
"fits the trigger rows, if any?",
criteria={"routine_automation": "...",
"attempt_fully_blocked_by_a_control": "...",
"no_benign_explanation_fits": "...",
...}),
"flagged_action_completed": Noul(
instructions="Did the flagged action actually "
"complete or succeed?"),
"enough_evidence_to_decide": Noul(
instructions="Is there enough evidence in the "
"state to make a confident call "
"without further investigation?"),
...
}
resp = client.system_one(state=alert, questions=questions)
resp.answers["decision"].probabilities
# {"NOT_ESCALATED": 0.92, "ESCALATED": 0.08, ...}Who Says Who Is Right
There are no human verdicts on these alerts to score against, and agreement with the agent only says whether Jev copies it. So we took every alert where Jev and the agent disagreed on any input plus a stratified sample of the rest, more than half of the agent's alerts on each tenant, and had each labeled by three judges: three separate runs of the same model, Claude Fable 5.1, each given the alert, the trigger rows and everything the agent gathered, but not Jev's or the agent's decision. They were unanimous about four times in five. Our security analysts then went over the same alerts by hand, verified the labels and corrected them where the judges were wrong, and the numbers below use the corrected labels.
Where Jev and the agent agree they are right or wrong together, so the gap between them lives entirely in the disagreements, and we labeled every one of those: the gap is exact, only the absolute level is sampled.
Side by Side
The alert in the figure is a Procdump execution on an endpoint, the kind of rule that fires on credential dumping and on a dozen legitimate things. The agent took 197 seconds and 9 model calls, a slower run than its two-minute median: it loaded the tenant's playbook, ran a round of queries and a file reputation check, read its own oversized output back in pieces, and decided not to escalate, because a Microsoft diagnostic script run by the system account had invoked a signed Sysinternals binary on a harmless process. Jev, from the alert alone, in about half a second, its median on both tenants, for about a hundredth of a cent: not escalated at 0.92, best benign explanation "SOC or security tooling workflow" at 0.92, actor is automation 0.96. The judges said not escalated.
Gate or Decider
Use Jev the way the contract said, one-sided. Look only at the alerts where Jev says "not escalated" at 0.8 or above, close those, and send everything else to the agent as today. What matters is whether Jev is right on the alerts it volunteers to take, and how many it volunteers for. The 0.8 was picked on the busy tenant and not touched for the quiet one.
| Gate on "not escalated", alert + tenant context | Busy tenant | Quiet tenant |
|---|---|---|
| Alerts Jev closes at 0.8 | 15% | 33% |
| Right, per the judges | 98% | 99% |
| Escalations inside, confirmed | one | two |
| Alerts Jev closes at 0.9 | 8% | 20% |
| Right, per the judges | 98% | 100% |
| The agent's own dismissals, right | 73% | 90% |
"Right, per the judges" is exact on the disagreements and sampled on the agreements. "Escalations inside" counts the alerts the judges would have escalated and Jev closed.
At 0.8, Jev's "not escalated" is right about 98% of the time on both tenants, and at 0.9 no confirmed escalation slipped through on either. Most of the alerts Jev closes, the agent had dismissed too. Where the two disagreed, the judges sided with Jev fourteen times out of fifteen on the busy tenant and twelve of fourteen on the quiet one. Over everything the agent dismissed, it was right about three times in four on the busy tenant and nine in ten on the quiet one.
The tenant context adds a second one-sided call on the busy tenant: Jev said "missing data" on a few dozen alerts, the judges agreed on three in four of them, and the agent had reached the same verdict on most of them after a full run each. What we would try with that answer is a lookup for the missing entity before anyone investigates, and that part is untested.
Deciding outright was never the goal, but we ran it as a check: take the raw decision and use it the way the agent's verdict is used today, on every alert. Jev loses to the agent on both tenants, with every input.
| Right, per the judges | Busy tenant | Quiet tenant |
|---|---|---|
| The agent | 72% | 75% |
| Jev, alert only | 66% | 42% |
| Jev, alert + tenant context | 69% | 65% |
| Jev, with the agent's evidence | 62% | 62% |
From the alert alone, Jev escalates far too much: four alerts in five on the busy tenant where the agent escalated about half, three in five on the quiet one where one in six deserved it, and the tenant context only trims that. The probabilities were not calibrated on that side either. A 0.9 on "escalate" was right about six times in ten on the busy tenant and far less on the quiet one, while the not-escalated side held. That asymmetry is why the gate reads one side only.
Giving Jev everything the agent's queries brought back did not help. On the busy tenant it was the worst of the three inputs, and on the quiet one it beat the bare alert but lost to the tenant context. More text is less signal for a model that reads the state once and picks.
What the Context Buys
Same model, same questions, same threshold, and the gate closes 15% of the agent's alerts on one tenant and a third on the other. Before you read on, take a guess at why.
The answer is in the tenant context. Nine in ten of the busy tenant's closes and five in six of the quiet tenant's were alerts whose detection had a clean record, its recent decisions all dismissals. Inside that group Jev closed 59% of the alerts, on both tenants. The yield did not move. The size of the pile did: a quarter of the busy tenant's replayed alerts sat behind a clean record, nearly half of the quiet tenant's. Where the record was mixed, Jev closed a few percent on the busy tenant and one in six on the quiet one. Where it held only escalations it closed none, and on detections seen for the first time a few percent. A gate's coverage is a property of the tenant's noise more than of the model.
A rule that closes every repeat of a detection that was always dismissed before would have closed the whole clean-record group, including the one alert in five the agent went on to escalate on the busy tenant. Jev closed six in ten of them and let one confirmed escalation through.
On the busy tenant, agreement with the agent went from 63% to 68% when the tenant context went in, and closes at 0.9 from under 1% to 8%. All of it is a handful of fixed queries over data the platform already indexes. The tenant context is how the gate gets to see the tenant's chronic noise, and building it is ordinary work for a query analytics platform.
One caution. "How this detection was triaged before" carries the agent's past decisions into the state, and it usually matched the agent's decision on the alert itself, so part of the extra agreement is Jev learning what the agent tends to say. A live gate should learn from whoever triaged the detection before it, agent or analyst, but it is why the judged numbers are the ones that count for this input.
What We Would Try First
If we take this further, the first thing we would try is the one-sided gate with the tenant context in front of the agent. Read only the "not escalated" side, above a threshold like the 0.8 we used here, and treat "missing data" as a cue to look up the missing entity before anyone investigates. In the replay that closed 15% of the alerts the agent was handed on the busy tenant and a third on the quiet one, right about 98% of the time by the judges. None of it has run on a live alert, and the threshold would need picking again on live data.
That is the small prize. Tens of thousands of alerts a month fire for the busy tenant and the agent gets to decide about one in seventy, because a full agent run was never on the table for the rest. Reading every one of them with the same questions costs under twenty dollars. We priced that pile. We did not replay it, and the escalate side was the weak side here.
Caveats. Two tenants, the same weeks, and we say so. The ratios, roughly 230 times faster at the median and about 2,000 times cheaper on average, compare one call with an investigation. They are true and unfair, and the gate is what makes them fair. If this ever runs on live alerts, its own closes would have to stay out of the history it reads, or it grades its own homework.
Wrapping Up
Three things to take away.
- Jev works as a gate in front of the triage agent: it closed 15% of the agent's alerts on the busy tenant and a third on the quiet one, right about 98% of the time by the judges, in one cheap call each.
- The tenant context is what made it work: which detections had a clean record and what else fired on the same entities. That is a handful of fixed queries over data the platform already indexes, and the part we already know how to build.
- Jev cannot replace the agent's verdict: deciding outright, it loses to the agent on both tenants. So read the "not escalated" side only and hand everything else to the agent.
Both replays together cost under a dollar in Jev credits.
We also pointed Jev at our MITRE mapping pipeline, but that is a post of its own. If you tried Jev on something SecOps-shaped, or you think fifteen yes/no questions is a silly way to read an alert, tell us, we'll probably agree :)
