Seven years of retrieval work teaches a particular lesson. Search ranking, entity resolution, query understanding, and long stretches of travel data where the same hotel arrives under eleven names and three of them are wrong.
The fix was an ontology: decide what a property is, what a rate plan is, which relationships are allowed to exist, then make every downstream system answer to that shape. Knowledge graphs did the work much better than vector similarity over similar text.
The habit persists outside work. An Obsidian wiki repo can hold personal memory as linked pages with typed edges, because a chat log only replays what was said and a vector index only returns what looks close. You maintain ground truth by hand or it rots while you keep querying it.
So the industry rediscovering ontologies last year was welcome news. Then the same techniques met security for AI agents, and one question turned out to be beyond what those graphs could answer.
‘Wrong’ is not a property of the action
When someone tells you an agent went wrong, ask what they compared it against.
At a four-person startup, reading that file, read_file("contracts/q3-renewal.pdf"), might be completely normal. At an insurer, the exact same action could trigger a compliance and reporting obligation.
The verdict comes out of a relationship: this subject, holding these entitlements, touching a document at that tier, inside a use case approved at that risk level, on behalf of an owner who may or may not still work there. Change any one of those and the same bytes flip from routine to reportable.
It goes both ways in practice. A block that reads as paranoid in one org is the only thing standing between another org and a fine worth four percent of revenue. A summarization agent touching internal memos is boring. The same agent pointed at a multi-million contract, translating a clause into another language, is a legal exposure with a named owner.
Security for agents turns out to be an ontology question wearing a different hat. You cannot judge an action without the entities and the edges between them.
The named layers all model the task
Every few weeks the agent community names another layer.
One week it is harness engineering: the execution loop, tool contracts, sandboxing, checkpoints, the boring operating-system work that keeps a long-running agent from falling apart on turn nine.
The next week it is context engineering: assembling what goes into the window on purpose, instead of dumping whatever top-k chunks a vector search handed back.
Then someone reminds the room that ontologies never left, and that a bank running 400 applications needs one definition of “customer” that billing, support and compliance all agree on.
Three real layers, and all three are worth building on. Every one of them models the task: what is true in the domain, what belongs in the window, how the loop executes safely. None of them models the organization the task runs inside.
The org model is stateful
People file the organizational picture as a policy PDF. It behaves like live state, and it moves:
- which data classes exist and where they sit
- which permissions a user actually holds, including the low-level entitlements nobody remembers granting
- which use cases got approved, at what risk tier, with which model version
- who is accountable when the output is wrong
- which agents and MCP servers somebody installed last Thursday
Each of those changes without anyone telling the agent. Someone gets added to a shared drive. A model gets swapped for a cheaper one. A use case that shipped as summarization quietly grows into reasoning. Legal signs off on a new region.
A hotel ontology holds still for months. An org ontology holds still for about a week. That is the difference retrieval work tends to underestimate: modelling the entities is the solved part, and there is no machinery for the fact that the model itself keeps moving under the agent.
Drift is two clocks disagreeing
Most drift talk points at the model. The version that keeps showing up runs the other way: the org moves and the agent’s copy does not.
Picture an agent making investment calls off market data, open-source intelligence and your own portfolio. You did the work: access control, prompt hardening, data protection, a signed-off model. Then someone manipulates one upstream data source. Every control still passes, the output moves, and your investment pattern moves with it.
You catch that only if you kept a picture of normal for this org and kept checking the live one against it.
Neither clock is wrong on its own. The gap between them is the signal, and you can only compute a gap if both sides are written down somewhere outside the model. So the ontology work comes before the monitoring work. You cannot diff two pictures of an organization until you have decided what an organization is made of.
Where this lands in code
This is the part Prismor builds. One graph of the organization, and every enforcement point reads from it.
The entities are the boring ones you would draw on a whiteboard: a device, a session, a tool, a data class, a destination, a subject carrying a permission profile. Plus a named agent instance, which is a different thing from a framework, so support-bot and release-bot can share a runtime and carry different controls.
The edges are where the judgment lives.
Tools carry tags, and tags compose into paths through a session. untrusted_content then private_data then external_comms -> block reads as an ordered path. Swap then for with and it means co-occurrence instead of sequence. Neither one is expressible as a regex over a command line, which is why a flat rule table kept missing it.
A data class crossed with a destination tier decides what may leave. One engine asks whether a host may be contacted at all, a second asks what may be sent there, and neither half is a finding on its own. Your own email address going to your own staging box stays quiet.
A session gets a subgraph of its own, synthesized once from the task and enforced for every call after that. Anything outside it is denied by omission, which is the property you want and also the one that bites the first time a tool is missing from the list.
One decision point reads all of it. Coding-agent hooks, an MCP gateway, mirrored built-in tools, in-process SDK adapters, a local evaluation server, a hosted inference-hook channel: six places that discover a tool call, one place that judges it. Six copies of a precedence rule is five too many.
Drift then has a definition you can compute. Inventory what is running on the machine, diff it against what the graph says is governed, and the difference is the answer. Same query whether the gap came from an MCP server someone installed on Thursday, a permission that widened, or a use case that grew from summarization into reasoning.
Narrow agents are what make this tractable
The agents that reach production are microscoped. One job, four tools, a small blast radius. A narrow agent has a subgraph small enough to diff against reality: these tools, these paths, this data class, this subject, this window of time. When the org moves under it, the diff shows up.
A broad agent with the whole toolbelt has nothing to compare against. You cannot detect drift from a baseline you never wrote down.
So you rarely need to sandbox everything. Scope the agent once per session, write the scope down, then let a deterministic engine check every call against it. The model gets one chance to reason about what this session needs. After that, matching is arithmetic.
What to keep outside the model
- Write the org graph to disk. Outside the context window, outside the transcript. If it only lives in the prompt, it is gone by turn twelve.
- Resolve scope once per session, enforce it per call. Reasoning is slow and probabilistic. Enforcement should be neither.
- Attach a name to every use case before go-live. Not the IT owner. The person who gets the call when the output is wrong. That name changes what controls people are willing to fund.
- Log state transitions, not log statements. Every tool call, every result, every scope decision, with a timeline you can replay. Your incident responders have never seen an eval harness. Give them something they can read at 2am.
- Make the agent check state instead of memory. If it already processed that file, it skips because a record says so.
What is left over
Nobody sells this layer yet. Buy any one AI security product and you have covered a slice: the prompt, the gateway, the endpoint. None of them holds a model of what your own organization currently permits, and none will tell you the day that model changes.
So write the picture yourself. Take one production agent, put its permitted tools, paths and data classes in a file today, and diff that file against reality next Monday. The lines that come back different are your incident report, three weeks early.
Where this goes
The Prismor enterprise version addresses it at the scale the problem actually has: one org model spanning every device, agent and session in the fleet, signed policy pushed from a console rather than copied between laptops, and a standing diff between what the graph says is governed and what is really running. The runtime that enforces it is open source at github.com/PrismorSec/prismor.
