Contain the Model, Govern the System

Summary

Frontier models are more capable than most traditional software — and far less traceable. A bug in a codebase can be traced to a line; an unexpected model output cannot be attributed to a training example or a weight configuration. Yet these systems are being wired to our most sensitive data and given the authority to take mission-critical actions. Capability without attribution is not a problem models will solve for us; it is a risk we have to engineer around. Responsibility for what intelligence does on our behalf cannot be outsourced to a model provider's assurances.

The shift

For decades, software shipped with a mechanical contract: behavior traced to a code path, failures reproducible, responsibility plausibly assigned. Super Intelligence systems break that contract. We deploy them inside the enterprise with access to the most sensitive data and the ability to act — without the ability to mechanistically explain what they did or why.

That gap changes what "trusting" a system means. We can no longer rely on the output alone, on the provider's guarantee, or on a single layer of reasoning transparency. The reframing that works: treat frontier models like insider risks. Not because they are malicious, but because a sufficiently capable actor with access to important systems can make mistakes or be compromised — and the architecture has to hold regardless. Insider-risk discipline is largely solved; it has been refined for decades: establish identity, limit privileges, log activity, create containment boundaries. The same principles apply to models.

This means separating the supply of intelligence from the authority over it. The model supplies answers; a deterministic system around it decides what it may see, what it may do, and what evidence it must leave behind. Controls must sit outside the model — a program must not be able to bypass or tamper with the mechanisms that enforce its own permissions, a principle that predates the industry.

Why it matters

The naive alternative fails quietly: an opaque model inside an opaque orchestration layer, continuously verified by another opaque model. Adversarial model-to-model verification still terminates in nested black boxes. Reasoning transparency ("chain of thought") is a necessary start, but we do not yet know how to make model outputs consistently faithful or transparent — observability cannot be delegated to the thing being observed.

Every organization deploying agentic systems inherits the same exposure: sensitive data, mission-critical actions, and no mechanistic way to reconstruct what happened after the fact. Operators and decision-makers stay accountable for outcomes whether or not a model provider offers assurances — regulators, counterparties, and clients do not accept vendor indemnities as a substitute for control.

The deeper consequence: whoever controls the harness, the action space, and the audit trail — not whoever owns the model — holds the actual accountability. The organization that operates the system is the organization on the hook. That is the strongest argument for owned, contained systems over rented black boxes.

What to do

Design agentic systems around principles of observability — the model inside, but never trusted alone:

• Model diversity — no single model should be the sole dependency for an important outcome, or be responsible for verifying its own work.
• Observe everything — every meaningful model action must leave tamper-proof, human-readable evidence. If an outcome cannot be reproduced without the model's own attestation, it has not been observed.
• Verifiability — test the entire system continuously: failures, attacks, edge cases, and system changes — not just successful happy-path tasks.
• Independent controls — the organization, not the model or its provider, determines what a model can access and what actions it can take.
• Independent auditability — validation must be independent of the intelligence being validated. No single model should control both a system's behavior and the evidence used to judge that behavior.
• Containment — assume the model is compromised from the start. An authorized person must always be able to pause or shut it down mid-task. More capable models will require more advanced containment technologies, and those need industry standardization.
• Incident disclosure — when systems fail or are compromised, disclose promptly to those affected — including the implementation details that change agent behavior at runtime — and share learnings industry-wide.

Separate the model from the harness that orchestrates its work and from the action space that bounds what it can do. Chain-of-thought transparency is non-negotiable but insufficient; model-to-model verification is useful but not a substitute for controls that live outside the model.

The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least.