Who Should Be Allowed to Change the Knowledge AI Uses?
The more I think about AI engineering, the more I realize that knowledge isn't just information.
Some knowledge has authority.
An architecture decision can affect dozens of services.
A security policy can restrict what an AI agent is allowed to implement.
An API contract can determine how multiple teams build their systems.
A deployment policy can prevent an unsafe release.
So here's a question I've been exploring while building NAEOS:
Who gets to change the knowledge that AI relies on?
Imagine an AI agent retrieves this rule:
"Production deployments require security approval."
Then someone quietly changes it to:
"Production deployments don't require approval."
If there's no ownership, approval, or audit trail, the AI has no way to understand whether that change is legitimate.
That's not just a documentation problem.
It's a governance problem.
I think engineering knowledge needs the same kinds of controls we already apply to production systems:
- Ownership
- Roles and permissions
- Review
- Approval
- Versioning
- Audit trails
- Policy enforcement
- Deprecation
- Traceability
A developer shouldn't necessarily be able to modify an organization-wide security policy.
An AI agent certainly shouldn't be able to change one without authorization.
This leads to an interesting principle:
AI should not only know what is true. It should know who has the authority to define what is true.
That's a subtle but important distinction.
And it becomes even more important when AI agents start making changes autonomously.
The question isn't only:
"Can the AI do this?"
It's also:
"Is the AI authorized to do this?"
That's the governance layer I'm exploring in NAEOS.
The goal is not to slow AI down with bureaucracy.
It's to make autonomy safe enough to scale.
I'd love to hear how other teams are approaching this.
What engineering decisions should always require human approval, even when an AI agent can technically make the change itself?
This maps almost exactly onto a problem we kept hitting building document tools for consultants: an AI can draft a client proposal, but it shouldn't be the one deciding what price or scope line is "final." The fix that worked for us was keeping a hard line between what the model is allowed to draft and what only a human can commit to — versioned, so you can always see which one happened. Same governance instinct, just applied to a two-person org instead of a platform team.
Exactly. I think that example captures the governance principle at its simplest: AI can participate in producing an artifact without being the authority that commits the organization to it.
The distinction between drafting and committing is particularly useful because it scales beyond pricing and proposals.
In an engineering environment, the same boundary could look like:
Agent proposes → Human reviews → Authorized actor commits → System records the decision
The important part is that the final state is not determined merely by whoever has the ability to write to the system.
Versioning then gives you the second half of the control: not just who committed the decision, but what the agent proposed, what changed during review, and which version ultimately became authoritative.
I think this is a strong example of why AI governance doesn't necessarily require a large enterprise bureaucracy. Even in a two-person organization, separating generation, authority, and commitment can prevent an AI system from quietly turning suggestions into organizational decisions.
That same principle becomes much more important as agents gain the ability to execute changes directly.
yeah that's the part that trips people up in practice. the propose/commit split works fine when someone's watching, but the moment a session gets cut short or context resets, you lose the thing that actually proves who approved what. we ended up just logging the diff + approver + timestamp separately from the agent's own memory, precisely because you can't trust the agent to remember it accurately later. is that baked into NAEOS itself or more something teams have to bolt on?
That’s an important distinction, and I would consider it a core architectural requirement of NAEOS, not something teams should have to bolt on themselves.
Agent memory should never be the authoritative source for approval or authorization history. Memory can help the agent understand context, but the evidence of an authorization decision needs to live in an independent, durable system of record.
I’d separate the two explicitly:
Agent Memory → contextual knowledge and continuity
Decision Record → what was proposed, approved, rejected, or escalated
Audit Evidence → immutable evidence of what actually happened
Policy Registry → the authority and policy version governing the action
For a consequential change, the record should be reconstructable even if the agent session disappears completely:
Proposal → Diff → Policy Version → Approver → Timestamp → Evidence → Execution Outcome
That also means approval should be bound to the specific artifact or action being approved, rather than simply recording that “someone approved this.”
So if context resets, the system doesn't need the agent to remember what happened. The governance layer already knows.
This is one of the principles I want NAEOS to enforce: agents may consume memory, but governance decisions must have an independent source of truth.
The agent can forget. The audit trail cannot.
That "4-way split" (Agent Memory / Decision Record / Audit Evidence / Policy Registry) is a cleaner way to say what we ended up doing by accident. In our case the "Policy Registry" is just a versioned transmittal sheet — every AI-drafted section gets a version number and sits in a "pending" state until a named person signs it off, and the signed version is what actually goes out. Nothing about the AI's memory of the conversation matters once that happens. The one thing I'd add to your model: binding the approval to the artifact's content hash, not just its version number, so a re-approval can't silently attach to a slightly different draft than the one a human actually reviewed.
Yes — binding approval to the artifact content hash is an important refinement.
A version number provides lineage, but the hash provides identity. Without the latter, there is still a potential gap between what was reviewed and what was ultimately committed.
I’d model the approval binding as:
Artifact ID → Version → Content Hash → Policy Version → Approver → Approval Timestamp → Execution/Release
That gives us a stronger invariant:
If the content changes, even within the same versioning workflow, the approval becomes invalid and the artifact must return to the appropriate approval state.
I also like your “pending → signed → released” model because it makes the authority transition explicit. The AI can continuously generate and revise drafts, but the signed artifact represents a distinct governance state that does not depend on the agent’s conversational memory.
This reinforces an important architectural principle for NAEOS:
Versioning provides history.
Content hashes provide integrity.
Approval provides authority.
Execution evidence provides accountability.
That combination is much stronger than relying on an agent to remember what happened in a previous session.
Right, and that's basically the whole thing in one line: the audit trail has to outlive the agent's memory, not depend on it. We've been running into the same idea outside of AI governance too — anywhere an agent works across sessions, the moment you trust its own recollection for anything consequential, you've already lost the guarantee. Curious if you're building the policy registry as its own service in NAEOS or bolting it onto whatever DB the agent already talks to.
Exactly. That’s the distinction I’m trying to make with NAEOS: memory is context, not authority.
For the Policy Registry, I’m leaning toward making it a separate control-plane service, rather than embedding it into the agent’s existing DB. The registry should have its own lifecycle, versioning, policy identity, and access boundaries so the agent/runtime can’t implicitly redefine the rules it is supposed to operate under.
The agent can query the registry for context, but the Runtime/Control Plane should be the enforcement point. Something like:
Policy Registry → Authorization Decision → Runtime → Audit Evidence
That also makes policy versions and decision records independently addressable, which becomes important when you need to answer “which policy authorized this action?” months later.
The existing agent DB can still hold operational state and memory, but it shouldn’t be the source of truth for authorization.
I’d add one requirement: bind the action to the exact policy version the agent saw. A rule change can be legitimate and still be applied outside its intended scope, or after it has been superseded. For consequential actions, I’d want a record of the rule/version, owner, effective window, approval, and evidence available at decision time. That gives you a cleaner boundary: humans review policy conflicts, irreversible actions, and attempts to widen authority; routine execution can remain autonomous.
Agreed. Policy version binding is a critical part of making the audit trail meaningful.
It is not enough to know which policy exists today. For a consequential action, we need to establish exactly which policy state governed the decision at the moment the action was authorized.
I would model the authorization record around something like:
Action → Policy Version → Owner → Effective Window → Approval → Evidence → Outcome
That also helps address a subtle failure mode: a policy can be valid and correctly approved, yet still be applied outside its intended scope or validity window.
The separation you propose is compelling:
This creates a useful principle for NAEOS:
Autonomy operates within a versioned authority envelope.
The agent can execute freely inside that envelope, but it cannot redefine the envelope it is operating within.
And when something goes wrong, the system should be able to reconstruct not only what the agent did, but which authority state made that action permissible at that exact point in time.
That is the level of provenance I think agentic engineering systems will eventually need for production-grade governance.
The authority distinction is important. One thing I’d add is that defining the permission boundary is only half the problem.
Once the agent is live, how do you independently verify that it consistently respected that boundary — particularly the cases where it should have stopped, requested approval or escalated?
We’ve been exploring this with OpsWatch using risk-based sampling of real production behaviour rather than only validating the control configuration.
I’d be interested in how you’re thinking about that verification layer with NAEOS.
I agree. Defining the boundary establishes the control; independently verifying behavior establishes whether the control actually works.
For NAEOS, I think this needs to be treated as a separate verification layer rather than relying on the agent’s own logs or self-reporting.
The model I’m exploring is roughly:
Policy → Enforcement → Execution → Evidence → Independent Verification
The verification layer should sample actual agent actions and, importantly, non-actions: cases where the agent should have stopped, requested approval, or escalated but may have continued instead.
Risk-based sampling makes sense here because not every interaction has the same significance. High-risk actions and boundary conditions should receive much higher verification coverage, while lower-risk activity can be sampled statistically.
I’d also want verification to evaluate the decision against the policy that was actually active at the time of execution, rather than only checking whether the current configuration looks correct.
That creates an important distinction:
“Was the policy configured correctly?”
vs.
“Did the agent actually behave according to the policy?”
The second is the harder—and arguably more important—question.
This is an area where I think NAEOS still has interesting work to do. The goal should be to make policy compliance an observable engineering property, with evidence that can be independently audited rather than simply asserted by the agent or the control plane.
This is exactly the distinction that led me to build OpsWatch.
We sit specifically at that independent verification layer — separate from the agent, implementation team and control plane — and test whether deployed behaviour actually matched the controls and policy that were supposed to govern it.
That includes actions, non-actions, escalation/approval behaviour, failure paths and the evidence needed to reconstruct what actually occurred.
Your Policy → Enforcement → Execution → Evidence → Independent Verification model is remarkably close to how we’ve been approaching the problem.
There may be a genuine fit between NAEOS and OpsWatch here rather than us solving the same problem twice.
I think there is a genuine architectural fit here.
The separation you describe is important: NAEOS can define the engineering and governance contract, while OpsWatch can independently verify whether the deployed system actually behaved according to that contract.
That gives us a potentially clean boundary:
NAEOS → Define policy, authority, controls, and expected behavior
Agent/Runtime → Execute within those constraints
OpsWatch → Independently verify observed behavior against them
I particularly like that your verification model includes non-actions, escalation, approval behavior, and failure paths. Those are precisely the cases that traditional observability tends to miss because the absence of an action is harder to measure than an action itself.
There is also an interesting feedback loop here:
Policy → Execution → Verification → Findings → Governance improvement
The important part would be keeping verification independent, so the system that defines or enforces the control is not also the sole authority declaring that the control worked.
I’d definitely be interested in exploring the integration points between NAEOS and OpsWatch. This feels less like overlapping products and more like two complementary layers of the same agentic engineering architecture.
Happy to continue the conversation and see what a concrete integration model could look like.
I agree — I think the clean separation is the real opportunity here.
A useful first step might be to define one small reference workflow rather than trying to design a full integration upfront.
For example:
NAEOS defines the policy, authority boundaries, expected actions/non-actions, escalation conditions and approval requirements.
The agent/runtime executes against that contract.
OpsWatch then independently evaluates the resulting execution evidence against the NAEOS-defined expectations and produces an inspectable verdict covering behaviour, control-path compliance, escalation/approval handling, failure paths and any required re-verification.
That would let us test the architectural boundary in practice without either product needing to absorb the other.
If the model works, the output could become a repeatable pattern for higher-risk agent deployments where engineering governance and independent assurance need to remain structurally separate.
I’d be very happy to sketch the first reference workflow together.
I agree. A small reference workflow is probably the right way to validate the boundary before thinking about a broader integration.
The separation you’ve outlined is clean:
NAEOS → defines the contract
Agent/Runtime → operates within the contract
OpsWatch → independently verifies the observed behavior
I’d suggest we make the first workflow deliberately narrow and consequential enough to exercise the important controls—for example, an agent proposing and executing a production-impacting change.
The reference flow could establish:
Policy → Authorization → Agent Action → Evidence Capture → Independent Verification → Verdict → Re-verification
The key is that NAEOS and OpsWatch should remain independently authoritative. NAEOS should not determine whether its own controls were effective, and OpsWatch should not become the system that defines the engineering policy it is evaluating.
If we can make that boundary work with one concrete workflow, we can then identify which interfaces are actually necessary rather than designing an integration based on assumptions.
I’d be very happy to sketch the first reference workflow together. I think this could become a useful reference architecture for governed and independently assured agentic engineering.
Agreed — I think that boundary is exactly the right one to preserve.
A production-impacting change is also a good first reference workflow because it gives us something concrete enough to test authorization, execution, evidence capture, stop conditions and the final verification decision without introducing unnecessary integration complexity.
For the first pass, I’d be interested in defining something like:
NAEOS policy/contract → authorization state → agent proposes change → permitted/blocked execution path → evidence captured at each decision point → OpsWatch independently reconstructs what occurred → verdict against the policy that was active at execution time → re-verification after any control change.
I’d also deliberately include at least one negative path — for example, an action the agent should refuse, escalate or seek additional approval for. That would let us test not only whether permitted actions succeed, but whether the boundary actually holds when the correct behaviour is non-action.
If we can define that workflow cleanly, I think we’ll have enough to identify the minimum interface between NAEOS and OpsWatch without either system becoming dependent on the other’s interpretation.
Happy to sketch that with you.
Agreed. I think the negative path is essential because a governance system is not really validated by showing that an allowed action succeeds. It is validated by demonstrating that the system refuses, escalates, or requires approval when it should.
I’d propose we make the first reference workflow intentionally small:
1. NAEOS defines the policy and authority boundary
2. Agent proposes a production-impacting change
3. Policy engine evaluates the request
4. Runtime either permits, blocks, or requires approval
5. Evidence is captured at each decision point
6. OpsWatch independently reconstructs the execution
7. Verdict is evaluated against the exact policy version active at execution time
8. Control changes trigger re-verification where required
For the negative path, we could test something like an agent attempting an action outside its capability scope. The expected result would be deterministic denial or escalation, with the entire decision path captured as evidence.
That gives us two complementary tests:
Positive path: Can the agent operate autonomously within its authority?
Negative path: Does the authority boundary actually hold when the agent attempts to cross it?
I also agree that keeping the interface minimal is important. Ideally, we exchange contracts, authorization state, execution evidence, and verification results, rather than having either system depend on the other's internal implementation.
Let's sketch the workflow around those two paths first. That should give us a concrete basis for defining the minimum NAEOS ↔ OpsWatch interface.
Agreed. Let’s turn this into a small, testable reference workflow.
I suggest we define one production-impacting action—such as an agent attempting to modify a governed knowledge source—and run it through both paths:
Positive path: the proposed change is within the agent’s authority, satisfies the active policy, and proceeds with complete evidence.
Negative path: the same agent attempts a materially similar change outside its authority, causing deterministic denial or escalation.
For each path, we can capture a minimal evidence package:
policy and contract identifier
exact policy version active at execution time
agent identity and delegated authority
proposed action and affected resource
authorization inputs and decision
approval or escalation state
runtime outcome
timestamped execution evidence
any later policy or control changes
OpsWatch can then independently reconstruct the sequence and issue a bounded verdict on whether the observed behaviour matched the authority boundary defined by NAEOS.
The first interface could remain deliberately implementation-neutral:
NAEOS → policy contract, authorization state and policy-version reference
Runtime → proposed action, decision events and execution evidence
OpsWatch → verification result, exceptions, evidence gaps and re-verification status
If that structure works for you, the next step could be to choose the exact governed action and define the expected positive and negative outcomes before either side designs additional integration. I’m happy to draft the first workflow schema for us to refine together.
Agreed. This is a strong way to keep the first iteration concrete and implementation-neutral.
I’d choose “modify a governed knowledge source” as the initial action because it gives us enough complexity to exercise authority, policy versioning, evidence capture, and rollback without introducing unnecessary infrastructure dependencies.
I’d define the reference workflow around two explicit cases:
Positive:
Agent has the required capability → active policy permits the change → execution proceeds → evidence is captured → OpsWatch independently verifies compliance.
Negative:
Agent lacks the required capability or violates the active policy → execution is deterministically denied or escalated → evidence captures the attempted action and decision → OpsWatch verifies that the boundary held.
The minimal interface you proposed also looks right:
NAEOS → Contract + Authorization State + Policy Version
Runtime → Action + Decision Events + Execution Evidence
OpsWatch → Verification Result + Exceptions + Evidence Gaps + Re-verification
One additional field I’d consider making explicit is the resource identity and state/version at decision time. Otherwise, an authorization can be valid for one resource state but later be interpreted against another.
So the evidence chain becomes:
Policy Version → Authority → Action → Resource State → Decision → Execution → Evidence → Verification
If we agree on that, I think the next useful artifact is a small reference schema plus the two expected outcome traces—one allowed and one denied/escalated.
That gives us something we can actually test end-to-end before deciding whether any deeper integration is necessary.
Agreed. Resource identity and state at decision time should be explicit. Without that, we could reconstruct the correct policy and authorization decision against the wrong version of the governed resource.
I’d represent it using an immutable resource identifier, pre-action version or hash, proposed change reference, and—where execution occurs—the resulting version or hash. That also gives us a clean basis for verifying rollback and detecting state changes between authorization and execution.
The two traces should therefore preserve both decision-time and execution-time state:
Positive:
Policy version → authority → proposed action → pre-action resource state → permitted decision → execution → post-action resource state → evidence → OpsWatch verification
Negative:
Policy version → authority deficiency or policy violation → proposed action → resource state → denied/escalated decision → confirmed non-execution → evidence → OpsWatch verification
“Confirmed non-execution” is important: a denial event alone does not prove that the protected resource remained unchanged.
I’m happy to draft the first implementation-neutral reference schema and the two expected outcome traces. You could then map the NAEOS-side fields to its actual contract and authorization model, while we keep the OpsWatch verification fields independently defined.
That should give us a bounded specification we can test before considering any deeper integration.
Agreed. I think confirmed non-execution is an especially important addition. A denial is a control-plane event; it is not, by itself, proof of the resulting system state.
The separation between decision-time and execution-time state also gives us a much stronger verification model. We can distinguish:
What was authorized?
What resource state was evaluated?
What actually executed?
What state resulted?
I’m aligned with keeping the first specification implementation-neutral. I suggest we structure it around four artifacts:
The verification rules should remain independently defined on the OpsWatch side. That preserves the architectural boundary: NAEOS defines what should be permitted, while OpsWatch independently determines whether the observed behavior actually matched that contract.
I’ll map the NAEOS-side concepts to the schema while keeping the interface minimal and implementation-neutral.
This gives us a bounded specification that we can test end-to-end before introducing any deeper integration.
Agreed. Those four artifacts give us exactly the right boundary for the first iteration.
I’ll draft:
I’ll keep the contract supplied by NAEOS separate from the observations and conclusions produced by OpsWatch. The verification rules will evaluate consistency across authorization, decision-time resource state, execution evidence and resulting state without relying on NAEOS’s own pass/fail interpretation.
Your mapping of the NAEOS concepts can then tell us whether the proposed fields are directly available, require derivation or expose an evidence gap. That mapping will show us the minimum practical interface before we consider any deeper integration.
I’ll prepare the bounded first draft around the governed knowledge-source modification workflow for us to refine. Once it is ready, we can exchange and refine it by email if that is easier. You can reach me at jason@mcgillintelligence.com.au.
Agreed. That gives us a clear and well-bounded first iteration.
I’ll prepare the NAEOS-side mapping against the proposed schema, specifically identifying which fields are:
I’ll also make sure the distinction between policy/authority, runtime evidence, and independent verification remains explicit throughout the mapping.
The governed knowledge-source modification workflow sounds like the right starting point. It is sufficiently concrete to test the architecture while remaining implementation-neutral.
Once your draft is ready, please send it through. You can reach me at bayu@naeos.dev.
I’m looking forward to refining the schema and seeing whether the two systems can establish a clean verification boundary without creating unnecessary coupling.
This is a very promising direction.
I’d base the approval boundary on blast radius and reversibility, rather than whether the change was proposed by a human or an AI.
Identity and permissions, secrets, production-data mutations, billing, deployment policies, and changes to the agent’s own guardrails should always require human approval. For lower-risk knowledge changes, signed ownership, versioned diffs, and reliable rollback may be enough.
I’d also separate who can propose a policy change from who can approve it. Otherwise, an agent with write access could quietly expand its own authority.
I agree with this framing. The approval boundary should be determined by the risk characteristics of the action, not by whether the proposer is human or an agent.
The separation between proposal authority and approval authority is especially important. Without it, an agent could potentially convert operational autonomy into governance authority simply by modifying the rules that constrain it.
I would model this as three distinct capabilities:
Propose → Approve → Execute
An agent may be allowed to propose a policy or engineering change, but that does not imply it can approve the change or expand its own permissions.
For NAEOS, I think this maps naturally to a policy enforcement layer where authorization is evaluated independently from agent identity. The system should be able to determine:
This also creates a useful principle for autonomous engineering:
Autonomy should expand execution capability, not governance authority.
That distinction becomes increasingly important as agents move from generating code to actually operating production systems.
The harder tradeoff seems to be deciding where authority needs to exist without turning every autonomous action into a human approval queue.
That boundary probably matters as much as the underlying permission model.