Key Takeaways
- Autonomous AI agents pose a challenge to traditional Kubernetes security models, making runtime decisions on external service calls, holding multi-domain credentials, and exhibiting unpredictable resource consumption.
- The Kubernetes Job pattern isolates agent workloads, giving each execution its own container, memory space, and lifecycle, preventing resource starvation by runaway tasks and enhancing security.
- Agent workloads require a different approach to secrets management than traditional microservices. An agent that reasons across network, database, and application domains needs credentials for all three, which expands the blast radius if a single container is compromised.
- Platform teams employ a four-phase graduated trust model (shadow, read-only, limited write, and autonomous) to incrementally expand agent permissions, governed by specific observability criteria for a structured and secure progression.
- Observing non-deterministic workloads is challenging because traditional request/response traces cannot capture the dynamic cycles of hypothesis evaluation and refinement.
The 2 AM Problem
It's 2 AM. Your dashboard flashes red as three hundred alerts flood in across the network, database, application, and security domains. The on-call engineer fires up six dashboards, correlates timestamps, and formulates a hypothesis, then fetches log information to test it, only to find the hypothesis was incorrect. This process repeats until a root cause is found three hours later: a firewall rule change that cascaded into application timeouts, causing database connection pool exhaustion.
Imagine an AI agent automating that entire triage process. That agent will need to pull network telemetry, query the application logs, examine database metrics, and cross-reference security events. In addition, the agent will need to authenticate each of those systems and determine which additional data sources to query at runtime based on what has been discovered so far. It must perform all of these actions within your production Kubernetes cluster.
That agent is neither a microservice nor a batch job. Its dependency graph, credential footprint, and execution path are all dynamic. In this article, we describe the infrastructure patterns we developed while operating an autonomous diagnostic agent system on Kubernetes: isolation, secrets containment, graduated trust, and observability for a workload category that breaks the assumptions behind most Kubernetes security models.
Why AI Agents Break Your Existing Kubernetes Security Model
Most Kubernetes security models assume workloads have a clearly defined dependency set, call a defined list of external services, and predictably consume resources.
Therefore, when developing your role-based access control (RBAC) roles, network policies, and resource limit sizing, keep the workload within established boundaries. However, be aware that autonomous AI agents violate every one of these assumptions.
AI Agents Create Dynamic External Dependencies
An autonomous AI agent does not have a fixed set of API calls. At runtime, an agent determines which data sources to query based on generated hypotheses. For example, during one investigation, the agent may only need to contact a log aggregation service and then stop. In another investigation, however, the agent may chain together data from network telemetry, application performance metrics, security event logs, or a topology graph. Therefore, it is impossible to develop a network policy that defines whatever the agent determines it needs.
AI Agents Require Multiple Domain Credentials
A cross-domain diagnostic agent needs credentials for network monitoring, application performance tools, log aggregation, security event streams, topology services, and the LLM inference APIs, storing many secrets in a single container.
AI Agents Have Unpredictable Resource Utilization
A diagnostic agent investigating a simple connectivity problem might use two hundred megabytes of RAM and complete it in ninety seconds. For a cascading failure spanning four domains, the agent could use four gigabytes of RAM to process two hundred thousand or more log entries and take up to fifteen minutes to complete. This fluid resource use makes static resource limits impossible to establish.
AI Agents Have Nondeterministic Execution Flows
Their execution flow depends on intermediate reasoning, which develops hypotheses, retrieves evidence, evaluates and refines or rejects the evidence against the hypotheses. As a result, two investigations may have the same initial problem statement. Still, the agent's execution path could be completely different and entirely dependent on what the data reveals. This dependency makes it impossible to define normal or baseline behavior for anomaly detection.
If you deploy an autonomous agent with the same assumptions you use for a normal service, the weak points usually do not show up in design review. They show up during the first messy incident.
Figure 1. Agent security zones on Kubernetes.
The Kubernetes Job Pattern: Isolation by Default
Treating each agent investigation as a separate Kubernetes Job rather than as a long-running Deployment had the greatest impact.
Initially, we created a Deployment, set a replica count, and let the scheduler manage the deployment. However, this approach didn't last. An investigation consuming more memory than expected affected all other investigations running in the same pod. A pathological case causing a long reasoning loop held onto the resources needed by other investigations. A timed-out investigation waiting for an LLM API response required recovery for the entire engine process. Running each investigation as a single Kubernetes Job gave us four properties by default: resource isolation, failure isolation, clean state, and an investigation-scoped audit trail.
Resource Isolation
Each investigation has its own container with its own CPU and memory allocation. A complex investigation that requires two hundred thousand lines of log information doesn't starve a simple one. You can set resource limits for each Job based on investigation complexity.
Failure Isolation
If a Job fails due to an out-of-memory error or an API timeout, only that Job is affected. No other investigations will need recovery. Once the Job fails, Kubernetes records the failure and moves on.
Clean State
Each Job starts from a fresh container image, eliminating the risk of state leakage between investigations. So there are no stale contexts, accumulated memory fragmentation, or leftover temporary files.
Audit Trail by Default
Each Job has its own log stream with start and end timestamps, as well as its own resource consumption metrics. As a result, if you want to debug a particular investigation returning an incorrect result, you can pull the logs for that Job without sifting through the interleaved output of a shared process.
In practice, the orchestration flow works as follows:
- Users initiate investigations using the API.
- The back end validates each incoming request, assigns a unique investigation ID, and stores the relevant metadata.
- After validation, the back end creates a Kubernetes Job for each investigation via a direct API call. When we began building out the integration, we chose a simple API-based pattern instead of a custom controller because our execution model was a single Job per investigation, with no queuing or reconciliation logic required.
- Each Job includes investigation-specific metadata, HashiCorp Vault-injected credentials, and tier-specific resource limits.
- The agent connects to the relevant data sources, performs the investigation, and streams intermediate results through the message bus.
- Final results are written to the database and surfaced in the UI.
- Once complete, the Job terminates. Kubernetes maintains the Job status for audit and debugging purposes prior to cleanup.
Below is a basic example of what a stripped-down Job spec might look like:
apiVersion: batch/v1
kind: Job
metadata:
name: investigation-{{ investigation_id }}
labels:
app: autonomous-diagnostics
investigation-id: "{{ investigation_id }}"
trust-phase: "{{ phase }}"
spec:
backoffLimit: 0
activeDeadlineSeconds: 900
ttlSecondsAfterFinished: 3600
template:
spec:
serviceAccountName: agent-phase-{{ phase }}
restartPolicy: Never
containers:
- name: agent
image: "{{ ecr_image }}"
env:
- name: INVESTIGATION_ID
value: "{{ investigation_id }}"
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2"
memory: "4Gi"
The important point was not the exact YAML. It was that the Job boundary became the unit of scheduling, timeout, retry, auditability, and cleanup. When a retry was appropriate, we used a Job-level failure policy instead of opaque application retries, so that terminal failures and retryable failures could be distinguished at the Kubernetes layer. A custom controller would only become necessary once admission, prioritization, or queue back pressure needed to move into the cluster control plane.
The overhead of creating a new Job (typically two to five seconds for scheduling and container startup) is negligible compared to the overall investigation, which can range from ninety seconds to fifteen minutes. The advantages of a Job’s isolation far outweigh its startup cost.
Figure 2. Investigation lifecycle: one Kubernetes Job per investigation
Secrets Management: Containing Blast Radius in a Multi-Domain World
The secrets management challenge for autonomous AI agents differs qualitatively from traditional microservices, because the compromised container's blast radius is much wider by design.
A cross-domain diagnostic agent requires at runtime: LLM inference API keys, log aggregation credentials, network monitoring authentication tokens, cloud storage access keys, and potentially topology graph database and security event stream credentials. In our system, a single agent container spans four or five distinct infrastructure domains.
If that container is compromised, the attacker gains access to your entire operational stack. This situation is not hypothetical. Containers running LLM-powered workloads making outbound HTTPS calls to multiple third-party APIs are highly network-connected. We addressed this risk using HashiCorp Vault:
Dynamic, Short-Lived Credentials
Agent Jobs authenticate with Vault at startup using their Kubernetes service account token, receiving credentials scoped to the investigation's duration. Once the Job completes, the credentials become invalid. The investigation's runtime, therefore, bounds the window of exploitation for a compromised container.
Distinct Secret Paths per Domain
Separate access paths for each domain, such as network-monitoring credentials, log-aggregation tokens, and LLM API keys, have their own audit trails. This separation allows clear tracking of when and by which investigation domain credentials are accessed.
No Secrets in Git or Environment Variables at Rest
The Vault agent injector handles authentication, retrieval, injection, and revocation. Even if someone pulls an ECR image or searches Git history, there is no valuable information.
Rotating Credentials Without Redeployment
If you decide to rotate an LLM API key or a monitoring service token, you update Vault. The next time a Job is started, it will pick up the new credentials. There will be no Helm upgrade, no ArgoCD sync, and no Deployment rolling restart. These facts matter more than you might think when you are managing credentials for more than half a dozen external services.
We debated assigning a unique Vault identity to each investigation or a single agent identity. We chose a single-agent role with per-domain policies because the operational complexity of managing unique Vault identities for each investigation far outweighed the marginal security benefits. The Job isolation pattern already limits the temporal blast radius. Vault limits the credential blast radius. Together, they provided a sufficient security posture for the first production phase. Short-lived credentials reduce exposure time, but per-investigation identity improves attribution. Those controls are related but different and the distinction became clearer as investigations entered our post-incident workflow. Let’s revisit this decision in the "What We Would Do Differently" section.
In practice, the per-phase Vault policy looks something like this:
# Phase 1 (Shadow): read-only data source credentials
path "agent/data/datasources/*" {
capabilities = ["read"]
}
path "agent/data/llm/inference" {
capabilities = ["read"]
}
# Phase 3 (Limited Remediation): adds scoped write credentials
path "agent/data/remediation/pod-restart" {
capabilities = ["read"]
}
path "agent/data/remediation/cache-clear" {
capabilities = ["read"]
}
An agent codebase requests the same Vault paths in every phase. What changes is whether Vault grants or denies those requests. Phase promotion is a Vault policy update and a Helm values change, not a code deployment.
The Four-Phase Trust Model: A Graduated Access Framework
Giving an autonomous agent production credentials on day one is corporate suicide, whether or not the agent is good. People operating the infrastructure will need to build their confidence incrementally. We created a four-phase trust model that provides platform teams with a clear, observable path to incrementally build confidence from zero trust to fully autonomous operation.
Phase 1: Shadow Mode
The agent runs on production data, but its output goes nowhere. Diagnostic results are reviewed by a human after the fact, compared to known outcomes. The agent has read-only access to data sources and can't take any action. Phase 1 asks: Does the agent produce useful results?
Phase 2: Read-Only Assist
The agent is presented to the operational teams as a recommendation engine. When an incident occurs, the agent runs automatically, presenting its hypothesis tree and evidence chain to the on-call engineer. The on-call engineer can either accept, reject, or modify the agent's diagnosis. The agent still lacks write privileges. Phase 2 asks: Will the operational teams trust the agent's reasoning?
Phase 3: Limited Remediation
This phase allows the agent to execute low-risk remediation actions (such as restarting a pod) upon explicit operator approval. In this phase, the agent’s write privileges are scoped by specific Kubernetes RBAC roles and limited to specific API operations on specific resources. Phase 3 asks: Can the agent take safe actions?
Phase 4: Autonomous L1
The agent autonomously resolves all routine incidents, with no need for an operator to intervene. When the agent’s confidence level falls below a defined threshold or it requests remediation outside its approved scope, automatic escalation occurs. All of the agent’s activity is logged into an audit log. Phase 4 asks: Can we rely on the agent to run the night shift?
Transitions between phases are gated by operational evidence, rather than by timelines or roadmap pressure. In our implementation, the promotion gates look like this:
| Transition | Min Sample | Success Metric | Safety Metric | Human Trust Metric |
| Phase 1 to 2 | 100 shadow incidents across 3+ domains | > 95% diagnostic accuracy | Zero unsafe action attempts | N/A |
| Phase 2 to 3 | 150 assisted incidents | > 90% operator agreement on diagnosis | No out-of-scope recommendations | > 80% acceptance rate with minimal edits by operators |
| Phase 3 to 4 | 100 approved remediations | > 99% successful completion rate for all remediated actions | < 1% rollback rate for all successfully executed remediations | Escalation correctness to ops lead/platform team > 95%; sign-off from both parties required |
The human trust metric is intentionally operational as opposed to a statistical one. "Minimal edits" indicates that the operator does not alter either the original root cause category nor the proposed remediation classes but changes only wording, ordering, or supporting evidence. The sign-off for Phase 3 to 4 promotion requires agreement from both the operations lead and the platform owner, because they carry different failure risks.
Promotion criteria should be combined with demotion criteria. The system will revert to the previous phase (pending review before re-promotion) based on one of three conditions:
- Diagnostic accuracy falls below ninety-two percent over the last thirty days.
- Rollback rates exceed two percent.
- The agent acts outside its permitted scope.
A framework without rollback rules is incomplete.
Every team's numbers will be different. Calibrate based on how many incidents you see, how severe they tend to be, and how much risk your organization tolerates.
Figure 3. Graduated trust model: infrastructure permissions by phase.
The infrastructure implication of this model is that your Kubernetes RBAC, Vault policies, and network policies need to be parameterized per phase. The same agent codebase runs across all four phases; what varies is the level of permissions provided to the agent through your infrastructure. Each phase has its own Helm values files, Vault policies, and network policies. This is where GitOps proves its value: All permission changes are documented, reviewed, and auditable in a Git commit.
Observability for Non-Deterministic Workloads
Standard observability approaches break down for autonomous agent workloads because the fundamental unit of work is an iterative hypothesis-evaluation-refinement cycle.
We run Prometheus for metrics, Grafana for dashboards, and Loki for log aggregation, which together form a standard cloud-native observability stack. However, the dashboards we've created are very different from those used to monitor traditional services.
Investigation-Level Metrics, Not Request-Level Metrics
We track investigations initiated, completed, and failed. We collect the average time to complete an investigation as a function of domain complexity. We collect the number of hypothesis evaluation cycles performed per investigation. We don't collect p99 latency for this type of workload because investigations can take anywhere from ninety seconds to fifteen minutes.
LLM API Consumption as a First-Class Operational Metric
The agent's reasoning depends on calls to external LLM inference APIs, so that your LLM provider's rate limits and latency are directly in your critical path. We track the total number of LLM API calls per investigation, the average time to complete each call, the number of tokens consumed per call, the error rate per call, and the cost per call. A spike in any of these metrics is a sign of a pathological investigation before it fails.
Investigation-Level Cost Attribution
The total cost of an investigation includes LLM inference, Kubernetes compute, retrieval and tooling, and storage or egress. Based on mid-2026 prices for frontier-class inference models, if a simple investigation were conducted using one domain that consumed approximately fifteen thousand to twenty-five thousand input tokens and three thousand to five thousand output tokens via four to eight LLM API calls, then the inference cost would be fifteen to forty cents. On the other hand, a complex multi-domain investigation could consume eighty thousand to one hundred fifty thousand input tokens and ten thousand to twenty thousand output tokens via fifteen to thirty LLM API calls, resulting in an estimated inference cost of one and a half to four dollars.
If you include the compute costs of the Job container within Kubernetes for a complex investigation, the total cost would range from two to six dollars. Therefore, an operations team conducting fifty investigations per day may see estimated monthly costs of three thousand to nine thousand dollars. The top ten percent of complex investigations account for over sixty percent of total spending. Without attribution for each individual investigation, this spending is hidden, embedded in aggregated cloud bills and LLM API invoices. Note that industry data shows inference costs declining roughly ten times per year for equivalent model performance, so the absolute numbers will shift, but the relative cost distribution across investigation types is durable.
Reasoning Depth as a Health Signal.
We track how many hypothesis-evaluation iterations each investigation takes. If an investigation requires more than a specified number of iterations without converging, the agent may become stuck in a reasoning loop. This fact provides a way to "circuit break" the agent from getting stuck, as you would with a deterministic service. For example, for simple, single-domain investigations, we circuit-break after eight cycles or twice the p95 cycle count of successful runs in that tier, whichever comes first. For multi-domain investigations, the ceiling is fifteen cycles or twice the p95. At that point, the agent escalates with its full evidence trail rather than continuing to consume tokens in a low-probability search.
Per-Job Log Isolation
We isolate the logs produced by each Job so that when an investigation produces incorrect results, you can pull that Job’s logs and view the exact reasoning chain the autonomous agent used to produce incorrect results. This approach provides both a method for debugging issues related to the autonomous agent's operational behavior and auditing the reasoning behind a result.
Our observability toolset monitors both the operational health and the reasoning quality of autonomous agents. You need both monitoring types to operate autonomous agents in production environments.
Deployment Pipeline: GitOps for Agent Workloads
For agent workloads, GitOps matters less because Helm and ArgoCD are best practices and more because agent risk lives in configuration. Once the trust phase, RBAC scope, Vault policy, network egress, and resource limits vary across dev, staging, and production, you no longer have a single deployment state. Instead, you have a matrix of security states. Four phases across three environments produce twelve distinct permission configurations, and each must be version-controlled and reviewable.
# values-production-shadow.yaml
trustPhase: shadow
agent:
serviceAccountName: agent-phase-shadow
rbac:
verbs: ["get", "list", "watch"]
resources: ["pods", "logs", "events"]
vault:
policy: agent-readonly
networkPolicy:
egressAllowed: ["datasources"]
# values-production-limited-remediation.yaml
trustPhase: limited-remediation
agent:
serviceAccountName: agent-phase-remediation
rbac:
verbs: ["get", "list", "watch", "patch", "delete"]
resources: ["pods", "logs", "events"]
vault:
policy: agent-remediation
networkPolicy:
egressAllowed: ["datasources", "remediation-targets"]
It is important to note that promoting an agent from shadow to limited remediation in production is a Helm values change and a Vault policy update reviewed in a pull request, not a code deployment.
In that model, drift detection is a required safety control, not a nice-to-have control for a convenient automated deployment. If an administrator were to broaden an agent's RBAC policy, for example, while troubleshooting an issue and then were to forget to revert it, the ArgoCD tool should ideally detect and automatically reconcile that drift, because for an agent-based workload category, configuration drift is identical to blast-radius drift.
What We Would Do Differently
If we started over, we would use Vault identities per investigation from day one. The operational complexity is manageable with automation, but the forensic value of knowing what credentials were created for what specific investigation is worth investing in.
We would also build investigation-level cost attribution earlier. As described in the observability section, the per-investigation cost variance is significant. By attributing those costs to individual investigations and the incident categories that generated them, operations teams can decide which categories are cost-effective to automate and which should remain human-triaged.
Lastly, we would have included the four-phased trust model into the infrastructure at the onset, rather than retrofitting it later. Using environment-specific Helm values is not sufficient; you need to define phase-specific Helm values for each environment. The process of promoting between phases differs from that of promoting between environments.
Conclusion
Autonomous AI agents are coming to your Kubernetes clusters, whether or not your security model is ready for them. Teams deploying them will start with the default patterns: a Deployment, some environment variables with API keys, and a prayer. The difference between this baseline and a production-quality security posture is built on the same cloud-native building blocks: Jobs for isolation, Vault for secrets, RBAC for access control, GitOps for auditability, and standard observability tools focused on non-standard metrics.
The harder problem is organizational, not technical. To give an autonomous agent access to production resources, you need a trust framework that allows operations teams to build confidence incrementally. While the four-phase model described here is one approach, the specifics matter less than having an explicit, observable progression. Begin by auditing your current Kubernetes security assumptions against the four properties from Section 2: static dependencies, single-domain credentials, predictable resource assumptions, and deterministic execution. If your RBAC, fault policies, and network policies were designed for any of those assumptions, then that is where your gaps are. First, fix the boundaries, and then ship the agent in shadow mode, and let the data decide when to promote the agent.