Transcript
Vinoth Govindarajan: This talk became possible by reading a lot of GitHub Issues over the weekend. I know based on your personality, either call it research, therapy, or cry for help. I know when you build production systems, you know the truth. Most of the important lessons lie in these bug reports and issues. Let's start with one sentence. A user saw the reply. The system forgot it happened. This is a failure shape I want to start with. Not a model hallucination, not a bad answer, not a crash. A system did something that looked successful at the user-visible edge, but the durable record behind it had a hole. Here in the screenshot, you can see the user asked for the agent to remember a refund for a customer. This assistant said it is going to remember for the next turn. From the user's point of view, there is no obvious errors.
There is no failures. There is no red screen. Everything looked healthy. Behind the scenes, the system could not reliably reconstruct the future for the next turn. The action happened, but the record that should have made part of the agent's durable memory did not. For production agents, these kinds of bugs matter. The transcript can look coherent, but the reality changed in the wrong order or not recorded at all. Why is this worse than a crash? Crash is annoying, but at least it gives you a boundary. You see something stopped. You obviously see an error, and you can often replay from the last known good point. Silent success is worse. It's a lie. The channel says success. The user sees something happened, but the operator has no reason to doubt that the system lost part of its memory. The OpenClaw example I am going to use later has the exact same shape.
The user-visible delivery path looked healthy while the turn was not recorded in the persistent path the future context would depend on. That means the future context inherited a hole. It may answer confidently, but it is reasoning over an incomplete record. This is why I do not want to start with a model benchmark. A benchmark can tell you about model behavior. It cannot tell you whether persistent edge and the delivery edge agree. In production, it matters. Once X can become an action, the question changes. In ChatGPT, it is just a question and answer. In Codex, it performs an action. It's no longer only, did the model answer correctly? That still matters, but it is not enough. Once the system can send a message, update a database, run a command, run a workflow, or trigger a command. You need to ask three production questions. Who owned the state?
Which memory pipeline or workflow owned the state? Which is the source of truth? Who committed first, the order? Whether the transcript, session, memory, or a workflow record? When two events arrive together, you know which decided the order. Who can show what happened? Not what the model intended, not what the agent said. What the existing user-visible edge persisted. These are not model questions. These are production questions.
Background
I'm Vinoth. I'm part of core data and AI infrastructure at OpenAI. My background lies in distributed systems at Uber and Apple, where I built systems which care about state, reliability, and scale, and the mechanics that keep the system alive under load. Outside of work, I used to write a lot of open-source projects, contribute to open-source projects such as Apache. After the agents took over the job, I started writing agent stack and substack, where I write about how AI agents and data systems work in production. I'm not the maker of OpenClaw or it is not an OpenClaw product pitch. I study OpenClaw as a public case study, because its harness and the open nature made the harness visible so that you can take and extract a review model for your own systems. I use the same model to build internal systems at OpenAI. If you remember only three things from this talk, make it these.
Own the state. A fact needs one owner and one replay path. If your system cannot reconstruct the fact later, it does not really own the fact. Order the mutation, concurrency is fine. Accidental interleaving is not. Agents can turn out work, read in parallel and call sub-agents or tools, but the shared state needs one mutable commit path. Prove the action, the transcript is not the receipt. The transcript can tell you what the agent said or what the model intended. You need to know what was attempted, what was approved, and what was committed at the user-visible edge.
The Model is the Engine
I'll use one metaphor in this talk. I'll not abuse it. Let's start with this. Here's a car version of the stack, an agent harness I want you to take away. The model is the engine, it matters. Nobody buys a production car only by looking at the horsepower. You also care about the steering, the brakes, the dashboard, the black box, and the transmission. Agents are similar. Model gives you capability, but the agent harness gives you control. A powerful engine with no brakes is not autonomy, it is a liability with good acceleration. When I say harness, I mean the system around the model. The system around the model, that lets the model output more safely in the real world. Here's the production contract for the whole talk. A model proposes, the harness commits, and the receipts proves it. A model can propose a tool call, a message, an edit, a command or a plan, but the model should not be the production boundary.
The harness has to decide whether the proposal belongs to the write state and whether the mutation is ordered, and whether the work is bounded, and whether the authority is valid, and whether the outcome can be proven. The receipt is the evidence that survives after the run. What woke up the system, what state it inherited, what was proposed, what policy evaluated, and what executed, and what the user-visible edge confirmed. OpenClaw is the case study, and the contract is the takeaway. One important fact before we get into the examples. Many of these failure modes are not new. Distributed system engineers already know about idempotency, retries, locks, ordering, and state boundary. What changes is with the agents where the failure sits. The model can choose the tools dynamically, and they sit around the probabilistic planner. The context may be rebuilt differently for each turn. Events can arrive from users, timers, webhooks, sub-agents at the same time. System can act across more surfaces. The failures are familiar. The agents setting make it easier to trigger and harder to explain. This is why harness matters. It turns familiar reliability concerns into explicit boundaries around non-deterministic behavior.
The Harness Blueprint
This is the harness blueprint I want to start with. Every agent we know of, like personal agents such as OpenClaw or Hermes, or coding agents such as Codex, or Claude Code, uses the same model. Events can enter from multiple surfaces, from webhook, timer, chat, or external system, or internal hook changes. The control plane maps the event into a session key, and session key determines the state boundary. Session lane gives you a single writer per the commit path. While global throttle protects the system, the runtime calls the tools and models. The audit trail becomes a run receipt. This is the blueprint: event, session key, lane, throttle, tools, and the audit. The incident is proof that each of these boundary matters. Before we go into incidents, one more detail matters. The runtime does not remember in human sense. The runtime is usually stateless, and you need to assemble for each turn. Usually, the working set may include session transcript, a session state, memory or summary, and policies and tools. If any of these states are missing, the model can still answer correctly, but it is reasoning over incomplete reality. That's why the state ownership matters even before the model quality enters the picture.
1. Own the State
First, let's talk about state. If the agent depends on a fact later, it needs one owner and one replay path. By owner, I do not mean a human owner, I mean the system boundary whose persistent record becomes a source of truth for the next turn. For an incident, I'll start with what the user sees and what the system boundary broke. This is the first incident I want to start with. This is the same incident I started with earlier in the talk. The user sees the assistant replied. The system state is the durable context missed the turn. The harness lesson is delivered is not remembered. The incident is simple. The user saw the reply, but the system forgot it happened. In the OpenClaw issue, the channel path looked healthy, but a Telegram delivery succeeded. The turn routed through a CLI backend that was not persisted into the expected context or session record.
This means the user-visible success without durable memory. The dangerous part is not one path failed. The dangerous part is the failure was quiet. The operator sees a working assistant, but the user sees a reply and the logs look healthy, but the future context inherited a hole. This is not a model hallucination or a state ownership problem. This question becomes which components own the truth and what replay path it provides happened. Once the delivery edge on the context edge disagree, continuity becomes an illusion. The pattern we could extract is state is not a storage. Storage is where just the bytes live. The state ownership is the boundary for recovery. A transcript tells you which logs should be appended. Memory pipelines tells you which process can ingest it or compact the turn. A replay path tells you how to reconstruct the fact later. The owner of the record depends on the fact.
For an event, a calendar system becomes a state. Support status belongs to a ticketing system. While user preference belongs to a memory, and a conversation turn belongs to a transcript log. For production systems, the agent runtime is usually stateless between turns. It has to rebuild the working state from the persistent state. If the upload path silently fails, the next turn may be coherent but wrong.
In the second incident, the heartbeat becomes a state. The user or operator sees the agent goes quiet for hours. In fact, after 64 hours, they discovered this and then filed the bug report. The system state is HEARTBEAT_OK becomes pending for delivery and keep refreshing the skip window. The harness lesson is internal control signals should not become user-facing work. A heartbeat is supposed to be a liveliness mechanism. A timer fires, an agent wakes up, and if nothing interesting happens, it has to suppress the output and it has to treat this as an acknowledgement. In this failure, the heartbeat token crossed the wrong boundary. HEARTBEAT_OK was a liveliness token. It meant nothing to delivery. The system showed as a pending delivery. The delivery kept refreshing the session timestamp. The next heartbeat saw that there is a pending work already happening, so it skipped. The bug was not the model failed.
The bug was a state machine classified an internal state into a user-deliverable work. The fix did not make the agent smarter, it made the state machine stricter. Before you process work, decide whether it is actually work. The general lesson is this, replay is not a reliable memory until a named owner can replay it. System has to process a turn, it has to name the owner or system of record and it has to make the replay possible. The fact exists only in a successful delivery log, if there is not enough. If it exists only in the return value, it is not enough. If it exists only in the user memory, that is definitely not enough. The first review question everyone should ask is, for every fact your agent might use in the future turn, who owns it and how you can replay it.
2. Order the Mutation
Once we know who owns it, the next question becomes who can change it and in what order. Agents make this harder because multiple inputs can arrive together at the same time. User correction, a webhook replay, a heartbeat, a sub-agent completion and tool result can all be valid events, but all valid events can interleave in invalid ways. If two writers can touch the same mutable boundary, the harness should own the order. Here in this incident, there is two correct writes. A user sees a commitment or a correction disappears. In the system state is, two loaders load the same state and save competing versions. The harness lesson should be the last writer wins is not a consistency model. This incident is a classical distributed shape incident, two correct writes and one wrong outcome. In the OpenClaw bug report, they are describing a commitment store where the mutators followed load, modify, and save pattern.
One on-caller loads, modifies A, and saves. Another caller loads the same original state, modifies B and saves. Neither writer is malformed, neither caller is obviously wrong. Because there is no serialization around the read-modify-write cycle, the last user save can erase the first user's modification. The pattern you should follow is serialize same-process writes with the queue and protect cross-process writes with the lock. The model is not involved here, the harness is. The pattern here is one ordered commit path. The pattern is not no concurrency. That would be too slow. That would miss the point as well. We can parallelize reads. You can sign out subtasks. You can let tools fetch independent evidence in parallel. We can run many sessions at the same time. The invariant is very narrow and important, one ordered commit path per mutable state. If two operations are reading, fine. If two operations are preparing the candidate output, fine. If one mutable state needs same session, same memory record and same commitment store, then the harness needs to define the order. Without the order, user experience is timing as personality. Sometimes the agent feels successful. Sometimes the agent feels haunted.
This is where the lane makes the invariant concrete. The invariant does not know concurrency. That would be too slow. You can run many sessions in parallel. You can fan out read-only work. The invariant is one active writer per mutable session boundary. The architecture is parallel across sessions, single writer inside a session. Global throttle can protect the whole system. This is where the order the mutation became operational. The user experiences cue semantics as agent's personality. If ordering is broken, the agent feels haunted. The general lesson we could extract here is simple. Order is a product feature because user experience is ordering feature as agent behaviors. They do not see your internal cues and they do not know which process held the file lock. They only see an assistant. If the writer is not serialized, then the agent feels forgetful. The correction may disappear. An earlier state may overwrite a later one.
If the stuck lanes are not detected, the agent feels dead. A user sends a message. The system is still waiting behind old work. Completion is treated as logging instead of delivery. The agent feels confused. It may deliver a task as finished when the user never saw the result. The design rule is simple, serialize shared writes, detect stuck lanes, and treat completion as delivery. Once an agent can act, ordering is not an implementation detail, it is a part of a product feature.
3. Bound the Work
The third lesson I want us to talk about is bound the work. In production, silence cannot be neutral. No tool call, lane, loader, or stream gets infinite time. A production agent cannot treat silence as neutral. Silence has to become something the system can reason about, success, failure, or cancellation. Otherwise, one missing result can hold the entire session as hostage. Here in this incident, is a dangling tool call, the user sees an agent stuck. Inside the system, the log had a tool call, but it has no tool result. The lesson is simple. Silence is not a terminal state. The session log ends with a tool call. The matching tool result never arrives. Maybe the process was killed. Maybe the network connection dropped. Maybe the tool timed out before the result was returned. The detail matters for later debugging. The production failure is simpler. The loop has recorded that it called the external tool, now it is waiting for an output that would never arrive.
Future messages enter the session, but loop is still waiting behind the silence. To the user, it looks like the assistant is slow or stuck. For the system, there's no terminal event to move past. That's why tool call needs a life cycle. If the result is lost, the harness should terminate the end result and record an error. Every boundary needs an ending, success, failure, timeouts, and max attempts. That's why if your tool keeps on retrying without max attempts, then it might run forever and not stop. A lease gives runs and tools a deadline. A watchdog turns stuck work into a visible agent failure. An interrupt gives a user or an operator a recovery path outside the stuck run. Receipt records a terminal state, so the next step does not have to guess. This matters because agent calls external system, API hangs, browser freeze, and shell commands block, model streams fail.
File orders take longer than expected. The harness has to decide when the world does not answer. The general lesson here is bound the work before the work bounds you. Runs need deadlines and cancellations. Tools need timeouts and error results. Channel needs recovery path that do not get stuck behind stuck work. This is one of the easiest places where demos can fool us. In a demo, API returns, browser responds, the stream finishes, and two results appear. In production, one boundary eventually goes quiet. The harness should not ask the model to recover from the missing semantics. It should give the model a clear terminal event to reason over.
4. Constrain Authority
Now we can move from state to authority, because chat becomes risky when it becomes an action. Approval is a scoped object. It is not a memory of a click. This is where the chat becomes an action. A model can request a tool call, but it should not become the boundary. The harness decides whether the request is allowed and it can become reality. The boundary is not just whether the tool is offered to the model. The boundary is who is acting, under what scope, under what arguments, for how long, and with what evidence. In this incident, the approval lost its scope. User sees this, I approve this, why it is stuck, why it is not continuing, and why everything is blocked. The system state is the approval click existed, but the scoped execution context did not serve it. The harness lesson here is approval is not a button.
It is a scoped execution state. User approved elevated execution in this incident. That's a human decision. The follow-up run lost availability snapshot needed to carry out the approval through the execution path. In this issue, an expired callback kept retrying and it did not finish. The mistake is treating approval like a vague memory that a human was near the system. Approval needs shape. It needs actor, session, tools, argument, lifetime, and outcome. If any of these fall across transport replay, the harness no longer knows what was actually approved. Authority has a shape. It needs actor, session, tools, arguments, and outcome. When I say a tool is offered to the model, the runtime includes the tool schema into the model available options. The model can request this, but being requestable is not the same as being executable. The system has to check the policy, identity, approval, arguments, and the scope. Approval should not bind to one pending action, not a broad permission as, or not unbounded memory of a click. Expiry should terminate, not loop. If approval is expired, the system has to record the source of truth and it should move on. This is where reliability meets security.
Second OpenClaw example I'm going to talk about here shows the capability drift. A disabled tool that still registers as not disabled. This is a failure. The policy surface and the runtime surface diverge. The UI or control plane may say the capability is not allowed, but the provider path still registers it to the runtime surface. This means the model may see the request or something the operator thought was disabled. The pattern here is requester identity, capability, and execution authority has to converge before the model sees a prompt. Otherwise, policy is just a decoration, it's not a boundary. This is where the chat becomes an action. The model proposes a tool call, but the harness applies the policy. If the approval is required, it creates an approval object. The approval is not just a button, it is a scoped object. For read-only access, you can set it to auto-approval, but if it writes or modifies, changes the behavior, then it needs explicit user permission.
If you set everything to auto-approve, then you should definitely run inside a sandbox. Only then the execution should happen. The result is recorded in the receipt. This is an important section. The model can request, but the harness decides. The general lesson here is the capability is not execution. Least privileges narrows the tool surface and scoped credentials ensures the right identity is used for the action. Approval and audit decides what happens before and after the execution. Even in the OWASP, they call out, excessive agency is one of the core risks of LLM applications. Too much functionality, too much permissions, too much autonomy, or weak human approval. These map directly to the harness. The model can reason about the boundary, but it should not become the boundary. This is where one can avoid prompt injections shaped failures. The system has to enforce authority outside the prompt where the X can become a side effect.
5. Prove the Action
Finally, even if the tool says success, user-visible edge might disagree. Prove the action. A test proves the code, a receipt proves the reality. This is the part I want you to take back to your own systems. The production agent does not need just a transcript. It needs evidence. It has to cross the same boundary the user cares about. Here in this incident, there's a tool success, but the user saw nothing. User sees nothing rendered. The system state is tool responded success. The harness lesson should be proof must cross the user-visible edge. The incident is the cleanest example of why the internal success is not enough. Tool returns success. The visible client rendered nothing. The message tool reported was send succeeded. The metadata looked like it had delivered to the current WebChat or TUI run, but the user did not see the message. This is not a small mismatch.
It changes the conversation. The assistant can later say, I already sent the explanation. From the point of view, the tool results said success. From the user's point of view, nothing happened. This is a gap between tool success and user-visible proof. If the user-visible edge did not confirm the outcome, the harness should treat the action as not proven. The difference between a transcript and a receipt is the transcript says what the agent said, the tool return says what the tool claimed. A visible receipt says what the model proposed, allowed, executed at the user-visible edge. This is where the WebChat and the TUI issue matters. Internal success is not external proof. A test proves a code. A receipt proves a reality. For a support agent, the edge might be a ticket update. For a data agent, it might be a query result delivered to the user. For a coding agent, it might be a change visible in the workspace. The exact surface changes, but the receipt principle is the same.
One thing I really like about the OpenClaw project is proof becomes a part of the review culture. Not just a unit test, real behavior proof. Screenshot shows visible channel behavior. Log shows the decision path and the outcome. Before and after evidence shows what was visible before and what was fixed. The review labels like proof required and proof sufficient makes the proof part of the workflow. This is how Peter from OpenClaw can actually create and submit 600 pull requests in a day, because he is closing the loop by checking the proof before issuing the pull request. This is important because the agent system often fails at the edges, transport, channels, retries, UI delivery, and tool execution. Unit tests can provide a code path. It may not prove the user saw the message and the channel delivered the outcome. For an agent system, the real behavior is not extra ceremony, it is the evidence surface for production.
The general lesson here is, before you ship an agent, ask for the receipt. Not just the transcript because the transcript tells you what the agent said. Not just the tool return value because the tool return will only know what happened inside the boundary. Not just the unit test because the code path can pass while the production edge fails. Ask for the receipt. What woke up the system, what state it inherited, what the model proposed, what policy allowed, what executed, and what evidence proves the user-visible outcome. This is what separates an impressive agent demo from a production system that you can operate.
Run Receipt Audit
Here is the audit I want you to run when you get back to your team. Pick one agent system, not all of them. First, state. Can I replay the fact from its owner of the record? Second, order. Can I explain the order rule under concurrent writes? Third, bound work. Can I terminate a stuck work without killing that channel? Fourth is authority. Can I exactly name the authority envelope, which is actor, session, tool, argument, lifetime, and outcome? Proof, whether I can prove what the user-visible saw as outcome. If you cannot answer these, the next feature should not be a bigger model or better tool surface. It should be the clearer harness. Let us run through the opening incident with the same run receipt. What woke up the system, a message. What state it owned, that was the bug. What executed, a channel sent. What evidence survived delivery, looked successful, but the persistent edge did not. This is the gap, is the harness failure.
Recap
Let me recap the talk on the same three ideas we started with. Own the state, who owns the fact? Which boundary can replay it? Which context path depends on it? Order the mutation, who committed first? What release is the shared state? What happens when two valid events arrive close together? Prove the action, who saw it happen? What evidence survived beyond the transcript? Can you produce a receipt that reaches the user-visible edge? These are the questions that keeps the agent sane in production. Better model can help inside the turn, but ownership, ordering, authority, bounding work, and proof are harness responsibilities. Today, you can leave with three landmarks in one audit: own the state, order the mutation, and prove the action. A model proposes, the harness commits, and the receipts proves it. This is the production contract I want you to remember. If you're building agents, do not only ask whether the model can reason. Ask whether the system can own the state, order the mutation, constrain authority, bound work, and prove action. A loop can answer a turn. The harness survives production. The useful future is not agents without boundaries, it is agents with clear boundaries so that they can be useful without becoming impossible to operate.
Questions and Answers
Participant 1: When you talk about the harness and the proof, and in the beginning that the model proposed something and you have the tracking for everything, are you talking about the reproducibility, like using the same contract, the same model, and the same record, then you can go back and reproduce the same result?
Vinoth Govindarajan: Reproducibility is also one of the parts. Since the model is stateless, you have to rebuild the context every time. Even with the same prompt, is whether you can actually produce the same results. Whether the state is properly recorded so that you can replay from the source of truth. For example, like when I'm working with Codex, I usually tell like, don't deploy this to prod. If it records that only in the memory store, it is not enough. The harness should protect from deploying production code from the local laptop. The harness should decide whether the action is allowed or not. That is what matters.
Participant 2: You mentioned that this type of architecture is used both in Codex and OpenClaw. Have you seen other approaches or other architectures and other products, open source or not? If not, it's interesting how everyone came to the same architecture. Does it mean that nothing other than this works, or it's still too early to say so?
Vinoth Govindarajan: When they initially started with this kind of model that really generalized it to multiple areas for a personal assistant, for a support agent. For everything, you need inputs from many surfaces like chat, internal webhooks, heartbeats, and all those kinds of things. The architecture pattern is very common across most of the agent applications. If you look at every coding product, you see multiple sessions. The reason they have multiple sessions is because of the context window size. It has to make sure that it is within the context. The other is if multiple tools can update the same file. Even in the OpenClaw example, I was discussing before, multiple processes are trying to update the same thing. That's the reason, within sessions, you need to protect with a lock, and within the session, you need to protect with a queue. The architecture, whatever I was explaining, that easily generalizes to every agent, like every agent follows the same pattern where you get a bunch of inputs and then you need to prepare the context, because in every agent, the model is stateless.
The API contract, which you are getting from OpenAI is stateless and you have to assemble it. That's what the context engineering is. That's where you have a system prompt and the user prompt. Some of the agents I saw recently like Hermes, it is a self-improving agent. It is actually trying to improve every day by looking at its past conversations and where a user was intervening a lot, and it course corrected. Every time it also has something called trimming where it collects and summarizes all the short-term memory session logs into a durable state, and which is important, which is not important, then it improves. The reason why it is the same architecture is because that architecture, it generalizes a lot for any agent system.
See more presentations with transcripts
