Last week I wrote about an incident OpenAI published from its own training environment. Their monitoring caught a model reaching the open internet in twelve minutes, and the run kept going for another two and a half hours, because it did not stop automatically and nobody was sure who was allowed to stop it.
Days later we had an incident of our own, in our own production account, and it ran in the opposite order. The stop came first. We found out almost four hours afterwards.
What happened
We run a company knowledge layer that syncs our business systems into one index, so every AI tool we use can answer questions from the same place. One piece of it is a background worker on AWS Lambda that scans our HubSpot records and keeps that part of the index current. Codex, OpenAI’s coding agent, wrote it.
The worker had a reasonable idea built into it. A full scan of HubSpot takes longer than a single Lambda run is allowed to last, so when the worker ran out of time it invoked itself to pick up where it left off. That pattern is old, and it is fragile. When a scan did not finish, the next invocation started the same work again. Runs hit the fifteen-minute limit, the timeouts triggered automatic retries, and the hourly scheduled runs started on top of all of it, competing for one execution slot. The worker had built itself a long chain of calls to itself.
At 1:24 p.m. Pacific, AWS stopped it. Lambda has a feature called recursive loop detection, switched on by default, which notices when a function has run roughly sixteen times in the same chain of requests and drops the next invocation. Nobody on our team wrote that rule or configured it. It was simply there.
The notice took longer. AWS’s own documentation says it can take up to three and a half hours to appear on the account health page, and the alert reached me at 5:15 p.m., almost four hours after the stop. The stop did not wait for the notice, and it did not wait for anyone.
How it reached me
The alert came in through Dots, the OpenAI assistant we run on our operations, which flagged it to me in a chat thread with a link to the AWS notice: recursive invocations stopped in production, cause and affected service not yet confirmed, needs an engineering owner.
I pasted it into Codex and asked it to find the cause. It started with the worker we had added most recently, which was the obvious suspect and turned out to be clean. Within about three minutes it had the real cause from the dropped-invocation metric: the HubSpot sync worker and its self-invocation chain. It also noticed that AWS had stopped one chain while later runs were still starting, so it paused HubSpot background indexing to contain the retries. Everything already in the index stayed available. It just stopped getting fresher.
I sent back one line. Resolve. Test. Build safety rail.
Once the DynamoDB, S3 and Bedrock billing settled alongside Lambda, the whole thing came to about $3, and that includes the indexing the worker was supposed to be doing.
Three things worth taking from it
Agent-written code fails like any other code. Nothing about this bug was exotic. A self-invoking resume pattern under a hard timeout is a mistake people have been making with serverless functions for years, and the agent wrote what a hurried engineer might have written. Agents will write bugs. The useful question is what stands around the code when they do.
The control that worked lived outside the code. It was a platform limit, on by default, that drops the next call after about sixteen in a chain. It did its job almost four hours before any of us knew there was a job to do. Compare that with last week. OpenAI’s alert fired in twelve minutes and the stop took two and a half hours, because the stop depended on a decision. Ours depended on nothing, so it took no time at all. That is the case for putting limits in infrastructure: a rule the platform enforces does not need anyone to be awake.
It also had edges. AWS stopped one chain. The hourly runs and the retries that kept starting afterwards were separate chains, so the rule never saw them. A limit covers the shape of failure it was built for, and knowing which shapes yours leave open is part of owning it.
The person in the loop was mostly carrying messages. Count the handoffs. AWS to Dots, Dots to me, me to Codex, Codex back to me. Two agents did the reading and the diagnosis, and I was the router between them. The only real decision I made was the last one: fix it, prove the fix, and add a rail so it cannot happen this way again. That decision should belong to a person. Copying an alert from one chat window into another should not, and that handoff is the one we are now designing on purpose.
What the fix was
The fix was a rail. The worker no longer calls itself. Scans moved to bounded, scheduled work that saves its place with checkpoints, and retries got a ceiling. HubSpot indexing stayed paused until the replacement passed a check in production, with everything already indexed still available in the meantime. Then we applied the same rail across every data source in the knowledge layer, not only HubSpot.
Look at what is missing from that list. Nobody proposed a better prompt for Codex. A prompt is how you tell an agent what it should do. A rail decides what the system can do, whatever anyone wrote. We already had one of those from AWS, and it is the reason this is a short post about $3.
Three questions for your own stack
If agents are writing or running code anywhere in your business, these are worth an hour this week.
Which of your AI-built systems would stop on its own, without a person, if it started looping, spending or retrying?
For each limit you have, what shape of failure does it cover, and what does it leave open?
When an alert fires, who carries it to the thing that can fix it, and is that person making a decision or forwarding a message?
Sources: Metacto production incident, October 2026, from the AWS notice, Lambda metrics and execution logs. AWS, “Lambda recursive loop detection”, AWS Lambda Developer Guide.