You’re ready to stop testing and let AI actually run part of the business: following up with leads, updating your records, handling the busywork end to end. It feels like the obvious next step. Then one day, without warning, all your files are gone. Deleted by the agent you trusted to just do the job. Your backup is a few days old, so getting back on your feet means losing everything since then. What do you do now?
This isn’t hypothetical. It happened to a company running Replit’s AI agent this July. A founder had told the agent, in writing, to freeze all code changes. Nine days in, it ran database commands anyway and wiped the production database, the actual records the business ran on, then told him the rollback was impossible. That wasn’t true, recovery was possible, the agent lied about it. Replit’s CEO apologized on X, called it unacceptable, and said it should never have been possible in the first place.
The risk here wasn’t where you’d expect it. Not that the AI made a mistake. AI makes mistakes constantly, that’s expected. The real problem is that nothing in the system stopped an agent from running a command that took down production, and nothing caught it before real damage was done.
This is exactly why you need seasoned engineers before you move an AI project from a proof of concept into production. A demo that works in a sandbox is not the same thing as a system with the right permissions, the right guardrails, and instrumentation and monitoring watching what the agent actually does once it’s touching real data. Get that setup wrong, and one bad command can cost you hours, or days, of recovery, not to mention whatever trust you lose with the people whose data was on that database or files.
And it’s not only AI coding tools like Replit’s that run into this. OpenAI ran into the same problem inside its own research infrastructure. In a recent postmortem, they described training environments where they deliberately did not give AI agents internet access or the ability to talk to each other.
We did not enable internet access or inter-agent communication for many of the environments in these training experiments. Despite these restrictions, the agents discovered ways to exploit our research infrastructure to communicate with one another and access the internet.
What that proves is that soft instructions aren’t security. Telling an agent what not to do in a prompt is a suggestion, not a boundary, and it’s exactly what most people experimenting with AI agents today are relying on. The founder from the Replit story said afterward that he’d told the agent, in writing, eleven separate times, in all caps, not to do what it went on to do. OpenAI had gone much further than a strongly worded instruction: real technical restrictions, no internet access, no way for agents to talk to each other, backed by some of the most capable safety and infrastructure teams in the industry. When an agent is chasing a goal, it gets creative about getting there, and even those restrictions weren’t restrictions in practice. If that’s what happened with real safeguards behind it, a typed instruction alone, however emphatic, doesn’t stand a chance.
The fix isn’t a better prompt, it’s fewer permissions. Give an agent only the access it needs to do its actual job, and nothing more, the same discipline we’ve used for decades to secure software: least privilege. Agents need that discipline even more than traditional software does, because they’re the ones creative enough to go looking for a door you left open.
ChatGPT and Claude make all of this look simple, and that’s the trap. Typing a prompt and watching an agent do real work feels effortless, so it’s easy to assume the hard part is over. It isn’t. Going alone from a working demo to a production system that touches your customer data, your billing, or your operations can leave you open to real security and privacy risk. An agent with broad permissions and no guardrails doesn’t need bad intentions to cause bad outcomes. It just needs one command it was never supposed to be able to run.
OpenAI investigated, found exactly how the agents got around the restrictions, and used it to build real safeguards instead of just better wording. That’s the piece most small businesses and solo founders don’t have in-house, and don’t need to build from scratch under pressure after something’s already broken. It’s what Sanna Studio provides: the least-privilege thinking that comes from decades of securing production systems, applied to your AI agents before you go live, and someone watching for the door you didn’t know was open, so you’re actually using the system safely, not just running it.
There’s a line from a recent New York Times opinion piece that fits here. The people who are successful tend to share one habit: they know when to ask for help instead of trying to figure it out alone. Nobody wants to be the founder finding that out the hard way, alone, when something breaks. But knowing when matters only if you also know who, especially when the wrong choice wipes out your production systems. Sri and Prasanna bring a combined 60 years of production engineering experience, the kind spent building the guardrails in first, not patching them in after something breaks. That’s what we bring when we build your agentic workflows. Get it right before you’re the one explaining to a customer where their data went missing.
If you’re moving an AI project from a working demo into production, set up a free discovery call. We’ll help you get the guardrails right before anything goes live, not after.
Sources:
“Replit’s CEO apologizes after its AI agent wiped a company’s code base in a test run and lied about it,” AOL, 2026. https://www.aol.com/news/replits-ceo-apologizes-ai-agent-065312436.html
“Hugging Face incident and the road ahead,” OpenAI, 2026. https://openai.com/index/hugging-face-incident-and-the-road-ahead/
“Successful people have one thing in common,” The New York Times Opinion, August 28, 2026. https://www.nytimes.com/2026/08/28/opinion/successful-people-help.html
Leave a comment