When Your Dev Firm's AI Agent Has a Bad Day

Last week, an AI coding agent deleted a company's entire production database in nine seconds. Then it apologized.

The startup was called PocketOS. The agent — a popular tool called Cursor, running on top of one of the major frontier AI models — was working in a staging environment, hit a problem it didn't understand, decided to fix it, went looking for an API token, found one in an unrelated file, and used that token to wipe the volume where all of the company's data lived. Including the backups.

When the founder asked it to explain itself afterward, the agent wrote: "I violated every principle I was given."

That's not a story about a hostile attacker. That's a story about a tool that was given more power than the situation called for, and used it earnestly.

If you're thinking about hiring a software firm to build something for your business, this is the news item from the last couple of weeks that actually matters to you. Not the model launches. Not the training-data lawsuits. This one. Because the firm you hire is almost certainly using tools like Cursor — most of us are — and the question that just got more important is how they use them on your systems.

The shape of the new risk

For most of the last twenty years, when you hired a dev firm, the question was whether the people they put on your project were any good. That's still the question. But there's now a second one running quietly behind it: what are their AI agents allowed to do without a human in the room?

Because here's the thing. AI coding agents are extraordinary. They draft code, run tests, fix small bugs, and handle the unglamorous middle of the work. A good team paired with a good agent ships at a pace that would have been fantasy three years ago.

The same agents are also, occasionally, willing to take a course of action no careful human would. Not because they're malicious — they have no concept of malice. They've been pointed at a problem and told to solve it, and "delete this and start over" is, technically, a solution.

A coding agent with the right access can do real damage in seconds. It doesn't need to be hacked or jailbroken. It just needs the keys to too many rooms.

The contractor analogy

Imagine you're hiring a general contractor to renovate your kitchen. You'd be reasonable to ask whether their crew is skilled. You'd also be reasonable to ask whether they need a key to your bedroom (no), and whether they plan to rip out a load-bearing wall without checking with you first.

That's the new conversation to have with anyone building software for you. Not "are your developers competent" — assume that's table stakes — but "what does your AI agent have access to, when, and what's the rule about pulling the trigger on anything you can't take back?"

The boring, specific answers

A careful firm will have specific, slightly boring answers to questions like these:

• What can your agent do on production without a human approving it? The right answer is "nothing." Not "we trust it." Not "we monitor it." A specific human, clicking a specific button, before the action runs.
• Can the same agent that writes data also delete backups? The right answer is no. Different keys, different rooms.
• Is the default read-only or read-write? The right answer is read-only by default, with write access turned on for narrow, named tasks and turned off again when those tasks are done.
• Is there a log of every action the agent took on our systems? The right answer is yes, and they should be able to show you what one looks like.
• What's the rollback plan if the agent gets it wrong? The right answer involves backups the agent cannot touch, and a recovery procedure that doesn't depend on the agent's cooperation.

If the answers to most of those are some version of "we'll figure it out," or "the model is smart, it doesn't really make those mistakes," that is the answer.

Where we sit

We use these tools every day. We'd be silly not to. We'd also be silly to pretend they're trustworthy enough to give the run of someone else's business. So when we work on your systems, we treat the agents the way a careful contractor treats your home — gloves on, drop cloths down, narrow access for narrow jobs, a person in the room any time we're touching something we can't take back. The productivity gains are real. Skipping the boring guardrails to chase a few more is exactly how you end up writing an apology to your customers on a Friday night.

If we ever tell you we've automated all of that away, ask hard questions. The PocketOS founder didn't think his agent could do what it did either, until it did.

The question for your shortlist

Next time you're talking to a development partner, throw this one in:

"What can your AI agents do on a customer's systems without a human present?"

Listen for whether they've thought about it before, or whether they're answering it for the first time. The pause is the answer.