Why Your AI Writes Great Code and Your App Still Breaks
You prompt, the model ships, the feature works. Then something you never touched breaks in production, or a report comes back with two customers merged into one, or a change that passed every check still takes down checkout. If the code your AI writes is so good, why does the app it builds keep surprising you?
I spend my weeks reading AI-generated codebases, and here is the part nobody says out loud: the code is usually fine. Often better than fine. I recently went through an app where an AI had written the overwhelming majority of it, and the hard, boring, easy-to-get-wrong things were done right. Database access was properly locked down. The payment webhooks verified their signatures. There were real tests on the money paths. If you had told me a careful engineer wrote it under deadline, I would have believed you. Code like that is not the model getting lucky; it usually means someone knew what right looked like and held it there.
And the app was still fragile. Not because of a bad line of code, but because of everything around the code that no single prompt produces.
(If you want the symptoms rather than the cause, they are in what breaks first when a vibe-coded app meets real users. This post is about why they keep happening to code that reads clean.)
The model is a brilliant contractor with no foreman
A frontier model is an extraordinary builder. Give it a well-scoped task and it hands you clean, idiomatic, working code, frequently better than what a rushed human would write. That is not the problem, and pretending it is will only make you distrust a tool that is genuinely earning its keep.
The problem is that shipping software is not a sequence of well-scoped tasks. It is a system. Someone reviews the change before it goes live. Someone notices a file is getting too big to reason about. Someone remembers that this new feature quietly touches the thing three features ago depended on. The model does the building. Nothing in a chat window does the foreman's job, and when there is no foreman the failures are not coding failures. They are workflow failures, and they surface later, in production, where they cost the most. Here are the four I see over and over.
1. Verification that verifies nothing
That same app had a real test suite, hundreds of tests, including on the money paths, and a hundred pull requests behind it. That is more discipline than most. But the main branch had no protection, so nothing those tests caught could actually stop a change from shipping. The verification existed. Requiring it did not. It is a smoke alarm wired to nothing: it tells you the kitchen is on fire, after the house has burned.
A model will happily write you tests. It will not, on its own, install the gate that makes those tests matter. That gate is a decision a human makes once, and almost nobody makes it, because the app is working and the gate feels like ceremony. It is not ceremony. It is the one thing standing between "the model made a mistake" and "the mistake is now live." The labs building these models say the same thing about their own output: OpenAI's alignment team starts from the position that we cannot assume that code-generating systems are trustworthy or correct, so we have to check their work. Checking is part of the job, not an insult to the tool.
2. Context fragmentation
The same codebase had a single file past thirteen thousand lines. It got there honestly. Every new feature asked the model to add just one more thing to the component that already did the most, and the model, with no memory of last week and no opinion about file size, obliged.
Here is why that is dangerous and not merely ugly. A model can only reason about what fits in its context. When the next feature needs to change that file, it edits a thirteen-thousand-line surface it cannot fully see, and neither can you. The code it writes for the part it can see may be perfect. The part it could not see is where the app breaks. No phrasing of the request fixes this, because the thing being edited is simply too big to hold in one head, human or machine. The vendors name this one directly: OpenAI's evaluation guidance says many LLM-based applications fail due to poor understanding of the context of the request, not to some deficit in the writing.
There is a second way that same limit bites, quieter than the first. A model applies only the conventions it can actually see, and in this app there was no file carrying them, no versioned, reviewable record of how the codebase does things or what it must never do. Whatever rules existed lived in someone's head or a platform's memory, never in the repo where the model and the next engineer would both read them. Writing them down once, where the model reads them every time, is the fix, and no cleverer prompt substitutes for it.
3. Operator pressure and the unread diff
Nearly every AI-generated app with real users shares one trait: one person, moving fast, approving changes they no longer fully read. Not out of laziness, out of volume. When the model produces a day of work in an hour, reading every line becomes the bottleneck you were trying to escape, so you stop. The diffs pile up unread. The app keeps working, which feels like proof that reading them was optional, right up until the one you skimmed was the one that mattered.
4. Nobody owns the cross-cutting parts
Security, data integrity, performance under load: none of these lives inside a single feature, so none is ever the subject of a single prompt. They are properties of the whole, and the whole is exactly what a prompt-by-prompt workflow never steps back to look at. That is how an app can have flawless payment code and still let two customer records drift into "ACME Ltd" and "Acme Limited," quietly splitting a founder's numbers in half without a single error being thrown.
The fix is a system, not a better prompt
Notice what none of these are. They are not the model writing bad code, and they are not something you fix by prompting more carefully. "Make it robust" produces something that looks robust. The gaps are in the workflow, so the fixes are too: a gate that inspects changes before they ship, a written set of house rules, a habit of breaking big files down before they calcify, the compiler turned up so it catches what the reader misses, and someone whose job is the whole rather than the next feature.
Anthropic's engineering team, writing about long-running agents, puts the same point plainly: out of the box, even a frontier coding model will fall short of producing a production-quality app when all it gets is a high-level prompt. Their agent guidance describes the loop that closes that gap as gather context, take action, verify work, repeat. The verify step is the one a chat window does not do for you. In a real codebase, verify means a test that runs whether or not anyone remembers to run it. If you want the order we install those in, without pausing feature work, that is how to harden an AI-generated codebase.
That last one, the person whose job is the whole rather than the next feature, does not have to be a full-time hire. It is often a few hours of the right person at the right moments: before a big feature, after a scary one, and once in a while to walk the whole thing and say "here is where this hurts you at ten times the users." The model builds. The system is what keeps what it builds from surprising you. Most AI-generated apps have the first and are missing the second, and the good news is that the second is far cheaper to add than the first was to build.
If you have shipped something real with AI and you can feel the surprises starting, that is not a sign you did it wrong. It is the sign that the building phase worked and the system phase is due.
That gap is exactly what a Rescue Audit maps: what is already solid, what will break first, and the shortest sequence to make it durable without pausing your feature work. If you want to see what that looks like before you talk to anyone, here is a real rescue audit, redacted. If you want a read of your own codebase, that is what we do.