Why human validation is still the constraint

On one of the products I oversee, the client's founder ships her own pull requests. She is not a professional engineer. With a model she produces working features, and the delivery constraint on that project is not headcount and it is not her — it is review capacity. Mine.

That is the shape of the whole change, in one project.

Generation got cheap. Verification did not.

Writing code was never the expensive part. The expensive parts were deciding what to build, knowing whether it works, and living with it afterwards.

A model collapses the cost of the first activity to near zero and leaves the others exactly where they were. So the bottleneck moves. It does not disappear — it relocates to whoever has to say "yes, this is correct, ship it."

And that person now faces more code, arriving faster, written in a register that actively discourages scrutiny: consistent naming, tidy structure, plausible comments. Handwritten code signals its own uncertainty — a scrappy variable name, an inconsistent pattern, a // TODO: is this right?. Generated code signals confidence uniformly, whether or not it is warranted.

What a human still has to supply

Four things, and no current tool supplies any of them reliably.

Intent. The model optimises the request it was given. It cannot know that the reason this endpoint exists is a regulatory requirement, that the odd shape of that table is a migration you cannot undo yet, or that the client's real constraint was never stated in the ticket. Requirements live in people, and mostly they live in one person's head.

Consequence. A model will happily write a correct-looking migration that locks a production table, a retry loop that turns a blip into a thundering herd, or a change to a payment path that is fine in every case except partial refunds. Correctness in isolation is not safety in context, and context here means your system, under your load, with your data.

Boundaries. Authorisation, tenancy, rate limits, spend ceilings, what may leave the network. These are policy decisions. They look like code, which is why they get generated, but they are not derivable from the surrounding source — a model cannot infer who is allowed to see what.

Accountability. When it breaks at 3am, a person is on the call. That person needs to have understood the change before it shipped, not after. This is not a technical argument, it is an organisational one, and it does not go away with a better model.

The review problem, honestly

The naive response is "review everything more carefully," which fails immediately, because the volume went up and the reviewer did not.

What has actually worked for us:

Tier the review by blast radius, not by diff size. A 400-line change to a marketing page is a skim. A 12-line change to an authorisation check, a payment path or a migration is a line-by-line read with the author present. Generated code makes diff size an even worse proxy for risk than it already was.

Make the author explain the change, not the code. "What did you ask for, and how do you know you got it?" A person who prompted their way to a working feature can usually answer the first half instantly and the second half not at all. That gap is the review.

Gate on the categories that generation reliably gets wrong. Ours is a short, boring checklist — ownership checks on every ID lookup, no secrets client-side, a ceiling on every model-calling route, every new dependency justified. It runs on every client-facing merge and it catches things weekly.

Test as an adversary, not as the happy path. Generated tests exercise the behaviour the generated code implements. They are tautological. The valuable test is the one written by someone asking what would break this — a second user who should get a 404, a malformed payload, a duplicate webhook.

The uncomfortable second-order effect

Reviewing generated code is harder than reviewing handwritten code, and it is also less rewarding. There is no author to teach, so the reviewing loop stops compounding into team capability. Do it long enough and you get a team that can produce a great deal of software while understanding less and less of it.

I do not think the answer is to slow generation down. I think the answer is to be deliberate that a portion of work stays hand-built by people who need to grow, and to treat review as the senior skill it now is rather than a chore delegated to whoever has time.

Where this lands

The tools are genuinely good and I would not give them up. But "the AI wrote it" has never once been an acceptable answer to "why is this in production?" — and the person who has to answer that question is still the constraint.

That is not a temporary state pending a better model. Verification is a different problem from generation, and we have only automated one of them.

---

Related: the security risks nobody checks in vibe-coded apps — the specific list I review against.