Practice

What AI coding tools actually get wrong, and what they do not

The tools are not producing novel vulnerabilities. They are producing familiar ones, faster, in code nobody read.

Dr Kelvin K Awagu23 June 20269 min read

There is a genre of commentary that treats AI written code as a new category of security risk, full of failures we have never seen. That has not been our experience and it is not what the findings look like.

The vulnerabilities are ordinary. What is new is the distribution: which mistakes appear, how often, and crucially who is positioned to catch them. A developer who writes an insecure direct object reference by hand usually knows the concept exists. A founder who prompted a feature into existence in four minutes may not.

The pattern behind most of it

A model generating a feature optimises for the feature working. Asked for “let users edit their profile”, it produces an endpoint that edits a profile. The endpoint works. You test it, on your own account, and it works.

What it does not do unprompted is ask whether the caller is allowed to edit that particular profile. Authorisation is not visible in the happy path, it is not visible in your manual test, and it is not visible in the demo. It is only visible if you go looking for it, and going looking for it requires knowing it is a category of thing that exists.

Nearly everything below is a variation on that: the security property is invisible when the feature works.

The list that actually recurs

  • Missing authorisation on endpoints that have authentication. You are logged in, so it feels protected. Nothing checks that the row belongs to you.
  • Row level security left off, or written permissively enough to be decorative, on a database that was set up in the same conversation as the feature.
  • Secrets committed early, when the project was a prototype and nobody had decided it was real yet. They stay in the history after they are removed from the file.
  • Server side checks that exist only on the client, because that is where the model was asked to put the validation.
  • Overly broad CORS, usually a wildcard added to make local development stop complaining and never narrowed.
  • File uploads with no type or size constraint, stored somewhere publicly readable.
  • Verbose errors returned to the client, including stack traces and query fragments, because that was useful while building.
  • Dependencies chosen for popularity in training data rather than for being maintained, then never updated.

None of these are clever. Every one of them appears in guidance that predates the tools by a decade. That is the point: the failure is not that the code is exotic, it is that nobody with the context to notice ever read it.

What the tools are genuinely good at

It is worth being fair about this, because a checklist written by someone who thinks the tools are useless will waste your time on the wrong things.

Generated code is generally competent at the mechanical parts of security that have one obvious right answer. It reaches for parameterised queries rather than string concatenation. It hashes passwords with something appropriate rather than inventing a scheme. It uses the framework’s escaping rather than building HTML by hand. Injection and cross site scripting, the two vulnerability classes that dominated for twenty years, are markedly less common in generated code than in the hand written code of an inexperienced developer.

The failures cluster in exactly the places where the right answer depends on your application’s rules rather than on a general convention. Who is allowed to see this. What happens when the caller is not the owner. Which of these fields should a client be able to set. A model has no way to know your rules unless you tell it, and mostly you did not tell it, because you were asking for a feature.

What to do with that

Two things follow, and they are both cheap.

First, review by category rather than by file. Do not read the codebase looking for bugs. Take the list above, and for each item ask a targeted question about your own project. “Show me every endpoint that reads a record by id, and tell me what it checks before returning it” finds a class of problem in one pass. Reading files hoping something jumps out does not.

Second, write down the answers. The value of a security review evaporates about a week after you do it unless the result is recorded somewhere. Not for the tooling’s sake, for yours: the next time someone asks what you have checked, the difference between “we looked at it” and a dated record of what was checked and what was found is the difference between a conversation that ends and one that continues.

That second point is most of why VibeLock exists. The checking is not the hard part and never was. Remembering what you checked, six weeks later, when a buyer asks, is the hard part.