The short answer: AI coding agents are genuinely useful on a production codebase, but not for the reason the marketing says, and the measured evidence is mixed — in METR's July 2025 randomised trial, experienced developers took 19% longer on tasks with early-2025 AI tools while believing they had been about 20% faster. What closes that gap is not a better model, it is repository discipline: a maintained instructions file that documents your conventions and their traps, tasks with a verifiable finish line, and a rule that every convention ships with a detector an agent can run. An instruction an agent can check beats one it has to remember.
There are two confident positions on AI coding agents and both are wrong. One says a small team now ships like a large one. The other says the output is slop and the whole thing is a bubble. I have been running agents against this site's codebase — a Next.js app of about 680 pages with no test suite — for long enough to have a less exciting opinion: they are useful in proportion to how well your repository explains itself.
What the measured research says
Start with the uncomfortable study, because most write-ups skip it.
In July 2025, METR ran a randomised controlled trial with 16 experienced open-source developers working on large repositories they already knew well. Developers using early-2025 AI tools took 19% longer to complete their tasks than those working without.
The finding that should change your behaviour is the second one. Those developers forecast a 24% speedup beforehand. Afterwards, having actually been slowed down, they still believed they had been roughly 20% faster.
That is a 40-point gap between perception and measurement, in the direction that flatters the tool. Whatever you conclude about the speed, conclude this: you cannot evaluate these tools by how they feel.
The quality picture is similar. GitClear's analysis of 211 million changed lines between 2020 and 2024 found code duplication rising from 8.3% to 12.3% while refactoring activity fell from about 25% to under 10%. That is the signature of code being added rather than reworked — which is exactly what a model does when it cannot see that the function it is writing already exists two directories over.
Neither result says "do not use agents". Both say the constraint sits outside the model.
Where the constraint actually is
An agent arrives at your codebase with excellent general knowledge and zero specific knowledge. It knows how React works. It does not know that in your repo the theme provider must never be gated behind a mounted flag, or that your CSS is deliberately inlined, or that a particular scan has to run after every build.
Left to infer, it will produce something reasonable-looking and locally wrong. The duplication number above is what "reasonable-looking and locally wrong" measures at scale.
So the useful question is not "which agent" but: what does this repository know that it has never written down?
The instructions file, and why it has to be maintained
The convention has settled. AGENTS.md was formalised in August 2025 as an open, vendor-neutral instructions file, adopted across tens of thousands of repositories, and donated to the Agentic AI Foundation under the Linux Foundation in December 2025. Tool-specific variants exist alongside it — Claude Code reads CLAUDE.md, Gemini reads GEMINI.md, Copilot reads .github/copilot-instructions.md. In a monorepo, the nearest file in the directory tree wins.
The filename is the least interesting part. What matters is the rule attached to it.
This site's CLAUDE.md opens with a maintenance directive, and it is the load-bearing sentence in the whole file: if anything about the architecture, routes, conventions or dependencies changes, that file is updated in the same commit. Not later. Not in a cleanup pass. In the diff.
The reason is unforgiving. An instructions file that lags the code is not neutral, it is misinformation delivered with authority. A confidently wrong convention gets followed. A missing one at least gets questioned.
The second rule in that file is the one that compounds: record what you learn. Any non-obvious trap discovered during a session gets written down so the next session does not pay for it again.
Write down the traps, not just the conventions
Conventions are easy to write and mostly guessable. Traps are neither, and they are where the time goes. Four real ones from this codebase, all of which cost a session before they became documentation:
- HTML entities work in JSX text and silently break in JavaScript strings.
'inside<p>renders correctly. The same characters in a data array field ship to the user as literal'. Nothing fails at build time. - A context provider gated behind a
mountedflag tears down the page. Returning a Fragment until an effect flipsmounted, then returning a Provider, changes the element type at that position — so React rebuilt every DOM node on the page, on every load, right after hydration. - A third-party script host missing from the CSP fails silently. Microsoft Clarity shipped broken for months: the inline snippet ran, the injected external script was blocked, and no build ever complained.
- An hourly cron on a Vercel Hobby plan blocks every deployment. Nothing about the schedule looks wrong until deploys stop.
None of those are inferable. Each of them is now a paragraph in the instructions file, and each one an agent would otherwise re-discover at your expense.
Every convention ships with a detector
This is the rule that makes the rest work, and it is the one I would keep if I could keep only one.
A written convention is a hope. A convention with a command that proves it is a check. So each of the traps above is paired with something runnable:
| Convention | Detector |
|---|---|
| No HTML entities in JS strings | Grep the built HTML for &[a-z]+; — any hit is a double-escape shipping to users |
CSS stays inlined, never a blocking <link> | Count rel="stylesheet" in the built pages; the answer must be zero |
| Titles stay under 70 rendered characters | Parse <title> out of every built page and print the offenders |
| Sitemap dates are real | No new Date() in the sitemap, ever — grep for it |
An agent can run those. It cannot reliably remember a paragraph of prose across a long session, and neither can a person at 6 pm on a Friday. This is also why the checks live in the repo's documentation rather than only in someone's head: this project has no test suite, which is precisely why the build-plus-scan discipline had to be written down. If you have real tests, you already have the best version of this — use them as the finish line.
Give the agent a verifiable finish line
The tasks that go well share a shape: there is something that can be run at the end which is unambiguously green or not.
Good finish lines: the build passes, this scan returns nothing, this test goes green, this page renders the expected schema. Bad finish lines: "improve the performance", "clean this up", "make it more consistent". Those are judgement calls, and judgement is what you are supposed to be supplying.
The same logic applies to scope. A task that touches one cluster of files with a clear check at the end is work an agent does well. A task that requires holding your whole system in mind is work it does confidently and badly.
Review what changed, not what it says it changed
Agent summaries are written to be reassuring. Read the diff.
The failure mode is rarely a syntax error — it is a second implementation of something you already had, a convention quietly not followed, or a "fix" that suppresses the symptom. That is the duplication trend in the GitClear data showing up in your repository, one plausible commit at a time. This is also where the rebuild-or-refactor question gets decided in practice: enough unreviewed near-duplicates and you have manufactured the technical debt yourself.
Two habits catch most of it. Search for the thing before accepting a new implementation of it. And when a diff is larger than the task warranted, ask why before merging.
What this looks like in practice
The setup that works, in order:
- Write the instructions file — stack, routes, conventions, and the traps that cost you time.
- Add the maintenance rule — the file is updated in the same commit as the change it describes.
- Pair every convention with a detector — a command that proves it, not a paragraph hoping for it.
- Scope tasks to a verifiable finish line — build, scan, test.
- Review the diff, not the summary — search for existing implementations first.
- Record what you learn — every new trap becomes a line in the file.
That is not an AI strategy. It is the documentation and review discipline good teams already claim to have, made mandatory because an agent will expose the gap between claiming and having it within an afternoon.
The same question arrives from the other direction once someone asks you to expose your systems to agents, which is a scoping and permissions problem rather than a documentation one — what to know before shipping an MCP server.
The teams getting real value out of these tools are not the ones with the best model access. They are the ones whose repositories can explain themselves — which, conveniently, is the same property that makes onboarding a new human developer fast. The same instinct applies to making a site legible to AI crawlers: structure it so a machine can read it correctly without guessing, and humans generally benefit too.
Common questions
Do AI coding tools actually make developers faster? Not reliably, on measurement. METR's July 2025 randomised trial found experienced developers took 19% longer on tasks in large codebases they knew well, while believing they had been about 20% faster. Gains are context-dependent, and the perception gap means you should measure rather than trust the feeling.
Does AI-generated code hurt code quality? It can. GitClear's analysis of 211 million changed lines found duplication rising from 8.3% to 12.3% between 2020 and 2024 while refactoring activity fell below 10%. The mechanism is simple: a model that cannot see your existing implementation writes a new one.
What is AGENTS.md? An open, vendor-neutral Markdown file at your repository root that tells coding agents how your project works. It was formalised in August 2025 and donated to the Agentic AI Foundation under the Linux Foundation in December 2025. Some tools read their own filenames instead, such as CLAUDE.md or GEMINI.md.
What should go in an agent instructions file? Your stack and versions, routing and file conventions, the commands that verify a change, and — most valuably — the non-obvious traps specific to your codebase. Pair each convention with a command that proves it. Update the file in the same commit as any change it describes.
Do I still need code review if an agent wrote it? More than before. Review the diff rather than the summary, and search for an existing implementation before accepting a new one. The common failure is not broken code, it is plausible code that duplicates or quietly contradicts what you already have.
The honest summary
The model is not the bottleneck. Your repository's ability to explain itself is. Write the instructions file, keep it honest in every commit, attach a runnable check to every convention worth having, and give each task a finish line something can verify. Do that and agents are a real multiplier. Skip it and you get the METR result: work that feels faster and measures slower.
Written by Jagatjeet — Jagatjeet (jagatjeet.com) is a web architect and digital marketer in Kamloops, British Columbia, working with founders on web architecture and marketing systems. Published 7 August 2026. Last updated 7 August 2026. The repository conventions described here are the ones in use on this site's own codebase.