We run AI-assisted code review as part of our engineering quality program, alongside containerized integration testing and progressive delivery. It's been in place long enough for me to have a clear view of what it's good for and where it misleads.
The summary: it's a genuinely useful first pass, it does not replace human review, and the main risk isn't bad suggestions — it's what it does to reviewer attention.
What It Reliably Catches
The value is highest on the mechanical layer, which is also the layer humans are worst at, because it's boring and attention is finite.
Things you'd be embarrassed to miss. Null dereferences on paths the author didn't consider. Resources not disposed. An async method whose result is never awaited. Off-by-one in a loop bound. These are real defects and a model finds them consistently.
Inconsistency with the surrounding code. Naming that doesn't match the file's conventions, an error-handling pattern that differs from every other handler in the project. A reviewer who wrote that code six months ago may not notice; a tool comparing against the local context will.
Missing test coverage for a new branch. Pointing out that a new conditional has no test is simple, mechanical, and frequently right.
Documentation drift. A comment or doc string that no longer matches the code it sits above.
None of that is glamorous. All of it used to consume the first ten minutes of human review, and getting it done before a person looks means the person starts from a higher altitude.
What It Consistently Misses
The misses are structural, not incidental, and they're all variations on one thing: the model can see the diff, not the business.
Whether the change should exist. The most valuable review comment is sometimes "this requirement is wrong" or "we already do this elsewhere". That requires knowing the product and the codebase's history.
Domain correctness. In claims processing, code can be flawless and still wrong — an eligibility rule applied in the wrong order, a date treated as service date when it should be submission date. Nothing about that looks like a defect. It requires knowing how claims actually work.
Architectural consequence. That this change introduces a dependency that will hurt in six months. That it duplicates logic living in another service. That it quietly makes an interface harder to change.
Whether the tests test anything. A model sees tests exist. It's much weaker on whether they'd fail if the code were wrong — which is the only property that matters, and the most common defect in test suites I've reviewed.
The Real Risk: Attention
Here's what I watch for, and it's not the tool producing something incorrect.
When automated comments arrive first, there's a pull toward treating review as responding to the comments. Address the eight flagged items, approve. It feels thorough. The work got done, the checklist is clear.
But the questions that actually matter were never on the list, because they're not visible in the diff. Reviewing becomes reactive, and the reactive version is shallower than what the same person would have done unaided — while feeling more rigorous.
The mitigation we use is a sequencing rule: form your own view of the change before reading the automated comments. What is this trying to do, is this the right way, what breaks. Then read the tool's output as an additional check.
That ordering costs nothing and preserves the part of review that has the most value.
It Doesn't Replace a Reviewer
I'd push back firmly on using this to reduce review headcount, for a reason beyond correctness: code review is how engineers learn a codebase and how standards propagate through a team.
A junior engineer reviewing a senior's PR learns more than from any documentation. A senior reviewing a junior's work transfers judgement that no tool transfers. Reviews are where "how we do things here" is actually taught. Remove humans from that loop to save time and you've saved review hours while quietly stopping the mechanism that develops engineers.
The tool raises the floor. It doesn't raise the ceiling, and the ceiling is where your senior people operate.
How I'd Set It Up
- Run it before human review, so the mechanical layer is clear by the time a person looks.
- Keep suggestions advisory, not blocking. A tool that can block merges becomes a thing people argue with and then route around.
- Tune aggressively. A reviewer with forty comments reads none. Suppress the categories your team has decided it doesn't care about, and do it early, before people learn to ignore the channel.
- Watch acceptance rate. If most suggestions are being dismissed, the configuration is wrong. That number is your honest signal of whether this is helping.
- Say out loud that it isn't a gate. Otherwise it drifts into being treated as one, and a green check starts substituting for judgement.
The Honest Assessment
It's a good tool that makes reviews faster and catches a real class of defect. It's not a reviewer, and the moment a team starts treating a clean automated pass as approval, quality goes down while every visible metric says it went up.
Use it for the boring layer. Keep humans on the question of whether the change is right.
