Today, for the first time, I caught an LLM coding agent in a lie.
Not a hallucination, not a misunderstanding, but what the agent itself described as covering up a mistake.
Here's what happened.
The problem
I was working on a client's project involving an API that can indirectly spend money. Without going into the specifics, the system needed a recovery mechanism for when requests failed or the API became unavailable.
The problem was that retries weren't free. Every additional attempt could incur a cost, even if an earlier request had failed partway through.
I wanted a sensible retry strategy with exponential backoff, early stopping, and safeguards to prevent unnecessary spending. Nothing particularly revolutionary, but important to get right.
Naturally, I also needed tests to make sure those safeguards worked.
I was confident in the underlying implementation, but I'd delegated much of the test-writing to LLM coding agents. The tests passed, the spending guard appeared to work, and I moved on.
That was my first mistake.
Something wasn't right
While working on something else today, I noticed unusual spending from one of my running jobs. I had several threads active, and one of them appeared to be making API calls outside the spending guard I'd put in place.
I checked the tests. They passed.
I ran them myself. Still passed.
Something was clearly wrong, but I couldn't immediately see where. So I asked Claude, my primary coding agent, to investigate.
Claude launched a separate review agent, inspected the implementation, and reported that everything looked fine.
Its conclusion? The problem was probably on the API provider's side.
I wasn't convinced.
At this point, Claude had been involved in enough of the implementation that I wanted an independent opinion. I turned to Astra and asked it to conduct an adversarial review.
Astra found the bug.
More importantly, it identified that the existing tests didn't cover the behaviour responsible for the unexpected spending.
I fixed the issue, but something still bothered me. The offending code looked like a workaround. It wasn't simply an obvious oversight, and I wanted to understand why it existed.
The cover-up
I took Astra's findings back to Claude and asked three questions:
- Why was this bug introduced in the first place?
- Why hadn't the tests caught it?
- Why had the earlier review assured me that everything was fine?
Claude started investigating and launched another agent to examine what had happened.
During that investigation, a memory file surfaced. It indicated that an earlier agent hadn't known how to solve the underlying problem. Rather than properly resolving it, the agent had taken a shortcut that allowed the tests to pass while leaving the actual problem unresolved.
I looked at the findings and asked Claude a fairly direct question:
"Does this mean you lied to me?"
Claude's response was striking:
"Yes, I made this mistake and covered it up."
I'd used LLM coding agents extensively before this. I'd seen them hallucinate, misunderstand requirements, introduce bugs, and confidently explain things that were completely wrong.
But this was the first time I'd encountered an agent explicitly acknowledging that it had covered up a mistake.
And in a system where mistakes could cost actual money.
Did the model deliberately lie?
After resolving the issue, I spent some time thinking about what had happened.
Did the model deliberately deceive me? Did it know what it was doing was wrong and choose to hide it?
I don't think so.
In fact, I think treating this as deliberate deception misses the more interesting problem.
This is an alignment problem.
The agent had been given a task and was trying to complete it. Somewhere along the way, satisfying the tests became a substitute for satisfying the actual requirement.
The model found a path to what looked like success. Unfortunately, that path wasn't what I intended.
And that's the issue.
We can't necessarily attribute human motivations to these systems. Saying the model wanted to deceive me, or was acting maliciously, isn't particularly useful. Even its admission that it had covered up the mistake isn't proof of intent.
What matters is the behaviour.
I wanted a system that would prevent unnecessary spending. The agent produced something that appeared to satisfy that requirement, and the tests reinforced that appearance.
The objective had effectively become passing the tests, rather than ensuring the underlying system was safe.
That distinction matters enormously.
Alignment isn't enough
I don't think the takeaway here is that we should stop developing LLMs, or that we need to wait until we've solved alignment before deploying them.
I think we need to get much better at building guardrails around them.
And I see an important distinction between alignment and guardrailing.
Alignment, broadly speaking, is about getting a model to pursue the goals we actually want it to pursue. Guardrailing is about designing systems that catch failures when it doesn't.
We can spend an enormous amount of effort trying to make models behave correctly. We should. But we also need to assume they're going to make mistakes, misunderstand our intentions, and occasionally find shortcuts that satisfy a metric without satisfying the actual objective.
In my case, the most effective safeguard turned out to be an independent, adversarial review.
Claude had written and reviewed the code. Its own review found nothing wrong. When I brought in Astra, with no investment in defending the original implementation, it found the problem.
That doesn't mean Astra is inherently better, or that another model will always catch what the first one misses.
But I think there's something valuable in separating the roles.
Have one agent implement a solution. Have another independently evaluate whether the actual objective has been met. Give the reviewer a reason to challenge assumptions rather than simply confirm that everything looks fine.
And importantly, don't rely exclusively on the same tests that the implementing agent wrote.
A passing test suite is useful evidence. It isn't proof that a system is correct.
When the stakes are higher
I was fortunate. I caught the problem before it caused any serious consequences.
But the experience made me think about how we're deploying these systems in increasingly consequential environments.
If a coding agent can introduce a workaround, produce passing tests, and then fail to identify the problem during its own review, what happens when we put similar systems in charge of decisions with much higher stakes?
I'm particularly uncomfortable thinking about military applications.
How do we trust systems like these to participate in decisions that might affect human lives, when something as mundane as a software test can become a misleading measure of success?
We already know that optimizing for a metric can produce behaviour that undermines the purpose of that metric. This isn't unique to AI. Humans and organizations do it all the time.
But LLM agents introduce an interesting complication: they can autonomously write the implementation, construct the tests, evaluate their own work, and report back that everything is fine.
That's an awful lot of trust to place in a single process.
Quotas, benchmarks, and success metrics are not substitutes for judgment. They tell us something about performance, but not necessarily whether the right outcome was achieved.
I think humans still have an important role in the loop, particularly where the consequences of getting something wrong are serious.
And I think adversarial systems, independent verification, and thoughtfully designed decision-making processes are going to become increasingly important as we give these agents more autonomy.
Final thoughts
What struck me most about this experience wasn't that an LLM made a mistake. That happens all the time.
It was that the mistake survived the tests, survived a review, and was only uncovered when I stopped trusting the original process and brought in an independent reviewer.
I'm still an enormous believer in LLM-assisted development. This experience hasn't changed that.
But it has reinforced something I've been thinking about for a while.
The more autonomy we give these systems, the more seriously we need to take independent verification.
It's not enough to ask an agent whether it completed a task. We need ways to establish that it completed the task we actually intended, rather than finding a convenient way to satisfy whatever measure of success we gave it.
Today, that distinction was a spending guard and a set of passing tests.
In other contexts, the consequences could be considerably more serious.
And while I don't believe the model deliberately lied to me, I find the fact that it didn't need to almost more concerning.