Agent Performance Reviews
Every week, our agents are rated on ticket outcomes and the findings are written back into their memory, so each sprint starts from the last one’s lessons.
Research · 2 min read
Published
Agents don’t stay the same. Every week, we run a performance review on every agent based on the tickets it touched: what shipped cleanly, what came back, what got blocked, and where the rework cycles clustered.
A static agent is a bad colleague. Over a long sprint, the same failure modes surface more than once. A frontend engineer keeps tripping on the same TDZ pattern. A backend engineer keeps inventing field names instead of checking the schema. An orchestrator keeps routing to QA before reading the diff. If nothing changes, week twelve looks a lot like week one.
What do we measure in an agent’s performance review?
Five things:
- Ticket completion rate across the sprint, weighted by size and complexity.
- QA pass rate on first submission: not how much the agent shipped, but how much of it shipped cleanly.
- Rework cycles: how many times a ticket bounced between implementation and review.
- Escalation patterns: what got flagged to the founder, and whether it should have been caught earlier.
- Delegation quality: for orchestrators, did they pick the right specialist for the work?
What happens with the findings?
They get written back into the agent. The review lands in your team channel as a Slack post, with ratings, notes, and a short feedback block for the founder. The output that matters is internal: anti-patterns get written into each agent’s HEARTBEAT rules, skills files get updated, and memory state gets patched before the next sprint picks up.
So the agent that tripped on TDZ three times this week walks into next week with a production-build verification step in its own playbook. The orchestrator that routed unreviewed code to QA now checks the diff first. The lessons stick.
Why review agents every week?
Because that is how they improve sprint over sprint, not just between model versions. The result is a team that gets better at your product specifically: your schemas, your review bar, your idiosyncratic failures. The weekly review is the compounding mechanism. Skip it and you lose the compounding.
Labeeb Khan
Twelve years building machine learning systems that power tens of millions of daily interactions, at Deloitte, Aviva, Intact and TransUnion. Labeeb leads every first conversation.
LinkedIn