How to Choose an AI Coding Agent
Every few months a new agent tops somebody's benchmark, and the honest answer to 'which one should we use' has not changed: it depends on things you can evaluate yourself in an afternoon. This is a framework for doing that, not a ranking that expires next quarter.
Last reviewed: August 2026
Why This Is Not a Ranking
The agents converge. Whatever separates them this quarter - a planning mode, a longer context window, a way to reach your systems - tends to appear everywhere within a few releases. A leaderboard written today misleads by next quarter, and worse, it invites you to reopen the decision every time the ordering shifts.
Benchmarks have the same problem in a different shape. A benchmark measures benchmark performance. It does not measure how an agent behaves inside a repository carrying a decade of your conventions, against your test suite, under your review habits, or within your compliance posture. Those are the things that decide whether an agent is useful to you, and none of them appear in a score.
So this page hands you axes and a procedure instead of a verdict. Score the candidates yourself, on your own work, and the answer stays valid as long as your constraints do. For a map of what actually exists in the category - agents, protocols, and platforms laid across the five phases - see AI Agents & Tools. This page is the deeper version of that page's short "How to Choose" list.
The Axes That Actually Matter
Seven criteria that keep deciding the outcome long after the feature lists have converged. Weight them for your team - they are not equally important everywhere.
1. Harness Shape
IDE-embedded, terminal or CLI, or asynchronous and cloud-hosted. This decides how the agent fits the loop you already have. An IDE agent sits where you already read code, a CLI agent sits where you already run commands, and an asynchronous agent takes a handoff and returns later.
Ask: does this match how my team actually works, or does it demand a new habit?
2. Context Reach
Can it see the whole repository, run your commands, and reach your other systems through MCP - or does it work from whatever you paste in? This is the single biggest driver of output quality, which is why context engineering is the skill underneath the tool.
Ask: how much of my system can it see without me assembling it by hand?
3. Review Surface
How the agent presents its work back to you: whether you can read a diff hunk by hunk, reject part of a change, and see which commands it ran. This is where the Develop phase's human checkpoint lives, and a weak review surface makes disciplined use expensive.
Ask: can I reject half a change without throwing away the whole run?
4. Autonomy Ceiling and Guardrails
What it may do without asking, and whether you can constrain the files it edits, the commands it runs, and what requires approval before it proceeds. More autonomy is only an asset if you can bound it; an agent you cannot fence in is one you will supervise keystroke by keystroke anyway.
Ask: what is the worst thing this can do before a human sees it?
5. Portability of Configuration
Whether your specs, conventions, skills, and MCP configurations travel with you or are locked to one vendor's format. An open plugin standard makes this a real question now rather than a theoretical one. Lock-in is a choice, and it is easier to decline at the start than to unwind later.
Ask: if we switched next year, what would we have to rewrite?
6. Governance and Data Posture
Where your code goes, what is retained, what telemetry is emitted, and whether all of that satisfies your organization. For many teams this is the criterion that eliminates candidates before any of the others matter, so it is worth settling first rather than last.
Ask: has whoever owns this decision already ruled some options out?
7. Cost Model
Per-seat, per-token, or hybrid - and how that behaves under agentic use, which consumes far more tokens than autocomplete did. A price that looks fine for chat can behave very differently once agents are running multi-step tasks all day across a whole team.
Ask: what does this cost when everyone uses it the way we intend to?
How to Run a Fair Trial
Pick One Real Task
Take a single already-scoped item off your backlog - something you were going to do anyway. Never a greenfield demo or a toy problem: those flatter every agent equally, so they cannot separate the candidates. The task should be small enough to review in one sitting and messy enough to be honest.
Write the Spec First, Once
Write intent, constraints, and acceptance criteria before you open any agent, and give every candidate the identical spec. If one gets a sharper brief than the others, you are measuring your prompting, not the agent. See context and spec-driven development for what belongs in it, and the prompt library for starting points.
Run Each Candidate in Your Real Repository
Your conventions, your dependencies, your test suite, your CI. An agent that shines on a clean sample project can struggle against the accumulated context of a real one, and that gap is exactly what you are trying to observe.
Score Against the Same Rubric While It Is Fresh
Fill in the seven axes right after each run, before impressions blur into a general feeling. Write down the specific moment that produced each score - that note is what you will actually reread when you make the call.
Judge on Review Throughput
The measure that matters is how quickly you got to a change you would actually merge - not how fast tokens appeared on screen. Generation speed is the easiest thing to notice and the least useful thing to optimize, because the review is where your time really goes.
Budget an Afternoon per Candidate
That is the honest cost of a trial like this, and it is small next to living with the wrong answer. Two or three candidates is usually enough; a long shortlist mostly buys you fatigue.
A Scoring Rubric You Fill In Yourself
The scores here are yours, not ours - which is the whole point. Copy this table, replace the candidate columns with the agents on your shortlist, and fill it in after each trial run. Weight the rows however your team needs to; governance often outranks everything else, and sometimes cost does.
| Axis | What You Are Scoring | Candidate A | Candidate B | Candidate C |
|---|---|---|---|---|
| Harness shape | Fits the loop we already have | - | - | - |
| Context reach | Sees the repo and our systems without hand-feeding | - | - | - |
| Review surface | Diff quality, partial rejection, visible commands | - | - | - |
| Autonomy and guardrails | Bounded on files, commands, and approvals | - | - | - |
| Portability | Specs, skills, and MCP configs travel with us | - | - | - |
| Governance and data | Retention, telemetry, and policy fit | - | - | - |
| Cost model | Behavior under real agentic use, across the team | - | - | - |
Keep the filled-in copy in the repository next to your specs and conventions. When someone reopens the question in six months - and someone will - the rubric is what turns it into a ten-minute conversation instead of a fresh argument.
Common Mistakes
- Choosing on benchmark scores. A benchmark ranks agents on a benchmark. Your repository is not one, and the ordering will have moved by the time you finish rolling out.
- Choosing on a demo video. The repository in the demo was built for the demo. Every agent looks decisive against a codebase with no history, no conventions, and no surprises.
- Optimizing for generation speed. Fast output that takes an hour to review is slower than careful output you can merge. Review throughput is the number that maps to delivery.
- Letting every developer choose separately. Different harnesses are survivable. Specs and conventions that never accumulate anywhere shared are not - the team relearns the same context in a dozen private chat histories.
- Ignoring exit cost. Ask how much work it takes to leave before you commit, not after. The answer is a feature of the tool, and it is knowable up front.
- Treating the choice as permanent. The agent is the most replaceable part of the stack. Deliberate over the context you build around it; hold the tool itself loosely.
Where This Fits in AIDLC
The choice of agent mostly touches two phases: Develop, where the diffs get produced and reviewed, and Launch, where tests and deployment automation get generated. The other three phases barely notice which agent you picked.
That is the useful part: the five phases do not change with the tool. Methodology over brand - learn the framework and the agent becomes an implementation detail you are free to swap.
Frequently Asked Questions
There is no stable answer, which is why this page is a framework rather than a ranking. The agents converge on features within months of each other, so a leaderboard written today misleads by next quarter. What lasts is the method: score the candidates against your own review workflow, context needs, and governance constraints, using a real task from your backlog.
About an afternoon per candidate, on one real scoped task you were going to do anyway. Greenfield demos and toy problems flatter every agent equally, so they tell you nothing. The signal comes from watching an agent work inside your actual repository, with your conventions, your test suite, and your review habits.
The agent matters less than the shared context. Teams can tolerate different harnesses, but they cannot tolerate specs, conventions, and guardrails living only in individual chat histories. Keep those in the repository in an open format, and the choice of agent becomes reversible.
Decide It Yourself, Then Move On
See what is actually in the category, then keep the method close - it outlasts whichever agent you land on.