Pick an AI Coding Tool — AI-Assisted Coding, Part 1

Shawn West
Jul 8
8 min read

Updated: Aug 6

AI-Assisted Coding · Part 1

The wrong way to pick an AI coding tool is the way most teams do it: someone watches a slick demo where the agent builds a to-do app from one sentence, a license gets bought for the whole team, and three weeks later half of them have quietly turned it off because it kept confidently suggesting an API that doesn't exist in your codebase. The demo was real. It just wasn't a measurement of anything, and a demo on someone else's toy repo tells you almost nothing about how a tool behaves inside your 40,000-line service with its weird auth middleware and a house style the model has never seen.

So this first part doesn't crown a winner. It gives you a way to run a real two-week trial with a scorecard, on your own code, so the decision is driven by numbers you collected instead of a video someone edited. That's the whole game: the best AI coding tool is the one that measurably helps you on your codebase — and the only way to know that is to measure it. By the end you'll have a filled-in comparison you can defend to a skeptical lead or a finance approver.

What you'll build

A trial scorecard — a small spreadsheet or markdown table — plus the protocol to fill it in over two weeks. You'll shortlist two tools, run both against the same real tasks from your actual backlog, and score each on acceptance rate, time-to-useful, and the failure modes that actually cost you time. The output is a defensible recommendation, not a vibe.

We'll run the whole path against one running example: you're a working engineer on a TypeScript/Node service — call it the orders service, ~40k lines, Postgres, a REST API, an established house style — and you're deciding which assistant to adopt across your real workflow (writing, reviewing, refactoring, testing, debugging). Every later part uses the tool you pick here.

The 2026 landscape (10 min)

Know the shape of the field before you shortlist. These aren't interchangeable; they cluster by how they insert into your work:

Cursor — a VS Code fork with tight AI integration; strong tab-completion, side-panel chat, and a multi-file "agent" mode. The popular default in 2026.
GitHub Copilot — an extension for VS Code, JetBrains, and others; the strongest inline autocomplete and the lowest-friction org rollout (GitHub already has your code).
Claude Code — terminal-first and agentic; reads the repo, plans, and edits multiple files for a task you hand it — closer to "give a teammate a ticket" than "autocomplete my line."
Windsurf — Codeium's IDE, in the Cursor mold.
Continue / Aider / Cody / Tabnine — open-source or bring-your-own-model options; matter most when self-hosting or code-search integration is the constraint.

The space genuinely changes every quarter — models get swapped under these products constantly — so treat any specific capability claim as perishable and re-evaluate every six months. What doesn't change is the evaluation method below.

Step 1: Pick your two real tasks (15 min)

The trial lives or dies on task selection. Do not evaluate on a greenfield toy — that's the demo trap. Pull tasks from work you actually have to do this sprint, in two categories, because tools that shine at one often flop at the other:

A net-new, well-scoped task — e.g. "add a PATCH /orders/:id/status endpoint with validation, following our existing handler pattern." This tests code generation inside your conventions.
A change deep in existing code — e.g. "our refund calculation double-counts partial returns; find and fix it." This tests whether the tool can navigate a real codebase, which is where most of them break.

Write these down concretely. The same two tasks go to every tool you trial, so the comparison is apples-to-apples.

Step 2: The three modes, and which one actually matters (10 min)

Every tool sells some mix of three modes. Be clear-eyed about which you're buying:

Inline suggest — completes the line you're typing; you accept with Tab.
Chat — you ask, it explains or drafts.
Agent — you hand it a multi-step task and it edits across files.

Here's the opinion I'll defend, and it's contrarian to how these are marketed: for an experienced engineer on a real codebase in 2026, chat and agent are worth far more than inline autocomplete. Autocomplete saves seconds on code you already knew how to write; agent mode saves minutes-to-hours on the navigate-understand-edit-verify loop that eats your day. If a tool's headline is "40% of your code written by AI," that's an autocomplete metric, and it's measuring the cheap thing. Score the expensive thing.

Step 3: Build the scorecard (10 min)

Here's the table. One row per tool, filled in over the two weeks:

| Metric | How to measure | Tool A | Tool B | |---|---|---|---| | Useful-suggestion rate | of AI outputs you kept vs. total, tracked by tally | | | | Time-to-useful (net-new task) | wall-clock from start to passing code | | | | Time-to-useful (deep-change task) | same, for the refund bug | | | | Codebase-fit | 1–5: did it follow your patterns without being told? | | | | Hallucinated APIs | count of made-up functions/imports it suggested | | | | Review burden | 1–5: how hard was its output to verify? | | | | Flow cost | 1–5: did switching to it break your concentration? | | |

The three that decide it are codebase-fit, hallucinated APIs, and review burden — because a tool that writes plausible code you can't trust doesn't save time, it moves the time from writing to reviewing, and often adds to it. A high autocomplete rate with a high hallucination count is a net loss, and this scorecard is designed to expose exactly that trade the demos hide.

Step 4: Shortlist to two (10 min)

Don't trial five tools; you'll do a shallow job of each. Shortlist to two using cheap filters before spending trial time:

Language and repo fit. JS/TS/Python: everything works. Go/Rust: most do. Elixir/OCaml/legacy mainframe: test skeptically — these tools learned from public code, and thin public corpora mean weaker output.
Repo scale. A 100k-file monorepo needs real code-retrieval (Cursor, Cody, Claude Code handle scale better than plain autocomplete tools). A small repo, anything works.
Deployment constraints. If code can't leave your network, that alone narrows you to self-hostable options (Continue, Cody with your own model) — decide this first, it's a hard gate.

For the orders service (TS, medium size, cloud-OK), a sensible shortlist is Cursor vs. Claude Code — one editor-integrated, one agentic — precisely because they represent different bets on how you'll work.

Step 5: Run the two-week trial (the main event)

This is where the decision actually gets made, so give it the two weeks. The protocol:

Week 1: Tool A. Do your real work through it, including the two designated tasks. After each meaningful interaction, add a tally mark for kept/discarded and jot any hallucinated API. At the end of a task, record the wall-clock time.
Week 2: Tool B. Same work-shape, same two tasks (or close equivalents from your backlog), same tallying.
Keep a "frustration log" — a one-line note every time the tool broke your flow, suggested something dangerous, or made you say "no, not like that." These qualitative notes catch what the numbers miss, like a tool that's fast but constantly fights your house style.

A concrete example of what you're hunting for: on the refund-bug task, one tool might read three files, find the double-count in calculateRefund, and propose a one-line fix with a test — ten minutes, done. The other might cheerfully rewrite the whole function using a refundService.recalculate() method that does not exist, and you lose fifteen minutes discovering that before you've fixed anything. That single divergence, on a task you actually had, tells you more than every benchmark on the internet.

Step 6: Do the cost math honestly (10 min)

Now put a number on it. Rough 2026 pricing: personal tiers run $10–30/month, team seats $20–60/user/month, enterprise is negotiated. The break-even is not mysterious — a tool at $30/month that saves you 30 minutes a day pays for itself several times over at any real engineering salary. So the cost question is almost never "is it too expensive"; it's "did it actually save the 30 minutes," which is exactly what your time-to-useful column now answers.

One honest trap: the free tiers usually run smaller, cheaper models, and the gap between a frontier model and a budget one on real code is large. Trial the paid tier, or you're measuring the wrong product.

Step 7: Check privacy and IP before you commit (10 min)

For any codebase that isn't fully public, verify three things in writing — don't assume:

Does the tool train on your code? (Copilot for Business and Cursor's business tiers say they don't; confirm for your plan.)
Does code leave your network, and is there an on-prem or bring-your-own-model option if it can't?
What's retained, for how long, and where?

For a regulated or IP-sensitive codebase this is a gate, not a preference — and it's the kind of thing that belongs in your team's written policy, which is where Part 10 of this path ends up.

Step 8: Make the call (5 min)

Total the scorecard, weight codebase-fit and review-burden highest, factor the frustration log, and pick. Write one paragraph on why — "Claude Code won on the deep-change task and hallucinated less against our patterns; Cursor's autocomplete was nicer but that's not where our time goes" — because that paragraph is what you'll show the skeptic, and it's what you'll re-check at the six-month re-evaluation.

What you just did

You replaced "we watched a demo and bought licenses" with a measured trial: real tasks from your backlog, two tools scored on the metrics that actually move engineering time, honest cost and privacy gates, and a defensible written call. The tool you picked is the one you'll use for the rest of this path.

Troubleshooting

Both tools scored about the same. Then pick on the tiebreakers that compound: flow cost (which one you enjoy using, because you'll use it more) and org fit (which one IT will actually approve). A tool you'll reach for beats a marginally-better one you won't.

You can't get clean time-to-useful numbers. Your tasks are too big or too fuzzy. Shrink them to something completable in under an hour so the wall-clock number means something.

The tool is great on new code, useless on the old codebase. That's the single most common and most important finding — and it's why Step 1 insists on a deep-change task. Weight it heavily; most of your career is spent in existing code.

Leadership wants a decision this week. Run a compressed 3-day version on just the two designated tasks. It's weaker than two weeks, but a scorecard on real tasks still beats a demo.

Common mistakes

Evaluating on a toy project. The demo trap. Trial on your real repo or you've measured nothing.
Trialling the free tier. You're testing a weaker model than the one you'd buy. Use the paid tier.
Counting autocomplete rate as the win. It measures the cheap thing. Weight codebase-fit, hallucinations, and review burden.
Committing forever. The field moves quarterly; put a six-month re-evaluation on the calendar now.
Skipping the privacy check. For sensitive code this is a compliance gate, not a footnote — verify it in writing before rollout.

Continue the AI-Assisted Coding path

Next — Part 2: Prompt for Code — get useful output from the tool you just picked.