Evaluating New Tools Without the Hype
Updated: 6 hours ago
In early 2025, the research group METR ran a randomised trial with 16 experienced open-source developers working on 246 real issues in their own repositories. Before starting, the developers expected AI tools to speed them up by 24%. With the tools, they took 19% longer. Afterwards, they still believed the tools had made them 20% faster.
The speed result has a short shelf life. METR's own follow-up in February 2026 says developers are probably sped up more by newer tools, though it calls its new data only weak evidence. The perception gap is the lasting lesson. Experienced engineers, in their own codebases, using the tool every day, misjudged its effect by about 40 percentage points in the wrong direction. A tool can feel faster and be slower.
That's the problem with the usual tool evaluation. A sensible filter decides whether a tool is worth trying, and then a trial decides whether to keep it. But in most trials the only measurement is how the tool feels, and feelings during a trial are biased: novelty and visible activity early on, the effort you've invested later. This post keeps the filter and fixes the trial.
Why trials confirm the hype
Three things push a trial towards "yes" regardless of the tool.
Visible effort goes down, invisible effort goes up. A tool that generates a draft makes the typing disappear, and typing is the part you notice. Reviewing, correcting and waiting are spread out and easy to under-count. That's one plausible reading of the METR gap: the effort that felt saved was visible, and the effort that was added wasn't.
Novelty. A new tool is more interesting than the old workflow, so the first weeks feel productive whatever the numbers say.
Sunk cost. Once you've migrated data and configured integrations, dropping the tool means admitting the time was wasted. Hal Arkes and Catherine Blumer's 1985 experiments showed people continuing to invest in something because of what they had already put in, even when the past cost should have no bearing on the decision.
There's also a timing trap. Once a team depends on a tool, nobody wants to measure the old way any more. METR found exactly this in 2026: many developers refused to work without AI at all, and some withheld the tasks they least wanted to do unaided, which made a clean comparison impossible. If you don't measure the baseline before adoption, you probably never will.
The test: for the last tool your team adopted, do you have any number from before you started using it? If not, your decision to keep it rests entirely on how it felt.
Name the problem as a number
The old filter's first question ("what specific problem does this solve?") is right. Make the answer a number, and measure it before anyone sees a demo.
A running example (composite). A QA lead is evaluating an AI tool that drafts test cases from user stories. Her team writes test cases for about 30 stories a sprint. Before the trial, she spends one sprint recording three numbers for every story:
Baseline measure | Result over one sprint (30 stories) |
Time to write the test cases (median per story) | 55 minutes |
Time a reviewer spends on them (median per story) | 10 minutes |
Stories whose test cases were sent back for rework | 6 of 30 (1 in 5) |
Defects found later that a missing test case should have caught | 2 |
That table does two jobs. It defines what "better" means (less total time, and no more rework or escaped defects), and it gives the trial something to beat. Without it, the team would have compared the tool against their memory of the old way, which is what the developers in the METR study effectively did.
Measure end to end, not just the step the tool touches. A tool that halves writing time but doubles review time has moved the work, not removed it.
The test: write the problem as "we spend X minutes, Y times a week, with Z errors" and measure X, Y and Z for one cycle before the trial starts. If you can't fill in the numbers, you're not ready to evaluate a tool for it.
Count the switching cost before the trial
Switching cost is real, and the pricing page never shows it. List it explicitly, in hours, so it can be weighed against the saving.
Cost | In the example |
Learning: until the team is as fast as before | About 6 hours per tester across 4 testers |
Migration: moving existing material in | None for the trial; 2 days later to import the template library |
Integration: connecting it to your tools | Half a day to link it to the issue tracker |
Upkeep: prompts, templates, admin, licence reviews | Estimated 2 hours a sprint |
Exit: getting your data out and going back | Export tested on day one: plain CSV, about an hour |
Then do the arithmetic the excitement skips. If the tool saves 20 minutes per story at 30 stories a sprint, that's 10 hours a sprint. Minus 2 hours of upkeep, it's 8. The one-off costs (24 hours of learning, about 20 hours of migration and integration) pay back in roughly six sprints. That's worth doing, but it's a three-month decision, not a one-week one, and it only holds if the 20 minutes is real.
The test: add up the one-off costs and divide by the measured saving per week, minus upkeep. If the payback is longer than you would expect to keep the tool, or longer than the vendor's pricing is likely to stay the same, stop here.
Design the trial: one workflow, a comparison, a stop rule
A trial is an experiment, and the most useful thing you can do is write down the rules before you start, while you're still neutral.
Pick one workflow, not the whole operation. One team, one type of work.
Compare against something. The cleanest option is to alternate: half the stories with the tool, half the old way, assigned before anyone looks at them. Comparing to the baseline sprint also works, if the work is similar.
Measure the same numbers as the baseline, end to end.
Run it past the learning dip. The first week with any tool is slower. Four weeks gives room for that; judge on the last two.
Write the decision rule now. For example: "Adopt if total time per story falls by at least 15 minutes and rework and escaped defects don't rise. Stop if total time hasn't fallen by week four."
The decision rule is the part that protects you from the trial. At the end of week four, sunk cost and novelty will both be arguing for the tool. A rule written in week zero isn't affected by either.
The test: before the trial starts, can someone else on the team read your plan and know exactly what result would make you drop the tool? If not, you haven't written a stop rule, you've written a hope.
Read the results honestly
In the example, the last two weeks of the trial looked like this:
Measure (median per story) | Baseline | With the tool | Change |
Writing test cases | 55 min | 12 min | −43 min |
Reviewing them | 10 min | 31 min | +21 min |
Total | 65 min | 43 min | −22 min |
Stories sent back for rework | 1 in 5 | 1 in 5 | No change |
Gaps found in review | Mixed | Mostly missing negative and boundary cases | New pattern |
The testers described the tool as "about five times faster", which is what the writing step alone suggests. The end-to-end saving was a third. That still clears the decision rule, so the team adopted it, but for a narrower job than the demo implied: drafting happy-path cases, with a checklist for negative and boundary cases added to every review.
The new pattern in the last row is the most valuable finding of the trial. It's a quality fact about the tool that no landing page would mention and no six-month user would have explained in a sentence. It came from measuring review, not just generation. Pilot theatre covers the same gap at the organisation level, where demos and pilots look like adoption without changing the work.
The test: compare the felt improvement (ask the people using it) with the measured one. If they differ by more than a factor of two, believe the measurement, and find the step where the hidden time went.
Check the exit before you import anything
The old filter asked what you lose if the tool disappears. Turn the question into a step on day one: export everything you've put in, and open the export somewhere else. If the export is a standard format (CSV, Markdown, JSON with a documented schema) and complete, you can leave. If it's partial, proprietary or "available on request", your exit cost is unknown, and it should count against the tool in the switching-cost table.
Check the pricing exposure at the same time. If the tool is priced per seat or per use, work out what it costs at the scale you would reach if the trial succeeds, not at trial scale. For open-source tools and libraries, evaluating open source before you depend on it covers the questions that matter instead: who maintains it, who can publish it and how its licence could change.
The test: on day one of the trial, run a full export and open it without the tool. Note how long it took and what was missing.
Flags that save time, turned into checks
Red and green flags are useful only if you can check them. Each of these takes a few minutes.
Instead of "watch out for…" | Do this |
Hype on the landing page | Write down, in one sentence, what the tool does. If you can't after reading the site, ask for a demo on your own data or skip it |
Enthusiasm with no specifics | Ask two people who have used it for six months what they use it for, and what they stopped using it for |
Changes how the team communicates | Treat it as a team change, not a tool trial: these carry the highest switching cost |
Only documents the happy path | Look for a page of known limitations. If there isn't one, your trial will be writing it |
Adds to the pile | Name the tool it replaces. If none, the saving must also cover one more login, bill and place to check |
To compare several candidates on the same criteria, the vendor evaluation scorecard template gives you the structure.
The test: pick the tool you're most excited about right now and ask a long-term user what they stopped using it for. Their answer usually tells you more than the demo.
Keep a record, and revisit it
Teams forget why they adopted things, and tools change after adoption. Keep a one-page record of every tool decision, in rings borrowed from the Thoughtworks Technology Radar: Adopt (use it by default), Trial (being tested on one workflow), Assess (worth a look, not yet tried) and Caution (Thoughtworks' name for what used to be called Hold: tried or watched, and not recommended for now). For each entry, record the date, the baseline and trial numbers, the decision rule and the result.
Revisit Adopt entries once a year, or when pricing changes. A tool that cleared the bar in a trial can stop clearing it as the work changes, and the record makes that visible without anyone having to argue from memory.
The test: for the three tools your team uses most, can you find when and why each was adopted? If not, start the record with those three.
What to do this week
Pick the tool you're most tempted to try. Before signing up, write the problem as a number, then measure it for one cycle. Draft the switching-cost table, and write the decision rule: what result in week four keeps it, and what drops it. Only then start the trial, on one workflow, with an export on day one.
Final takeaway
A tool trial without a baseline measures how the tool feels, and feelings during a trial lean towards yes: the effort that disappears is visible, the effort that's added isn't, and by week four the investment argues for itself. Measure the problem before you start, count the switching cost in hours, write the stop rule in advance and judge on the end-to-end number. Most tools still won't make the cut, and the ones that do will be adopted for the job they actually do well.
Next in the Tools That Work learning path: We'll cover integrating new tools into existing team workflows without the disruption that kills most tool adoptions.
Sources
Becker, J., Rush, N., Barnes, E. and Rein, D., "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (METR, July 2025): 16 developers, 246 issues; forecast 24% speed-up, measured 19% slow-down, post-hoc belief of 20% speed-up. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
METR, "We are Changing our Developer Productivity Experiment Design" (24 February 2026): 57 developers, 800+ tasks; developers likely more sped up in early 2026, but selection effects (including developers refusing to work without AI) make the new data weak evidence. https://metr.org/blog/2026-02-24-uplift-update/
Arkes, H. R. and Blumer, C., "The psychology of sunk cost", Organizational Behavior and Human Decision Processes 35(1), 124–140 (1985). https://doi.org/10.1016/0749-5978(85)90049-4
Thoughtworks, Technology Radar FAQ: definitions of the Adopt, Trial, Assess and Caution rings. https://www.thoughtworks.com/radar/faq
Related reading
Keep learning. This article is part of the Automation & Process Improvement path in the ShiftQuality Learning Center. Remove the repetitive work — and keep the automation from becoming new debt.
