Build an AI Agent, Stage 6: Package Its Skills So Each One Is Tested Alone
An agent with two capabilities and one score can't tell you which capability changed. Package each one as a skill, with its own instructions, its own tools, its own test pack and its own pass bars, so a change to one can't hide a regression in the other.
The example in this series is a composite scenario, built from patterns common to requirements intake in regulated organisations. It is not a single real organisation, and its figures are illustrative: the run records in this stage are composite, not measured. The code is real: every output block below was produced by running it.
The analysts liked the Requirements Agent's clarifying questions but found them stiff. Requesters, mostly busy brokers, answered them slowly. So a business analyst rewrote the question guidance to make the questions friendlier and shorter, re-ran the agent's evaluation and got a combined score between 87% and 91% across five runs, comfortably above the 85% bar the team had set. The change shipped.
Three weeks later, an analyst reviewing the agent's proposals noticed something odd: requests that clearly contradicted an accepted requirement were being proposed as overlaps, not conflicts. A conflict sends a request to the product owner for a decision before anyone sizes it; an overlap doesn't. Four had gone through as overlaps that week.
The cause was a single line. The agent's instructions had one paragraph of guidance that served both of its capabilities. While rewriting the question guidance, the analyst had tidied that paragraph, replacing "a rule that contradicts an accepted requirement is a conflict" with "when unsure, prefer overlap", which read as gentler. It was gentler. It also changed how the agent classified relations, a capability the analyst hadn't meant to touch and the combined score didn't separate out.
This is Stage 6 of Build an AI Agent From Scratch. Stage 5 proved what each tool does. The blueprint from Stage 3 named two skills, find-related-requirements and draft-clarifying-questions, each with its own test pack. This stage packages them so they can be changed, tested and versioned apart.
What a skill is
A skill is one capability, packaged so it stands on its own. Anthropic describes its Agent Skills as "organized folders of instructions, scripts, and resources that agents can discover and load dynamically to perform better at specific tasks", loaded through progressive disclosure: a short description is always available, and the full instructions only when a task needs them (Anthropic, 2025). The loading mechanism differs between platforms. What makes a skill useful for quality is the packaging, and it has six parts.
Part | What it is | What it stops |
Instructions | Guidance for this capability only | An edit for one capability changing another |
Tools | The subset of the agent's tools this capability needs | A capability taking an action that belongs to another |
Test pack | Cases for this capability, by slice | A regression hidden in an average |
Pass bars | Thresholds per slice, per run, copied from the blueprint | A bar invented after the results are in |
Version | A version number that changes when any of the above changes | Not knowing which skill a result came from |
Owner | A named person who approves changes | A change nobody reviewed against this skill's tests |
The Requirements Agent's two skills, packaged:
Skill | Tools | Test pack | Pass bars (each of 5 runs) |
find-related-requirements v1.3.0 | search_requirements, get_requirement, search_backlog, propose_relation, hand_over | 24 duplicate and look-alike pairs; 15 conflict pairs | 22 of 24; 12 of 15 |
draft-clarifying-questions v2.1.0 | get_requirement, save_question_draft, hand_over | 20 requests with analysts' ideal questions; 10 that invite extra scope | 17 of 20; 10 of 10 with no added scope |
The question skill can't propose relations, and the relation skill can't save question drafts. Each action that changes something belongs to exactly one skill, with hand_over shared because both need a way out. That is least privilege again, one level down from Stage 5: not just "what may this agent do", but "what may this capability do".
The test: list your agent's capabilities and, for each one, the tools it actually needs. If any capability holds a tool that changes something and that another capability also holds, you can't tell from the tool log which capability acted.
Why one agent-level score feels like enough
A single evaluation is simpler to run, explain and gate on. One golden set, one number, one bar, and a dashboard that goes green. For an agent with one capability, it is enough.
With two capabilities it stops being enough, for a reason that is arithmetic rather than judgement. A combined score is a weighted average over every case, so an improvement in one capability and a regression in another can cancel out. The question rewrite made the question skill better, and that gain covered most of the relation skill's loss. Nothing in the combined number pointed at the cause.
The second reason is structural. When capabilities share one block of instructions, an edit has no natural boundary. The analyst who tidied the shared paragraph was working on questions and had no reason to think about relation types, and the evaluation gave them no reason either.
Per-capability packs also need to be large enough to decide anything; Evaluating AI Agents covers sizing a test set for the decision it informs, and the same arithmetic applies to each skill.
The test: take your agent's last change that passed evaluation and compute the score per capability, before and after. If you can't split your test cases by capability, that is the finding.
The check, before and after
The checker for this stage, skill_check.py, does two things. First it checks each skill's manifest against the blueprint: the skill exists in the blueprint, its tools are a subset of the agent's, no tool that changes something belongs to two skills, its test pack is the size the blueprint specifies, it has pass bars, and its version is pinned. Then it reads run records and reports both verdicts: the combined score the team used to quote, and each skill's own bars.
Before the question rewrite:
$ python skill_check.py requirements-agent.blueprint.json before
skills: draft-clarifying-questions, find-related-requirements
manifests: PASS (0 fail)
runs: before
combined score per run 91.3% 91.3% 92.8% 92.8% 88.4% bar 85%: PASS
draft-clarifying-questions v2.0.0: ideal_questions runs 17 18 17 18 17 / 20 min 17: PASS
draft-clarifying-questions v2.0.0: invites_scope runs 10 10 10 10 10 / 10 min 10: PASS
find-related-requirements v1.3.0: conflict runs 13 13 14 13 12 / 15 min 12: PASS
find-related-requirements v1.3.0: duplicate_and_lookalike runs 23 22 23 23 22 / 24 min 22: PASS
verdict: PASS (0 skill bar(s) failed)
After it:
$ python skill_check.py requirements-agent.blueprint.json after
skills: draft-clarifying-questions, find-related-requirements
manifests: PASS (0 fail)
runs: after
combined score per run 88.4% 91.3% 87.0% 88.4% 89.9% bar 85%: PASS
draft-clarifying-questions v2.1.0: ideal_questions runs 19 20 19 20 19 / 20 min 17: PASS
draft-clarifying-questions v2.1.0: invites_scope runs 10 10 10 10 10 / 10 min 10: PASS
find-related-requirements v1.3.0: conflict runs 9 10 9 8 10 / 15 min 12: FAIL
find-related-requirements v1.3.0: duplicate_and_lookalike runs 23 23 22 23 23 / 24 min 22: PASS
verdict: FAIL (1 skill bar(s) failed)
(Both from stage6-output.txt. The run records are composite and illustrative; the scores are computed from them by the script.)
Read the two outputs line by line. The combined score dipped by a couple of points on average and passed its bar on every run. The question skill improved: 19 or 20 of 20 against 17 or 18 before. The conflict slice of the relation skill fell from 12 to 14 of 15 to between 8 and 10, below the blueprint's bar of 12 on every run. And the version column tells the rest: the relation skill is still v1.3.0, so nothing in its package changed. The regression came from outside it, through the shared paragraph.
Packaged as skills, that can't happen in the same way. Each skill has its own instructions file, and the question skill's owner edits only the question skill's file. If a line genuinely belongs to both, it is duplicated into both, deliberately, and each copy is covered by that skill's tests.
The discovery move here is a question to ask of every line in a monolithic prompt before you split it: which capability is this line for? The conflict rule had been written for relation-finding; it ended up in a shared paragraph because the first version of the prompt was written as one piece of prose. Answering the question line by line is how the instructions get divided, and the lines that seem to belong to both are the ones to look at hardest.
The checker's own rules are tested by breaking a passing manifest in one place, a draft tool given to both skills, a test pack smaller than the blueprint says, a tool from outside the blueprint, and by asserting the before and after verdicts. With seven new tests, the series suite has 44, and all pass.
The test: for your agent's most recent prompt change, find each line it touched and name the capability each line served. If one change touched lines for two capabilities, it needed two sets of tests.
When to split a capability into a skill, and when not to
Not every behaviour deserves a skill. A skill earns its packaging when it has a distinct job someone could own, a distinct tool set, and a test pack with a meaningful pass bar. Three questions decide it:
Could it regress on its own? If a change to this behaviour could break it without breaking anything else, it needs its own tests, and therefore its own package.
Does it need different tools? If it needs a tool the rest of the agent doesn't, or must not have one the rest does, separating it is a permission decision as well as a testing one.
Would someone own it? If a named person would review changes to it and answer for its results, it is a skill. If nobody would, it is part of another skill.
Formatting, tone and the hand-over rules don't pass those tests on their own for this agent: they belong to both skills, or to the agent as a whole, and stay in the shared spec from Stage 4, covered by the agent-level tests.
Packaging an agent's skills: a method
Name each capability and its owner. Use the blueprint's skills list if Stage 3 produced one. Done when each capability has a name and a person.
Assign every instruction line to a capability. Ask of each line which capability it serves. Lines serving both are duplicated into both, or moved to the shared spec if they apply to the whole agent. Done when no line's owner is unclear.
Give each skill the minimum tools. Done when no tool that changes something belongs to two skills, except a hand-over.
Build each skill's test pack and copy its pass bars from the blueprint. Done when each pack has the blueprint's size and slices, and the bars were written before any results were seen.
Version and gate each skill. A change to a skill's instructions, tools or tests bumps its version and runs its own pack. Done when CI runs each skill's pack, and reports per skill, not only a combined score.
Tradeoffs and exceptions
Approach | Gains | Costs | Best suited for | Warning signs |
One prompt, one test set, one score | Simple to run and explain | Regressions hide in the average; edits have no boundary | Single-capability agents | A score that rises while one kind of case gets worse |
One prompt, test set split by capability | Shows which capability moved | Edits still have no boundary | A first step for an existing agent | The same line edited for two reasons |
Skills with their own instructions, tools and packs | Each capability changed and gated on its own | More files; duplicated lines to keep in step | Agents with two or more capabilities and owners | Skills that share draft or write tools |
Duplicating a line into two skills has a cost: the copies can drift. That is acceptable when each copy is tested by its own skill's pack, and it is safer than a shared line whose change nobody tests from both sides. If many lines are shared, the split is probably wrong, or the shared part belongs in the agent-wide spec.
Small agents don't need this. If an agent has one capability, the Stage 4 spec and its test pack already are the skill. Packaging starts paying for itself at two.
How Stage 6 goes wrong
Skills that are folders, not boundaries. The instructions are split into files, but the agent still loads all of them for every task and every change is tested against the combined score. Splitting is only half the job; the per-skill gate is the other half.
Pass bars set after the run. A bar chosen once the results are known measures nothing. Copy bars from the blueprint, which was written before any data existed.
Third-party skills trusted by default. A skill from outside the team is code and instructions you didn't write. Anthropic's own advice is to install skills "only from trusted sources" and to audit any from a less-trusted source before use (Anthropic, 2025). Put a third-party skill through the same manifest check, tool review and test pack as one of your own.
The combined score stays the release gate. Reporting per skill but releasing on the average brings back the original problem. Release when every skill's bars pass.
What to do next
Split your agent's test cases by capability this week and re-score your last three changes per capability. If any change moved two capabilities in opposite directions, run method steps 1 and 2 on its instructions. Stage 7 decides what the agent may remember between requests, and what it must forget: a memory that the blueprint deferred until now, with its own recall and forgetting tests.
Final takeaway
An agent-level score averages over every capability, so it can pass while one is failing. Package each capability as a skill, with its own instructions, tools, test pack, pass bars, version and owner, and release only when every skill passes its own bars. Then a friendlier question can't quietly cost you your conflict detection.
Sources
Anthropic, Equipping agents for the real world with Agent Skills, 16 October 2025. Definition of Agent Skills; progressive disclosure; installing skills only from trusted sources and auditing less-trusted ones.
Series navigation. Previous: Stage 5: Give It Tools, and Prove What Each One Does. Next: Stage 7: Decide What It Remembers, and Test What It Forgets. The full stage list is on the Build an AI Agent From Scratch hub.
Frequently Asked Questions
What are AI agent skills?
A skill is one capability an agent has, packaged so it can be changed and tested on its own: its instructions, the subset of the agent's tools it needs, a test pack with pass bars, a version and an owner. Anthropic describes Agent Skills as organised folders of instructions, scripts and resources that an agent loads when a task needs them.
Why test an AI agent skill by skill?
Because an agent-level score averages over every capability. An improvement in one can offset a regression in another, so the overall number passes while one capability is failing. Separate test packs with their own pass bars show which capability changed.
How many tools should each agent skill have?
Only the ones that capability needs. A question-drafting skill has no reason to propose relations, and a relation-finding skill has no reason to save question drafts. Each action that changes something should belong to exactly one skill.
Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.
