AI Quality Engineering
Five series on building AI systems you can actually vouch for. The order below is the order to read them in: call an API, learn to prompt, build agents, then learn to measure whether any of it works.
Building with LLMs
Nine of ten parts published. The ground floor — what an LLM call is, and how to put one in a real application.
1. Call Your First LLM API
2. Add an LLM to a Real App
3. Stream Responses to a UI
4. Add Embeddings and Search
5. Build a Basic RAG System
6. Add Conversation Memory
7. Function Calling and Tool Use
8. Cache LLM Responses
9. Cost-Monitor an LLM App
Practical Prompt Engineering
Nine of ten parts published. Prompting as an engineering discipline — versioned, tested, and budgeted.
1. Set Up Your Prompting Workspace
2. Write Your First System Prompt
3. Few-Shot Examples That Work
4. Structured Outputs (JSON and Beyond)
5. Chain Prompts to Break Down Hard Tasks
6. Prompting for Code Generation
7. Multi-Turn Conversations
8. Guardrails: Prompt Injection and Refusals
9. Token Budgeting
Building AI Agents
Complete. From a single-tool agent to multi-agent workflows, with the failure handling and guardrails that separate a demo from a system.
1. What Is an Agent (With Code)
2. Build a Single-Tool Agent
3. Add Multiple Tools to an Agent
4. Agent Memory and State
5. Handle Agent Failures
6. Build a Planning Agent
7. Multi-Agent Workflows
8. Evaluate Agent Reliability
9. Production Observability for Agents
10. Agent Guardrails and Safety
AI Evaluation & Monitoring
Complete. How to know whether your AI system is getting better or worse — the discipline most teams skip.
1. Build Your First Eval Set
2. Automated Grading with Rules
3. LLM-as-Judge for Subjective Criteria
4. Track Prompt Regression Over Time
5. A/B Test Prompts in Production
6. Monitor for Prompt Drift
7. Build an Eval CI Pipeline
8. Capture and Replay Failures
9. Cost-Aware Evaluation
10. Continuous Evaluation in Production
LLM Application Engineering
Seven of ten parts published — parts 2, 8, and 9 are still in progress. The architecture layer: shapes, patterns, and the road from prototype to product.
1. LLM App Shapes
3. Structured Outputs
4. RAG in Production
5. Tool Use and Agents
6. LLM Evals
7. LLM Cost and Latency
10. From Prototype to Product