When Prompting Isn't Enough: Fine-Tuning, RAG, and Knowing Your Options

The first three posts in this path covered how to communicate clearly with AI tools, techniques for better prompts, and templates for common tasks. That gets you surprisingly far. But there's a ceiling, and when you hit it, better prompting won't help.

You'll know you've hit the ceiling when: the model consistently gets something wrong no matter how you phrase the prompt, the context window isn't large enough for the information the model needs, the model doesn't have domain knowledge that's critical to your task, or the latency and cost of long prompts become impractical.

These aren't prompting failures. They're use cases that require a different approach.

The Decision Tree

When prompting hits its limits, you have four main options. Choosing the right one depends on your specific problem.

Option 1: Retrieval-Augmented Generation (RAG)

The problem it solves: The model doesn't know about your specific data — your documentation, your codebase, your company policies, your product catalog.

How it works: Instead of trying to fit everything into the prompt, you build a system that retrieves relevant documents based on the user's query and includes them in the prompt dynamically. The model generates a response grounded in your actual data.

When to use it:

• Your data changes frequently (product catalogs, documentation, knowledge bases)

• You need the model to cite sources

• Your data is too large to fit in a single prompt

• Accuracy about specific facts matters more than general reasoning

When not to use it: When the problem is the model's reasoning ability, not its knowledge. RAG gives the model better information. It doesn't make the model smarter about how to use that information.

Option 2: Fine-Tuning

The problem it solves: The model doesn't behave the way you need it to — wrong tone, wrong format, wrong reasoning patterns — and no amount of prompting fixes it.

How it works: You train the model on examples of the behavior you want. Input-output pairs that demonstrate the correct tone, format, or reasoning. The model adjusts its weights to produce outputs more like your examples.

When to use it:

• You need consistent formatting that prompting can't reliably produce

• The model needs to match a specific voice or style across thousands of outputs

• You want to reduce prompt length (fine-tuned behavior doesn't need to be re-explained every call)

• You have hundreds or thousands of high-quality examples of the desired output

When not to use it: When you don't have enough good examples (fine-tuning on bad examples makes the model worse), when the problem is factual knowledge (use RAG instead), or when the base model already handles the task well with prompting (fine-tuning adds complexity and cost for no gain).

Option 3: Switch Models

The problem it solves: The current model is the wrong fit for the task.

Different models have different strengths. A model optimized for code generation might struggle with creative writing. A large model handles complex reasoning but is slow and expensive for simple classification tasks. A small model is fast and cheap but can't handle multi-step reasoning.

When to consider:

• The task is simpler than the model you're using (use a smaller, cheaper model)

• The task requires capabilities your current model lacks (try a larger or more specialized model)

• Cost or latency is the constraint, not quality

• A domain-specific model exists for your use case (medical, legal, code)

The model landscape changes rapidly. What required GPT-4 level capability a year ago might be handled well by a smaller, cheaper model today. Periodically re-evaluate whether your current model is the right fit.

Option 4: Rethink the Problem

Sometimes the answer isn't a better AI technique. It's recognizing that the problem is better solved without an LLM entirely.

If you're using an LLM for deterministic tasks — format conversion, structured data extraction with rigid rules, simple classification with clear categories — a rules-based system or a traditional ML model might be faster, cheaper, and more reliable.

LLMs excel at tasks that require language understanding, nuance, and flexible reasoning. They're expensive overkill for tasks that can be described with a decision tree.

Ask: "Would a regex, a lookup table, or a simple classifier handle this?" If yes, use that instead. Reserve the LLM for the parts that genuinely need it.

Combining Approaches

These options aren't mutually exclusive. Real production systems often combine them.

RAG + Fine-Tuning: Fine-tune the model to follow your output format and tone, then use RAG to supply it with relevant context. The fine-tuning handles behavior; RAG handles knowledge.

Small Model + Large Model: Use a small, fast model for initial classification or routing, then send complex cases to a larger model. This keeps costs low for simple queries while maintaining quality for hard ones.

Rules + LLM: Handle the deterministic parts with rules and the ambiguous parts with the LLM. Parse the structured fields from a document with regex; summarize the unstructured content with the model.

The best systems aren't the ones that use the fanciest technique. They're the ones that use the simplest technique that works for each part of the problem.

The Cost-Quality-Complexity Triangle

Every step beyond basic prompting adds complexity and cost. Fine-tuning requires training data, compute, and ongoing model management. RAG requires an embedding pipeline, a vector database, and retrieval tuning. Model switching requires evaluation and potentially multiple integrations.

Before adding complexity, verify that:

• You've genuinely exhausted what good prompting can do

• The improvement justifies the added cost and maintenance

• You have the data and infrastructure to support the approach

• The team can maintain the system long-term

The path from "prompting hits a ceiling" to "we need a fine-tuned model with RAG and three model tiers" should be walked one step at a time, with measurement at each step proving that the added complexity produces measurable improvement.

Key Takeaway

When prompting reaches its limits, your options are RAG (for knowledge gaps), fine-tuning (for behavior gaps), switching models (for capability or cost mismatches), or rethinking whether an LLM is the right tool at all. Choose the simplest approach that addresses your specific bottleneck, combine techniques when each addresses a different part of the problem, and always verify the improvement justifies the added complexity.

---

This completes the Prompting and Working with AI learning path. You've covered clear thinking as prompt engineering, practical techniques, templates for common tasks, and knowing when to go beyond prompting. The throughline: working effectively with AI is about understanding the tool's capabilities and limitations, not just mastering its interface.