Add an LLM to a Real App — Building with LLMs, Part 2

Building with LLMs · Part 2

A standalone LLM call is easy. Integrating it into a production service requires discipline. This tutorial walks through it.

Step 1: Pick a Concrete Use Case (5 min)

Don't add "AI" generically. Pick:

• Summarize support tickets when they arrive
• Generate alt-text for uploaded images
• Suggest replies in a chat interface
• Classify incoming emails by urgency

Specific. Bounded. Measurable.

Step 2: Build the Service Layer (30 min)

Wrap the LLM call in your own service:

# services/llm_service.py
from anthropic import Anthropic
import os

class LLMService:
    def __init__(self):
        self.client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
        self.model = os.environ.get("LLM_MODEL", "claude-sonnet-4-6")
    
    def summarize_ticket(self, ticket_text: str) -> str:
        response = self.client.messages.create(
            model=self.model,
            max_tokens=200,
            system=SUMMARIZE_PROMPT,
            messages=[{"role": "user", "content": ticket_text}]
        )
        return response.content[0].text
    
    # Other methods for other tasks

Service layer = single place to swap providers, add caching, add retries.

Step 3: Add Error Handling (15 min)

def summarize_ticket(self, ticket_text: str) -> Optional[str]:
    try:
        response = self.client.messages.create(...)
        return response.content[0].text
    except anthropic.RateLimitError:
        # Backoff and retry
        time.sleep(2)
        return self.summarize_ticket(ticket_text)
    except anthropic.APIError as e:
        logger.error(f"LLM API error: {e}")
        return None
    except Exception as e:
        logger.error(f"Unexpected LLM error: {e}")
        return None

Don't crash the request because the LLM is slow. Degrade gracefully.

Step 4: Wire to Your Existing Flow (15 min)

# When a ticket is created
def create_ticket(ticket_data):
    ticket = Ticket.create(ticket_data)
    
    # Generate summary async (don't block the user)
    background_jobs.enqueue(generate_summary, ticket.id)
    
    return ticket

def generate_summary(ticket_id):
    ticket = Ticket.find(ticket_id)
    summary = llm_service.summarize_ticket(ticket.full_text)
    
    if summary:
        ticket.summary = summary
        ticket.save()

LLM work happens async. User experience isn't degraded by LLM latency.

Step 5: Handle the "No Summary" Case (10 min)

If the LLM call fails or is delayed:

{ticket.summary ? (
  <p>{ticket.summary}</p>
) : (
  <p className="text-muted">Generating summary...</p>
)}

UI handles the absence gracefully. User doesn't see broken state.

Step 6: Add Logging (15 min)

def summarize_ticket(self, ticket_text):
    start = time.time()
    
    try:
        response = self.client.messages.create(...)
        result = response.content[0].text
        
        logger.info("llm_call", {
            "task": "summarize_ticket",
            "tokens_in": response.usage.input_tokens,
            "tokens_out": response.usage.output_tokens,
            "latency_ms": (time.time() - start) * 1000,
            "model": self.model,
        })
        
        return result
    except Exception as e:
        logger.error("llm_call_failed", {
            "task": "summarize_ticket",
            "error": str(e),
        })
        raise

You'll need this data: cost analysis, performance monitoring, debugging.

Step 7: Add Feature Flags (10 min)

LLM features behind flags:

def summarize_ticket(ticket):
    if not feature_flags.is_enabled("llm_summarization"):
        return None
    return llm_service.summarize_ticket(ticket.full_text)

Easy to disable if something goes wrong. Easy to roll out gradually.

Step 8: Monitor Quality (varies)

In production:

• Sample 1% of LLM outputs for human review
• Track user feedback (thumbs up/down)
• Watch error rates
• Alert on unusual patterns

LLM behavior drifts. Monitor for regression.

Step 9: Build a Fallback (15 min)

If the LLM is down:

def summarize_ticket(ticket_text):
    summary = llm_service.summarize_ticket(ticket_text)
    
    if not summary:
        # Fallback: simple extractive summary
        return extractive_summary(ticket_text, max_sentences=3)
    
    return summary

System works even if LLM service has issues. Critical for production.

Step 10: Document the Integration (15 min)

For the team:

• What the LLM is used for
• Which prompts (versioned)
• Expected output format
• Error handling behavior
• How to disable
• How to debug failures
• Cost estimates

Future engineers (including future you) will benefit.

What You Just Did

You integrated an LLM into a real application with proper discipline: service layer, error handling, async execution, logging, feature flags, monitoring, and fallback.

Common Failure Modes

Sync LLM calls. User waits 5 seconds for the response. Move to async.

No fallback. LLM down = feature broken.

No logging. Can't debug; can't measure cost.

No feature flag. Can't disable in production.

LLM error crashes request. Wrap and degrade gracefully.

---

Continue the Building with LLMs path

• Previous — Part 1: Call Your First LLM API
• Next — Part 3: Stream Responses to a UI

Part of the Building with LLMs learning path.