Tutorial 3: Stream Responses to a UI

Contributor
4 days ago
3 min read

A 5-second wait for a response feels broken. Streaming token-by-token feels alive. This tutorial walks through implementing it.

What You'll Build

A streaming response from LLM through your backend to your frontend, displayed token-by-token.

Step 1: Stream from the LLM (10 min)

def stream_llm_response(prompt: str):
    with client.messages.stream(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    ) as stream:
        for text in stream.text_stream:
            yield text

Generator yields tokens as they arrive.

Step 2: Server-Sent Events Endpoint (20 min)

from flask import Response, request, stream_with_context

@app.route('/api/chat/stream', methods=['POST'])
def chat_stream():
    prompt = request.json['prompt']
    
    def generate():
        for token in stream_llm_response(prompt):
            # SSE format
            yield f"data: {json.dumps({'token': token})}\n\n"
        yield "data: [DONE]\n\n"
    
    return Response(
        stream_with_context(generate()),
        mimetype='text/event-stream',
        headers={
            'Cache-Control': 'no-cache',
            'X-Accel-Buffering': 'no',  # nginx
        }
    )

SSE is simpler than WebSockets for one-way streaming.

Step 3: Frontend Reader (20 min)

async function streamChat(prompt: string, onToken: (token: string) => void) {
  const response = await fetch('/api/chat/stream', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ prompt }),
  });
  
  const reader = response.body!.getReader();
  const decoder = new TextDecoder();
  
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    
    const chunk = decoder.decode(value);
    const lines = chunk.split('\n');
    
    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const data = line.slice(6);
        if (data === '[DONE]') return;
        const { token } = JSON.parse(data);
        onToken(token);
      }
    }
  }
}

Reads the stream; calls onToken per token.

Step 4: React UI (15 min)

function ChatBox() {
  const [response, setResponse] = useState('');
  const [streaming, setStreaming] = useState(false);
  
  const sendPrompt = async (prompt: string) => {
    setStreaming(true);
    setResponse('');
    
    await streamChat(prompt, (token) => {
      setResponse(prev => prev + token);
    });
    
    setStreaming(false);
  };
  
  return (
    <div>
      <pre>{response}</pre>
      {streaming && <Spinner />}
    </div>
  );
}

State updates per token. UI re-renders smoothly.

Step 5: Handle Cancellation (10 min)

User clicks "stop":

const abortController = new AbortController();

const response = await fetch('/api/chat/stream', {
  signal: abortController.signal,
  // ...
});

// On stop button click
abortController.abort();

Stops the stream immediately. Saves tokens (you don't pay for what's not generated, but you stop early).

Step 6: Handle Errors Mid-Stream (10 min)

Streams can fail partway:

def generate():
    try:
        for token in stream_llm_response(prompt):
            yield f"data: {json.dumps({'token': token})}\n\n"
        yield "data: [DONE]\n\n"
    except Exception as e:
        yield f"data: {json.dumps({'error': str(e)})}\n\n"

Frontend handles error events:

if (data.error) {
  setError(data.error);
  return;
}

Step 7: Backpressure (15 min)

If the client is slow, server can buffer too much. Use HTTP/2 or careful flushing.

Most setups handle this OK for typical LLM rates. For high-volume, consider WebSockets with proper backpressure.

Step 8: Reconnection (varies)

If the connection drops, the stream is lost. For high-value cases:

Cache the full response server-side as it streams
On reconnection, send what's accumulated
Resume the stream from where it stopped (if possible)

Complex. For most apps, just retry the whole thing.

Step 9: UX Polish (15 min)

Small things that matter:

Cursor indicator while streaming (blinking caret)
"Stop" button during streaming
Auto-scroll as content grows
Markdown rendering at end of stream (or smartly during)
Code syntax highlighting when relevant

Step 10: Production Concerns (varies)

For production:

Proxy buffering: disable in nginx/load balancer (X-Accel-Buffering)
Timeout: server timeout must exceed expected stream duration
Rate limiting: still apply, but care about stream durations
Cost capping: monitor; some users may abuse

What You Just Did

You built end-to-end streaming. Responses feel responsive; users engaged; UX matches modern AI products.

Common Failure Modes

Buffering proxy. SSE doesn't flush; user waits then sees all at once.

No cancellation. User waits for unwanted content.

Sync UI updates. State updates per token; React re-renders too aggressively. Batch.

No error handling mid-stream. Failure halfway leaves UI confused.

Stream from cache. "Stream" the same response over and over from cache — defeats UX.

ShiftQuality