Module 11: Production & MLOps for GenAI

11.1 Serving LLMs

Deploying LLMs for low-latency, high-concurrency client requests requires specialized serving frameworks:

• PagedAttention (vLLM): Traditional KV caching creates massive memory fragmentation because context lengths vary dynamically. vLLM solves this by borrowing Virtual Memory principles from operating systems. It shards the KV cache into small blocks and maps them to a lookup table, saving up to 96% of KV cache memory and doubling server throughput.
• Continuous Batching: Traditional batching requires waiting for the longest request in a batch to complete. In continuous batching, completed requests are ejected immediately, and new incoming requests are hot-swapped into the running GPU iteration, maximizing hardware efficiency.

11.2 Observability & Monitoring

Unlike deterministic APIs, LLM systems are non-deterministic, making tracing vital:

• Langsmith & LangFuse: Open telemetry platforms that capture step-by-step traces of agent loops, nested tool calls, prompt variables, and retrieved database chunks, logging cost and latency.
• Monitoring Metrics: We track:
  1. **TTFT (Time to First Token):** Measures system responsiveness (crucial for streaming chat).
  2. **Latency per Token:** Generation speed (tokens/sec).
  3. **Cost per Transaction:** Input vs. output token token pricing logs.
  4. **User Feedback Logs:** Capturing thumbs-up/down clicks to build SFT alignment training sets.

11.3 Cost Optimization

Running models at scale quickly becomes expensive. Optimizations include:

• API Prompt Caching: Large commercial providers (like Anthropic or OpenAI) cache static prefix system prompts and RAG contexts. If consecutive queries share identical prefixes, the server loads them instantly from cache at a 90% discount.
• Model Routing & SLMs: Deploying lightweight Small Language Models (SLMs, e.g. 1.5B/3B parameters) for simple routing, classification, or formatting tasks, and escalating to frontier 70B+ parameter models only for complex reasoning queries.

11.4 Guardrails & Safety

To prevent models from outputting toxic, illegal, or malformed data:

• NeMo Guardrails (Nvidia): An open framework to build programmable rails around LLMs, enforcing topical constraints, safety checks, and jailbreak defenses.
• Llama Guard: A specialized classifier fine-tuned to classify incoming prompts and outgoing completions across safety categories (e.g. self-harm, cyberattacks).
• Structured Output Libraries (Outlines, Instructor): Enforce strict structural formats (like JSON schemas or Pydantic models) directly at the decoding level. It modifies the LLM's output token logit distributions, ensuring the model *only* generates tokens that match the target syntax.

from pydantic import BaseModel, Field

class UserProfile(BaseModel):
    name: str = Field(description="The user's full name")
    age: int = Field(description="The user's age in years")
    skills: list[str] = Field(description="A list of technical skills")

# Passing this schema ensures the API returns valid JSON matching these fields
print(UserProfile.model_json_schema())

11.5 CI/CD for LLM Apps

Updating prompt templates, code tools, or model weights requires robust integration testing:

1. Prompt Versioning: Commit prompt templates to Git repositories alongside application code, avoiding hardcoded string constants in servers.
2. Automated Regressions: Set up build assertions that query models against evaluation test sets. If accuracy drops below a threshold (e.g., code output parser fails), the CI pipeline halts deployment.
3. Shadow Deployments: Send incoming production queries to both the stable model and the new prompt candidate in parallel, analyzing latency and output distributions before shifting traffic.