LLM Integration

Integrating an LLM into a product is less about the model and more about the contract around it: reliable structured output, graceful fallbacks, and observability. I treat the model as a component with well-defined inputs and outputs, not a magic box.

On the engineering side I bake in JSON-schema-constrained responses, tool/function calling, retries, and token budgeting. Prompt work is versioned and tested the same way code is, so drift gets caught instead of silently degrading answers.

Model selection is intentional too: frontier models where judgment matters, small or local models where speed and cost do. A router or a simple lookup decides per-call, and caching + batching keep spend predictable.

Structured output is the load-bearing feature. Rather than parsing prose, I constrain the model to a typed schema so downstream code gets predictable objects — no regex archaeology, no "the answer came back differently" surprises. That is the difference between a demo and a product.

Tool and function calling turn a model into an actor over systems. I define a small, typed tool surface — read, write, search — and let the model decide when to call it, with hard limits on calls per turn and explicit permission on anything destructive.

Reliability is engineered, not hoped for. Retries with exponential backoff, timeouts tuned per model, fallback chains down to a smaller model or a rule-based answer, and idempotency so a duplicate LLM call cannot double-effect. The model is a dependency like any other, and dependencies fail.

Cost and latency are designed in from the start. I budget tokens per request, cache identical/similar calls, batch where safe, and route requests by complexity — because a backend that burns the expensive model on every trivial question is not an integration, it is a bill.

Prompts are versioned, evaluated, and deployed like code. I keep them with the code, tie them to an eval set that runs in CI, and review output samples on every change so a prompt tweak cannot silently degrade quality.

I keep provider abstraction honest: a narrow integration layer that can route between OpenAI, Anthropic, Gemini, or local models at runtime. That is less about vendor-hopping and more about resilience — if one provider degrades, the product does not.

I define the contract before I pick the model. The integration begins with a typed in/out: what the LLM receives, what it must return, and the schema that makes the output usable by the rest of the system.

Structured output is the default, not the exception. I constrain the model to a JSON schema so downstream code gets predictable objects and the parsing layer disappears — that is the difference between a demo and a product.

Tool use is a typed, bounded surface. I give the model a small set of capabilities with explicit side effects and limits, and let it call them when it must — with hard ceilings and approvals on anything destructive.

Reliability is engineered like any dependency. Retries with backoff, timeouts tuned per model, fallback chains to a smaller model or a rule, and idempotency so a duplicate call cannot double-effect.

Cost and latency are designed in, not audited after. I budget tokens per request, cache repeated calls, and route simple work to a cheap model — so the LLM is a tool, not an open wallet.

Prompts are versioned and evaluated. They live with the code, are tied to an eval set that runs in CI, and output samples are reviewed on every change so a prompt tweak cannot silently degrade quality.

Models are abstracted behind a narrow provider layer. The integration can route between OpenAI, Anthropic, Gemini, or local models at runtime, which is resilience rather than vendor-hopping.

Observability covers the model, not just the pipeline. I trace every request, sample output, and track latency, tokens, and cost per call — so the LLM is a measured component, not a black box.

I am honest about when an LLM is the wrong tool. Where a deterministic rule or a search index answers better, faster, and cheaper, I use that instead — the model is the last resort for judgement, not the first for everything.