Production systems
From AI Prototype to Production: What Actually Changes
The work that begins when a promising demo meets permissions, evaluation, failure, cost, and real users.
A prototype answers “can this behavior work?”
A prototype should reduce a specific uncertainty. Can the available documents support the questions users ask? Can a model classify the real inputs? Can latency remain acceptable across a tool call? Does the proposed interaction help a user complete the task?
That is valuable evidence. It is not the same as an operated system. A production application must behave across permitted and restricted users, incomplete input, upstream failure, provider changes, concurrency, retries, support cases, and the long tail of real work.
The gap is not solved by “hardening” one prompt. It is solved by designing the product and engineering system around variable model behavior.
Replace anecdotal success with evaluation
Teams often begin by trying several prompts and keeping the examples that look good. Production work needs a representative task set with explicit criteria. Include ordinary inputs, ambiguous cases, missing context, adversarial content, permission boundaries, and known operational failures.
OpenAI’s current Evals API formalizes the same basic pattern: an evaluation combines a data source with testing criteria and can run against model configurations. A team does not have to use one vendor’s evaluation product, but it does need repeatable evidence.
Measure the layers separately when possible:
- retrieval relevance;
- tool selection and argument validity;
- policy and authorization adherence;
- answer support and usefulness;
- task completion;
- latency and cost;
- fallback and escalation behavior.
A single aggregate score can hide dangerous trade-offs. Improvements on common cases may coincide with worse behavior on restricted or high-consequence ones.
Put identity and permissions outside the prompt
The application authenticates users and authorizes data and actions. A prompt may explain the user’s role to improve behavior, but it must not become the enforcement layer.
For retrieval, filter accessible sources before they enter model context. For tools, check authority when the function is executed. For remote MCP servers, follow the protocol’s authorization and token-audience requirements rather than forwarding a client token indiscriminately. The current MCP authorization specification explicitly prohibits token passthrough and requires protected-resource token validation.
Audit events should record the identifiers and outcomes required for diagnosis and accountability without turning logs into an unnecessary copy of sensitive input.
Design the unsuccessful path
Production systems need answers to questions a demo avoids:
- What if retrieval returns nothing useful?
- What if the model produces invalid structured output?
- What if a tool times out after partially succeeding?
- What if a request is duplicated?
- What if the user lacks permission?
- What if the provider is unavailable or rate-limits the call?
- What if the model is uncertain but still sounds confident?
The response may be a deterministic fallback, bounded retry, alternate provider, abstention, saved draft, queued job, or human review. The right behavior depends on the task’s consequence and timing.
For writes, use idempotency and confirmation where appropriate. Validate model-generated arguments exactly as you would validate other untrusted input. Never assume well-formed JSON means a requested action is permitted.
Operate latency, cost, and dependencies
Model applications often contain several serial dependencies: retrieval, reranking, a model call, tool execution, and another model call. Each adds latency and a failure surface. Measure the complete user path, not only one provider response.
Reduce unnecessary context and tool definitions. Cache stable results where permission and freshness rules allow it. Move long-running work out of the request path. Stream only when it improves comprehension, not to disguise an architecture that is too slow for the task.
Cost belongs in evaluation alongside quality. A more capable model may be justified for a narrow difficult step while a smaller model or deterministic code handles routine work. Provider selection is an architectural decision that should remain reviewable rather than embedded throughout the product.
Make releases reversible
Use feature flags, staged audiences, clear model and prompt versions, and rollback paths. Run the evaluation set when changing a model, retrieval configuration, tool schema, or important instructions. Review production failures and user corrections to extend that set.
Versioning matters because behavior can change even when an API remains compatible. Keep enough trace information to connect a user-visible result to the relevant model configuration, retrieved sources, and tool outcomes.
Assign operational ownership
Someone must own evaluation quality, incidents, access changes, provider updates, source synchronization, tool contracts, and user feedback. Without ownership, the prototype becomes a fragile feature no one feels safe changing.
Production readiness is not a single gate. It is the point at which the team can observe, explain, and improve the system under real conditions.
A useful launch checklist
Before widening access, confirm that:
- representative evaluation tasks run repeatedly;
- permissions are enforced at data and tool boundaries;
- consequential actions require the right validation or approval;
- timeouts, retries, duplicates, and partial failure are handled;
- important behavior is observable without excessive data collection;
- model, prompt, retrieval, and tool changes are versioned;
- the product has a fallback and a responsible operator;
- the rollout can be paused or reversed.
Sources and further reading
See How We Work and Custom AI Product Development for the delivery and product-engineering context around these controls.