LLM Provider Migration Checklist: Move an AI Application Without Breaking Prompts, Tools, or Outputs

Overview

LLM provider migration usually affects several layers at once: authentication, request formats, model behavior, tokenization, context limits, tool calling, structured output, streaming, safety controls, observability, latency, and billing. Two APIs may expose similar chat-completion concepts while differing in how they interpret system instructions, validate schemas, report usage, or handle failed tool calls.

The safest approach is to define a provider-neutral contract around your application. Your business logic should ask for capabilities such as “return a validated invoice classification” or “select one of these tools,” rather than depending directly on a vendor-specific response shape. A thin adapter can then translate that contract to each provider.

Before changing code, capture a baseline from the current system. Record representative prompts, model settings, tool definitions, expected output formats, latency measurements, error rates, token usage, and human or automated quality judgments. Include difficult cases, not only successful examples. For a retrieval-augmented app, preserve questions with weak retrieval, conflicting documents, and no-answer cases. For an agent, include tool failures, ambiguous requests, and incomplete execution.

Use a versioned evaluation set throughout the migration. A practical prompt testing framework can help compare prompt versions and model responses without relying on memory or isolated demonstrations.

Provider capability comparison

Checklist by scenario

Scenario 1: Migrating to another provider

1. Inventory dependencies. List every model identifier, endpoint, SDK method, environment variable, retry rule, timeout, streaming handler, moderation check, and usage parser.
2. Separate application logic from transport logic. Put provider-specific request and response translation in an adapter. Keep business rules, validation, routing, and persistence outside that adapter.
3. Normalize the internal response. Define fields such as text, structured data, tool calls, finish reason, usage, request ID, and provider error. Preserve the raw response for debugging where your data-handling rules permit it.
4. Recreate the baseline. Run the same evaluation set against the existing integration and the candidate provider. Compare correctness, refusal behavior, citation quality, tool selection, schema validity, latency, and cost-related usage signals.
5. Roll out gradually. Start with offline tests, then shadow traffic or an internal cohort, followed by a small production percentage. Keep the old integration available until the new path meets predefined acceptance criteria.

Scenario 2: Moving between model families at the same provider

A model swap can change behavior even when the API contract remains stable. Recheck instruction priority, verbosity, reasoning patterns, context handling, tool selection, structured output, and sensitivity to sampling settings. Do not assume that a prompt optimized for one model will produce equivalent results on another.

• Freeze the prompt and application version before testing the new model.
• Test short, long, multilingual, adversarial, and ambiguous inputs.
• Compare outputs against task-specific rubrics rather than judging only fluency.
• Revisit maximum input and output sizes, truncation behavior, and timeout settings.
• Recalibrate routing rules if the new model is used for only certain task classes.

Scenario 3: Migrating an agent or tool-using workflow

Tool calling deserves its own compatibility test. Create a worksheet for each tool with the following fields:

• Tool name and description
• Parameter names, types, required fields, and allowed values
• Whether the model may call zero, one, or multiple tools
• How invalid arguments are reported and repaired
• How tool results are serialized and returned to the model
• What happens when a tool times out, fails, or returns no data
• Whether the system requires confirmation before side effects

Use deterministic application-side validation before executing a tool. The model should not be trusted to enforce authorization, rate limits, transaction boundaries, or irreversible-action safeguards. For broader agent design considerations, see this guide to AI agent architecture and stack selection.

Scenario 4: Migrating structured outputs

First classify your current guarantee. Is the application receiving valid schema-conforming data, JSON-like text, or ordinary prose that a parser happens to accept? Then test nested objects, optional fields, arrays, escaped characters, null values, and refusal or no-result states.

Prefer a typed internal schema with explicit validation and a controlled repair or retry path. A failed parse should become an observable application state, not an unhandled exception or silent data loss. If the destination provider cannot offer the same structured-output guarantee, decide whether to add a constrained decoding feature, a parser-and-validator loop, a fallback model, or a human review queue.

What to double-check

Prompt adaptation

Keep the task objective, constraints, examples, output contract, and refusal instructions distinct. Avoid changing all of them at once. A useful migration sequence is:

1. Send the original prompt through the new adapter.
2. Identify differences caused by message formatting or unsupported fields.
3. Make the smallest prompt change that addresses the difference.
4. Run the complete evaluation set and record the result.
5. Commit the prompt, adapter, test results, and model configuration together.

For example, a provider-neutral instruction might say: “Classify the ticket into exactly one allowed category. Return an object with category and confidence_band. If the evidence is insufficient, use needs_review.” The application should still validate the category and confidence band after the response. Few-shot examples can help, but test whether they improve the target task rather than assuming more examples are better.

Privacy, retention, and observability

Review what data is sent to the destination, which logs contain prompts or outputs, how request identifiers are propagated, and whether sensitive fields are redacted. Provider migration is a useful time to revisit your AI data privacy controls rather than copying existing assumptions into a new integration.

Fallback and rollback template

Primary integration: [provider/model/version]
Fallback integration: [provider/model/version]
Fallback trigger: [timeout, rate limit, validation failure, quality signal]
Maximum retry budget: [count and time window]
Traffic split: [primary percentage / fallback percentage]
Rollback owner: [team or role]
Rollback command or flag: [configuration reference]
Acceptance criteria: [quality, latency, error, and cost thresholds]
Review date: [date]

Fallbacks should be tested under realistic failure conditions. A fallback that has never received current prompts, tool schemas, and traffic volume is only a theoretical safety net.

Common mistakes

• Changing provider, model, prompt, and business logic simultaneously. This makes regressions difficult to attribute. Change one layer at a time where practical.
• Testing only happy paths. Include malformed inputs, empty retrieval results, tool errors, refusals, long context, and concurrent requests.
• Comparing prose instead of outcomes. A different wording may be acceptable, while a subtly wrong extracted field may not be.
• Assuming SDK compatibility means behavior compatibility. An SDK can make requests succeed while semantics, limits, and output guarantees differ.
• Ignoring usage accounting. Verify that input, output, cached, batch, or reasoning-related usage fields—when exposed—are mapped correctly for monitoring and budgeting.
• Letting the model enforce security. Authorization, tenant isolation, input validation, and approval requirements belong in application controls.
• Deleting the old path too early. Retain a tested rollback route until the migration has passed normal traffic patterns and an agreed observation period.

If the application uses retrieval, compare retrieval and answer quality separately. A model migration can appear to reduce answer quality when the actual change is in context formatting or document truncation. Use a RAG evaluation checklist to isolate those layers.

When to revisit

Keep this checklist with the application’s runbooks and review it before seasonal planning cycles, contract or vendor reviews, major model changes, SDK upgrades, endpoint deprecations, and significant workflow changes. Revisit it after adding tools, changing schemas, introducing a RAG pipeline, modifying retention controls, or altering traffic volume.

At minimum, schedule a migration rehearsal on a representative test set. Refresh the capability comparison whenever a provider changes model identifiers, limits, SDK behavior, structured-output support, tool-calling semantics, or usage reporting. Do not rely on an old comparison table for current purchasing or architecture decisions; verify details in the provider documentation and your own integration tests.

Before approving a switch, confirm these final actions:

1. Tag the current production configuration and store a rollback artifact.
2. Run baseline and candidate evaluations with versioned prompts and fixtures.
3. Review tool, schema, privacy, observability, and usage compatibility.
4. Launch behind a feature flag with clear owners and alerts.
5. Monitor quality and operational signals during a staged rollout.
6. Record the decision, known differences, follow-up tests, and next review date.

A provider migration is complete when the application’s important behaviors are measured, not merely when requests succeed. Treat the checklist as a living production document so the next model or vendor change starts with evidence instead of guesswork.