Controlling model changes: evaluation to retirement

Treat a new model version as a new route with a distinct model_name. Keep the candidate disabled in production until it has passed the checks that matter to its workload. Use a standalone route and pin the provider identity when the provider contract permits it and an exact version is required.

This page covers the whole change: evaluation, enablement, observation, reversal, retirement, and the record of who decided. It applies when a route enters a production pool or a change can affect selection, the application contract, a data boundary, cost, or a business outcome. For a metadata-only change, test only the selection or reporting behavior it can affect.

Follow the change from creation to retirement

Lifecycle of a model changeCreate the new route disabled in the target Tenant. Enable a copy in an evaluation Tenant and test representative work. If it fails, keep it disabled. If it passes, enable it in the target pool and observe application outcomes. If outcomes regress, disable the new route and keep the previous one. When outcomes are accepted and the previous route is no longer needed for rollback, retire the previous route.

Classify what can change

Use the affected boundary to choose the checks:

A managed preset can change preset-owned fields, and a rolling provider alias can move to another vendor revision without a Tenant edit. A preset-backed route is therefore not an exact rollback snapshot. If exact comparison or rollback matters, use a standalone route, pin the provider version where possible, and retain a secret-redacted copy of the resolved configuration.

Define success for this change

Before you enable the candidate, define the acceptable application result and the condition that will make you reverse the change. Use representative requests and check only the affected behavior. Depending on the change, this can include:

• the input, response, and Tool contracts used by the application;
• domain rules and External Effects;
• latency, deadline, quota, and cost limits;
• behavior when the provider or complete pool fails; and
• an applicable security, privacy, contractual, or regional boundary.

A response that matches its schema is not necessarily correct or acceptable for the business process. Model-reported confidence is not an approval signal.

Qualify and enable the route

1. Keep the candidate disabled in the target Tenant. If it does not exist yet, create a disabled evaluation route first: new routes default to enabled. Saving validates the configuration shape but does not test the provider.
2. Enable a copy in an evaluation Tenant. Create a copy with the same route settings and enable it as that Tenant’s only route. A disabled route cannot serve requests. Send representative requests with the same response schema, Tools, and business validation used by the target application.
3. Check the declarations against measured behavior. Open Settings -> LLM Providers -> Performance and read the route’s Structured-output errors, Tool-call errors, and Provider cache hits.
4. Test the intended pool when competition matters. In the evaluation Tenant, add the other eligible routes when the change depends on fallback or relative selection. The test does not establish a fixed order of provider requests.
5. Enable the target route after acceptance. Observe application outcomes after the change, as the next sections describe.

A structured-output error means that one provider request did not produce a valid answer. Compare the rate with your evaluation evidence before you accept the route for this workload. Provider cache hits show whether a context_caching declaration earns the cached-input rate you priced it at. The Performance view covers the first provider request of each model response on enabled routes, so read it as evidence about the route you selected, not as a record of what happened after a failover.

Routing contract and limits defines pool selection and the limits on provider requests.

Test only the claims you rely on

Prepare the application’s failure path before you rely on recovery. Test the no-route path first. If a schema-valid but unacceptable result can cause material harm, test that the application rejects it before any business action. Secure integration covers controls for External Effects.

Add a focused test only when the application depends on the corresponding claim:

The normal Task Result does not identify a route or the order of provider requests. A test can prove the terminal behavior of the application, but not which fallback route ran. Provider analytics, model analytics, and DNS data alone do not prove the provider account or processing location.

Observe after enablement

Use each signal for what it proves:

Task Results do not contain route, provider-request, fallback, or cost data. Analytics can be delayed, so a missing entry does not prove that no provider request occurred. Compare configured-cost analytics with the provider invoice when spend matters; routing preferences are not a hard budget.

Act on a visible problem

• No route succeeds: use the application’s terminal failure path. Do not add a route that has not passed the checks required by the workload.
• Business output regresses: stop using the affected result and disable the candidate when the rollback condition is met. Use application records to assess affected work.
• Cost or quota is unexpected: limit the work your application submits, then compare application use, documented analytics, configured prices, and provider records.
• Configuration differs from what you saved: read the configuration back and save the intended values again. Reading a configuration back shows what you saved, not that every new provider request already uses it.
• A credential or data path is compromised: stop the affected transfer and revoke the credential or block the network path.

Stopping submissions in your application stops new work. A route change does not cancel active work or undo an Effect. A caller holding agent:stop_task on the Task can request its Stop (Task actions); acceptance is not a terminal outcome. Stop the Task you intend to stop defines what Stop ends.

Reverse the change

Disable the candidate to remove it from route selection. The change applies to new provider requests after a short delay. It does not cancel active work or undo a provider request or Tool Effect that already started. If work must continue through another route, make that stable path available before you disable the candidate; those configuration writes are separate operations.

For a compromised provider credential or network path, revoke the credential or block the path. Disabling a route only changes later selection. Use one deadline for each Task and Stop the Task you intend to stop own the deadline, Stop, and active-work boundaries.

Retire the previous route

Keep the previous route while it is still required for rollback. When it is no longer needed:

• disable it and confirm that no intended workload still depends on it;
• revoke only credentials and quotas that no other route uses; and
• delete the configuration only when you no longer need recovery from it.

Deletion is irreversible and has no restore operation or write precondition. Do not delete a published preset key while another configuration can still reference it.

Assign owners and record the change

Assign only the responsibilities that a routing change needs. Every production change needs an outcome owner, an authorized person who applies it, and someone who verifies the result. Add a boundary approver and a named stop owner when the change affects provider, data, security, legal, financial, availability, or customer commitments.

One person can cover several responsibilities in a low-risk environment. For a material change, separate the requester, approver, and change implementer when your policy requires independent approval. Product roles do not enforce that separation. Duale AI permissions authorize product actions; a custom policy can separate existing actions, but it cannot add proposal or approval states. Access needed to change a route lists the configuration actions.

Keep a decision record in your own change system when a change:

• adds or changes a provider, account, region, Tenant, deployment, route, Tool, Connection, or credential;
• changes an approved data, contract, cost, capability, availability, or security boundary;
• accepts an exception or failed test; or
• responds to an incident or retires a shared dependency.

Record only what the decision needs: the affected scope and purpose; secret-redacted old and new state; outcome owner, approver, implementer, verifier, and stop authority; supporting evidence and tests; effective and review dates; rollback or cutoff; and any accepted residual risk. Product configuration and audit events can support verification, but they do not record the approval decision of your organization.