AI Incident Readiness

Practice an AI Provider Outage Without Losing Your Controls

Rehearse reduced service, safe retry behavior, and approved fallback routes in an isolated drill that checks which controls remain active during an outage.

Top One Percent AI · Practical guidance with linked sources.

An AI provider becomes unavailable while a business workflow is running. The team has another model endpoint ready, and someone suggests switching immediately. Before traffic moves, several questions matter. Is that endpoint approved for this data? Does it use the same access controls? Will incomplete requests be repeated? Can staff keep working without copying sensitive material into personal accounts?

An outage exercise should test how the service operates under those constraints. Its purpose is to discover whether the organization can reduce functionality while preserving its essential controls, rather than simply proving that another endpoint returns an answer.

Choose the reduced service before the incident

NIST's contingency-planning guidance recognizes alternate equipment and manual processing among the ways to restore disrupted services. Its AI Risk Management Framework also includes consideration of viable non-AI alternatives under MANAGE 2.1.

For a practical exercise, choose one workflow and define its reduced mode. A support assistant might stop generating answers while staff continue using an approved knowledge base. A document-processing service might accept uploads into a controlled queue without generating recommendations. A higher-consequence workflow might stop entirely until authorized processing is available.

Document the trade-off explicitly. State which functions continue, which stop, who makes the switch, and how users learn what is happening. Include the people and capacity needed for manual work. A fallback that depends on an unstaffed team is not an operational plan.

Record the controls that must survive

Create a short control record for the normal path and each proposed alternative. Include approved data destinations, access restrictions, content checks, output validation, required human approvals, logging, and retention. Mark differences honestly rather than assuming that two models with compatible interfaces behave identically.

Before the exercise, require an owner to explain how each essential control operates in the reduced mode. If an alternative cannot meet a requirement, remove that function from the fallback or obtain the appropriate review in advance. Do not let urgency silently authorize a new data recipient or a broader set of actions.

These are recommended preparation steps. They turn “we have a backup provider” into a testable claim about the complete workflow, including its dependencies and human procedures.

Limit retries and resolve uncertain work

An outage can become worse when every component keeps trying. Amazon's Builders' Library explains how retries can increase load on an already struggling service. It describes backoff, increasing the wait between retries, and jitter, adding random variation to that wait, as ways to reduce synchronized retry pressure. It also warns that a timeout does not prove that a side effect failed to occur.

Apply that warning to the business action, not just the model request. If an assistant submitted a record update before losing its connection, resending the whole workflow could repeat the update. In the exercise, include one request whose outcome is unknown. Require the team to reconcile its state with the destination system before repeating the action, or demonstrate an effective duplicate-prevention mechanism.

Check retry limits across the application, gateway, and client library together. A local setting can look reasonable while the combined behavior creates repeated work. Give queued requests an expiry and a cancellation path so that stale instructions do not execute long after the user has moved on.

Test routing without assuming equivalence

Infrastructure can support controlled failover. An application programming interface, or API, connects one software component to another. For example, Microsoft documents circuit breakers and backend pools in Azure API Management. A tripped circuit breaker stops requests to a backend for a defined time, while priority-based pools can direct requests to lower-priority backends under specified conditions.

Those capabilities do not establish that a particular fallback preserves the organization's controls. Test the actual configured path, including what happens to conversation state, data restrictions, approvals, and queued work. A route that stays available but bypasses a mandatory check should fail the exercise.

Give organizers a bounded drill

Run the drill in an isolated test environment with synthetic data and permission from the responsible service owner. Do not disrupt a live provider or business workflow for a conference demonstration. Simulate unavailable responses, increased latency, and an uncertain completion, then observe the application rather than relying only on a verbal walkthrough.

Ask participants to produce three artifacts. First, a reduced-service card stating what users can still do. Second, a control record showing which safeguards remain active. Third, a work ledger distinguishing completed, queued, cancelled, and unresolved requests.

Measure time to enter the reduced mode, the amount of duplicate work, and any control bypasses found. Treat these as exercise observations, not universal performance targets. Assign owners to the gaps and repeat the affected tests after fixes. The most useful result is evidence that a provider failure does not force the organization to choose between uncontrolled improvisation and an unexplained halt.