The production failure I would design for first is the half-finished operation. A dependency times out after useful work may already have happened, and the application no longer knows whether the action completed. Resolve that ambiguity before users depend on the workflow.
Suppose a leave submission times out. The application cannot tell whether HR accepted it, and a blind retry could create a duplicate. Record the outcome as uncertain, look up the operation using its idempotency key and return the authoritative status. If the status still cannot be resolved, tell the user that it is unknown and provide a safe escalation path.
Use bounded retries for temporary failures, then stop. Invalid input and denied operations need different handling. A circuit breaker can suspend a repeatedly failing dependency. Queued work also needs expiry and cancellation so an old request cannot execute after the user's intent has changed.
Keep an operating record that supports decisions
Track completion rate, latency percentiles, queue depth, token use, tool failure patterns, permission denials and review burden. Percentiles reveal slow experiences that averages can hide. Group the results by task and deployment version, using only privacy-appropriate segmentation.
For audit, record verified events and concise decision metadata. For a write, that means the actor, validated operation, approval evidence where required, execution time and authoritative result. Generated explanations can help people understand what happened, but the audit record should rely on events the system actually verified.
Set capacity and cost limits against expected demand. A shared model endpoint can become slow under concurrency even when a single request performs well. Load tests should include realistic input and output lengths, retrieval, tool calls and cancellation. Optimize the complete task, not an isolated token-speed number.
Every material change to a model, prompt, index, policy or tool should leave an evaluation record and a rollback path. Pin versions where possible. When a provider changes behaviour without a fixed version, increase monitoring and keep either a tested alternative or a graceful reduced-service path.
An incident procedure should identify who can disable a tool, revoke a credential, stop ingestion and preserve relevant evidence. Notify affected owners through the established process. After recovery, turn the failure into a regression case and revisit whether the original scope is still appropriate.
I would call the service operable only when the team can explain its failed states as clearly as its successful path. Hosting the model elsewhere does not transfer responsibility for ownership, recovery or rollback.
Part D goes underneath the application layer. Most readers need to understand the relationships between training, inference, model adaptation and hardware. The deeper implementation detail matters when those tradeoffs become part of your job.
- 23 Separate training from inference
- 24 Choose model adaptation and deployment
- 25 Read the seven-layer map correctly
- 26 Understand GPU and memory constraints