Serval - Third Party: OpenAI provider outage – Incident details

All systems operational

Third Party: OpenAI provider outage

Resolved
Major outage
Started about 1 month agoLasted 9 hours 35 minutes

Affected

Third Party: OpenAI

Major outage from 5:07 PM to 5:14 PM, Degraded performance from 5:14 PM to 5:21 PM, Major outage from 5:21 PM to 5:32 PM, Degraded performance from 5:32 PM to 8:16 PM, Operational from 8:16 PM to 10:51 PM, Degraded performance from 10:51 PM to 11:13 PM, Operational from 11:13 PM to 2:43 AM

Third Party: OpenAI → APIs → Chat Completions

Major outage from 5:07 PM to 5:14 PM, Degraded performance from 5:14 PM to 5:21 PM, Major outage from 5:21 PM to 5:32 PM, Degraded performance from 5:32 PM to 8:16 PM, Operational from 8:16 PM to 2:43 AM

Third Party: OpenAI → APIs → Responses

Major outage from 5:07 PM to 5:14 PM, Degraded performance from 5:14 PM to 5:21 PM, Major outage from 5:21 PM to 5:32 PM, Degraded performance from 5:32 PM to 8:16 PM, Operational from 8:16 PM to 10:51 PM, Degraded performance from 10:51 PM to 11:13 PM, Operational from 11:13 PM to 2:43 AM

Updates
  • Resolved
    UTC
    Resolved

    We are continuing to use Anthropic for all agent responses as we monitor OpenAI's recovery. We will return to normal operation once API responsiveness returns to expected baseline.

  • Update
    UTC
    Update

    We are continuing to use Anthropic for all agent responses as we monitor OpenAI API incident response and recovery. Follow OpenAI's status updates here: https://status.openai.com/incidents/01M1CYZDD84XD0H56XNEV2PT98

  • Update
    UTC
    Update

    We are continuing to use Anthropic for all agent responses as we monitor OpenAI API recovery.

  • Update
    UTC
    Update

    We are continuing to observe issues with OpenAI API responsiveness, so will maintain failover to Anthropic models and continue monitoring.

  • Update
    UTC
    Update

    We are recording improved API responsiveness from OpenAI and are evaluating a progressive rollback to normal operations.

  • Update
    UTC
    Update

    We have completed failing remaining inference activity to Anthropic and are continuing to monitor OpenAI endpoints for recovery.

  • Update
    UTC
    Update

    We are continuing to failover to Anthropic and are implementing other mitigations.

  • Update
    UTC
    Update

    We are continuing to failover to Anthropic models for remaining affected agent behaviors and monitoring rollout.

  • Update
    UTC
    Update

    We are continuing to observe inconsistent response latencies with OpenAI endpoints. We are deploying a mitigation to failover all remaining affected model usages to Anthropic.

  • Monitoring
    UTC
    Monitoring

    We are seeing consistent availability from the OpenAI endpoint, but performance remains degraded. We are continuing to monitor the situation.

  • Update
    UTC
    Update

    We are continuing to see fluctuating availability and response times from OpenAI across multiple model classes. We are continuing to investigate and implement mitigations.

  • Update
    UTC
    Update

    We are currently investigating this issue with OpenAI. We are seeing some inconsistent response times from OpenAI.

  • Investigating
    UTC
    Investigating

    We are currently experiencing elevated error rates and API hangs communicating with OpenAI. We are investigating the issue.