Voice AI Service Disruption Report November 11
Resolved
Dec 12 at 03:08pm CET
Resolution
Our team implemented an automated restart mechanism to quickly restore database connectivity. The fix ensures faster recovery from similar connection issues in the future.
Prevention
- Enhanced monitoring for Redis connection health
- Automated restart mechanisms now in place
- Investigating additional redundancy options
Affected services
Courserev Voice API
Created
Dec 12 at 02:05pm CET
Post-Incident Report: Voice AI Service Outage - November 11, 2024
Summary
From 7:00 PM to 11:30 PM EST on November 11th, our Voice AI booking service experienced an outage due to a Redis database connection failure. Our automatic failover system immediately detected the issue and successfully routed all incoming calls to golf course pro shops, ensuring uninterrupted service for customers.
Impact
- Voice AI booking system unavailable
- All calls automatically and successfully redirected to pro shop staff via failover system
- Web booking remained fully operational and unaffected
- Zero dropped calls
Root Cause
A connection outage occurred on one of our Redis database instances, marking the first such failure in over 2 years of operation. The connection loss prevented the Voice AI service from accessing necessary session data.
Resolution
Our team implemented an automated restart mechanism to quickly restore database connectivity. The fix ensures faster recovery from similar connection issues in the future.
Prevention
- Enhanced monitoring for Redis connection health
- Automated restart mechanisms now in place
- Investigating additional redundancy options
Our automatic failover system performed as designed, ensuring customers could complete their bookings without interruption. We continue to invest in system reliability and redundancy.
Affected services
Courserev Voice API