Critical third-party quotas need operational headroom
- Source incident
- Podcast outage·Post-mortem
- Product area
- Podcast·Product area
Lesson
Any critical generation dependency needs capacity planning, quota thresholds, and a fallback path. Treat vendor quotas like production capacity, not a billing detail.
Why
A single exhausted text-to-speech quota stopped podcast generation for every affected learner, and the team first heard about it from support tickets, not monitoring.
What we changed
- Quota usage alerts fire at 80 percent for every critical vendor
- A fallback speech provider takes over when the primary is out
- New dependencies get a capacity review before launch
From the Jul 4 incident review