Digest emails are delayed

We are scaling notification workers after elevated queue depth. Core intranet features remain available.

Impacted services: Notifications pipeline

Availability metrics

Rolling 30-day uptime

99.94%

Rolling 90-day uptime

99.97%

Data refreshed Oct 4, 2025, 9:45 AM

Serving all regions with sub-200ms median response times.

Operational

Google Workspace SSO and session refresh performing as expected.

Operational

Digest emails are delayed ~10 minutes while we scale queue workers.

Degraded

New content indexed within 2 minutes of publish.

Operational

Sep 28, 2025, 12:12 PM → Sep 28, 2025, 1:05 PM

Resolved

Users saw intermittent errors when searching for events. Refresh would occasionally succeed.

Follow-up: Implemented automated shard reallocation alerts and increased cluster capacity by 20%.

Sep 28, 2025, 12:18 PM
Engineers engaged search cluster monitoring after alert firing.
Sep 28, 2025, 12:32 PM
Identified shard imbalance after overnight reindex job; rebalancing cluster.
Sep 28, 2025, 1:05 PM
Traffic stabilized; monitoring for 30 minutes before all-clear.

Sun, Oct 12, 1:00 AM EDT – 3:00 AM EDT

Applying security patches to the Aurora MySQL cluster. Read-only mode expected for up to 30 minutes during the window.

Impacted: Database, Intranet web app

Sun, Oct 26, 2:00 AM EDT – 4:00 AM EDT

Upgrading the Kafka cluster backing the digest service. Brief interruptions to scheduled digests may occur.

Impacted: Notifications pipeline