Notice: Resolution of Performance Slowdown April 1st, 2025 (Updated as of: 8:30 PM ET)
We experienced a temporary performance slowdown earlier today, April 1st, which has now been fully resolved.
Root Cause Analysis (RCA):
The slowdown was triggered by an unexpected spike in system load at the start of the quarter. This surge resulted from two concurrent processes:
- A high volume of end-of-quarter (Q1) reporting by users.
- Scheduled day 1 background jobs, including user notifications.
Both processes competed for the same backend resources, primarily database capacity and background processing queues, leading to performance degradation.
Resolution:
We took the following immediate actions:
- Restarted and increased the number of background workers.
- Significantly scaled up our primary database memory threefold (3x).
Current Status:
System performance is back to normal, and we are actively monitoring all systems.
Future Improvements:
To prevent recurrence, we will:
- Separate report processing into dedicated queues.
- Utilize replica databases for report reads to reduce primary database load.
- Migrate reports to the NextGen Reporting Engine.
Thank you for your patience as we worked through these issues.
Previous Update: April 1st, 2025 (Updated as of: 5:30 PM ET)
We are continuing our investigation into the reported intermittent platform issues and are actively working on implementing updates. While we believe some system improvements have been made, we are still monitoring the system closely and working towards a complete resolution. We will provide further updates as they become available.
Previous Update: April 1st, 2025 (Updated as of: 2:50 PM ET)
We are currently experiencing platform-related issues that are impacting the following areas:
- OKR/Goal Updates: Users may encounter difficulties updating or modifying their OKRs and Goals.
- UI Lag Times: The user interface may exhibit slow response times or general lag.
- Reports/Notifications: Issues related to receiving reports and notifications are being observed.
Our team is actively investigating these issues. While we do not currently have an ETA for resolution, rest assured we are working diligently to restore full functionality as quickly as possible. We understand the inconvenience this may cause and appreciate your patience.
For further assistance, please contact our support team at support@betterworks.com.