Post-Incident Summary

Date: 29 April 2026

Summary

A gradual increase in Records API response payload size eventually exceeded the memory available to the pods serving those requests, causing them to be terminated. As pods were terminated and replaced, the load balancer returned 502 errors for requests routed to them. The growth had been building over several days and was not caught by internal monitoring before a customer reported the issue.

Timeline (UTC)

• Issue began: 24 A...

Post-Incident Summary

Date: 12 April 2026

Impact: Degraded sync execution and delayed actions and webhook processing

Status: Resolved

Summary

A webhook flood originating from a single customer environment caused one of our databases to saturate, resulting in broad degradation of asynchronous job processing. Sync execution dropped to near zero, and a large portion of actions and webhook-driven work were delayed or unable to run. A secondary bug in the scheduling system amplified t...

Functions recovered. Post mortem:

Post-Incident Summary

Date: 6 March 2026

Impact: Degraded function execution for actions and webhooks

Status: Resolved

Summary

A bug in the task scheduling system disabled per-environment concurrency limits, allowing a single tenant to generate an unbounded burst of invocations. At the same time, the tenant’s functions were significantly longer running than typical workloads (120–150 seconds), which caused execution environments...