All affected lists have been processed and the system is performing normally.
Posted Dec 14, 2023 - 00:39 PST
Update
List processing of the remaining lists is still occurring. We will continue to monitor and provide another update within the next hour.
Posted Dec 13, 2023 - 22:30 PST
Update
We are continuing to monitor these fixes in production. We have seen the entire load on the database reduce drastically and are back to performing like normal. The queue continues to process, and some customers will see their lists processed this evening to get everything back to normal and fully completed.
We will provide another update as the final lists get processed and verify the state of things continues to look performant.
Posted Dec 13, 2023 - 17:45 PST
Monitoring
A fix has been applied and now the backlog of tasks is moving fast and should catch up within the next hour or so. We are monitoring the progress to make sure everything gets back on track, and will provide another update soon.
Posted Dec 13, 2023 - 13:54 PST
Identified
We have identified the issue isolated to the database. All data is still being successfully committed, but slow database performance is happening due to some storage cleanup.
We will provide another update as the progress of the storage cleanup is completed. We expect it to be completed within the next hour or so.
We are also applying database operation optimizations as well to prevent this type of buildup in the future. Another update will be provided soon once these changes are also applied.
Posted Dec 13, 2023 - 13:32 PST
Investigating
We are currently investigating issues with list processing on the Orbee Platform. At this time, all requests are queued and stored, but are being processed slower than usual. We will provide another update soon once we have identified details on the problem.