This issue has been resolved. The real-time pipeline is caught up and data in our reporting dashboards are up-to-date.
This morning at 7:12 AM PST it was noted that our real-time pipeline was lagging behind by a few hours. It was identified that our infrastructure configuration was having issues with the internal cloud provider systems that would keep our pipeline caught up. This was exacerbated by the fact that we want it to scale even larger to compensate for the events and time it needs to catch up.
Around 1:28 PM PST, after applying some changes to our real-time pipeline and tweaking our pipeline's configuration, we found a configuration that allowed us to scale large enough to catch up efficiently.
At 3:23 PM PST, the pipeline completed catching up and is up-to-date and remaining that way.
Moving forward the configuration that let us scale to 8x our current daily throughput peaks will continue to stay in place to ensure we can scale between our current scale well into the future.
Posted Jun 30, 2023 - 15:32 PDT
Update
We are continuing to monitor the situation, however, we are rapidly moving the timeline forward on the pipeline.
We will provide an update in another three hours or when it completes, whatever is sooner.
Posted Jun 30, 2023 - 14:31 PDT
Monitoring
We have applied a fix to speed up the recovery of our real-time pipeline. We will continue to monitor its progress and provide an update when we have more information.
Posted Jun 30, 2023 - 13:28 PDT
Update
We are continuing to monitor the situation on getting our pipeline caught up, and are seeing improvements in the speed at which that can happen. However, we are still behind.
We will provide an update in an hour or as soon as we have an update on the status of our reconciliation.
Posted Jun 30, 2023 - 11:25 PDT
Identified
We have identified issues with our real-time enrichment pipeline that started causing it to fall behind. This causes our reporting statistics to report lower than it actually is.
We can note that all events were collected correctly and no data has been lost -- once the pipeline completes catching up, all reporting data will also reconcile correctly.
We have identified and addressed the issue with our pipeline and are working on getting it caught back up. We will provide an update in an hour or as soon as we have an update on the status of our reconciliation.
Posted Jun 30, 2023 - 10:10 PDT
This incident affected: Analytics (Data Pipeline).