Summary
We saw elevated lag on our new Unified Analytics pipeline from 8AM to 12PM UTC on November 22nd. The lag was caused by analytics systems not registering that events had been successfully published to our analytics pipeline, so the events were repeatedly published as if they’d failed.

The issue resulted not only in delays for event publishing, but also events being published repeatedly. Downstream systems, including the databases that show data in our new Unified Analytics dashboard, recorded the same event occurring between 10 and 100 times.

For these time periods our new Unified Analytics dashboard may show inflated revenue ranging from 10 to 100 times actual revenue, and total event counts may be skewed by the same multipliers. Ad network postbacks were impacted, so customers with ad network integrations through our Universal Ads beta may see inflated conversion metrics for these dates as well.

Further investigation showed that the days of 11/14 and 11/15 had been impacted by the same error.

Most customers using legacy dashboard analytics are not affected. Unique event counts on our new Unified Analytics system are also not affected; uniques show a maximum of 1 event per user for the given time range and therefore do not show duplicate events.

We will correct commerce event counts and revenue data data for the affected times by December 1st. We can also provide event exports for customers who normally export Branch data. Please reach out to data-requests@branch.io if this is of interest.

Mitigation
To avoid this happening again:
We’re hardening our pipeline to make it less susceptible to repeated events, no matter the root cause.
Our legacy analytics pipeline will continue to run in parallel until the new pipelines are fully stable.
While we pride ourselves on our development velocity, we have a scaling and stability plan as well as a code freeze in place for the holidays to reduce the likelihood of incidents occurring during critical times.