On May 5, at approximately 2020 PST, Braze, a leader in mobile attribution, ‘went down’ across all of its customers. When a major vendor ‘goes down’, its network calls originating from the mobile app result in an error. Typically, the vendor will notify customers immediately, along with an estimated time to resolve the issue. But even worse, in this instance, when we looked at Braze’s site, no downtime was self-reported.
Although total outages are infrequent, given that almost every app depends on an attribution vendor, we compared five of the largest for error rates to help all app developers better understand on which attribution vendor to rely. The five vendors are Braze (formerly known as AppBoy), AppsFlyer, Branch (aka Tune), Adjust, and Kochava.
The effects of a mobile attribution vendor ‘going down’
The effect of the network calls misfiring from the mobile app are two-fold: one affects the mobile app’s core analytics for understanding their users and paying their install sources – and the other has the potential to freeze the app.
Firstly – amiss attribution. When a user installs an app, the attribution vendor calls itself and attempts to understand the originating source of the install. When this call misfires, the user install may not record. The source of attribution – for instance Facebook, Twitter, Google and AppLovin – will not be attributed to the install and it may not get paid. In fact, the install may be incorrectly attributed to a different vendor or an organic source, like the Apple or Google app stores. Analytics for understanding effective CPIs, lifetime values, and retention, will be incorrect.
With regard to app freezes, in some cases the developer may make the app reliant on the call. For example, if the user clicked and installed from a Nike Lebron’s shoe display ad, then an in-app promotion that should have been displayed to the user will not be shown. Even worse, the app might freeze because the code assumes that the attribution vendor never goes down.
Attribution vendor comparison
We compared the most implemented attribution vendors by their network call error rates. Error rates are a good proxy for the stability of their platform, especially in terms of how their integration affects the stability of the app and the quality of the app’s core tracking metrics.
The primary network errors tracked are 4xx and 5xx. These errors are returned by a network call that does not complete correctly. At the highest level, 4xxs generally represent user-side access issues – the most common being the 404 which indicates that the endpoint most likely does not exist. 5xxs represent more serious client-side server issues and are generally pretty rare. If 5xxs show up, it generally means there was a server outage.
While each vendor had relatively low error rates in absolute terms, small percentages can have a large effect on the app, payments send to vendors, and lifetime values of cohorts by channel. Consider a crash – the impact is high even when the crash-free session percentage is 99.9%.
Taking a deeper dive into the relative error rates when compared to the total call percentage are as follows:
The worst performing domain by error type is Braze, which over-indexes error percentage to total call percentage by roughly 22 percentage points. Appsflyer was separated from the sample given that 99.9% of its total errors returned an intentional 4xx rather than an inadvertent 5xx. Always sending a 4xx is considered poor practice; even so, some third-party SDKs use 4xx intentionally to keep their own server costs lower.
The next step would be to understand if these errors are actually due to server-side issues or if they were a result of an unintentional device error, which can mostly be explained by the ratio of 5xx to 4xx.
If we adjust our findings to the percentage of 5xx to total call percentage, the adjusted results are as follows:
By the adjusted results, it seems that Braze had no change given that 99.9% of its errors return a 5xx, while Branch and Appboy actually improved given that a material amount of its errors resulted in a 4xx instead of a 5xx.
Perhaps no vendor is more important to understand in terms of stability than attribution vendors. When making a key decision that impacts understanding of your users, key marketing purchase decisions, and downstream analytics, percentages matter. Without proper attribution, teams will operate on incomplete or erroneous data. Think about a lifetime value that is off by pennies; these pennies are shaved off of a bid price, meaning less users are acquired.
Having an attribution vendor with minimal error rates will give teams the most accurate insights into their users and best converting channels.
Editor’s note: About the methodology for this research: Embrace collects network data for close to 100 mobile apps. The company aggregated the network calls across all apps and did counts of error rates as well as counts of successful network calls. Along with this, Embrace collects different statistics around these calls, including duration and size.
Interested in hearing industry leaders discuss subjects like this? Attend the co-located IoT Tech Expo, Blockchain Expo, AI & Big Data Expo, and Cyber Security & Cloud Expo World Series with upcoming events in Silicon Valley, London, and Amsterdam.