Web analytics and statistical bias

Sample bias

The people who block ads and cookies differ from those who don’t by more than just their aversion to ads and cookies.

Around 30% of users actively block ads. Sometimes this also means blocking the JavaScript that drives analytics.

Technologically savvy, privacy-conscious users are more likely to use VPNs, ad blockers, etc. This group may skew younger and higher income. Therefore the missing 30% will not be a random sample from the overall population.

This becomes most important when these tech-informed users are the target audience, both in terms of ad targets and analytics. This could impact analytics to the point where Google Analytics is not helpful in measuring sales, even though may still provide some useful metrics in terms of inbound traffic.

On the other hand, sometimes the target audience is less likely than usual to block ads or cookies, such as large governmental, educational, or corporate organizations that enforce the use of a browser like MS Edge, which does not block third party cookies by default. Another example would be ads aimed at older or lower income users who are less likely to be blocking.

Cookie acceptance statistics

Studies on the percentage of people blocking first or third party cookies deliver wildly variable results, which suggests true figures are not well understood.

This study suggests that around 40%-60% of users accept all cookies, 20% reject all the cookies they can, and the remainder reject some cookies and not others.

In the majority of cases using standard cookie walls, first party cookies would be treated as essential and therefore no opt-out needs to be given. The exception would be sites using first party cookies for analytics, which are a small minority.

Firefox, Safari, and Brave block third party cookies by default. That’s probably somewhere around 21% of users. No major browsers block first party cookies by default because that can significantly degrade functionality.

Statistical help

Conclusions drawn from biased samples can be misleading.

Harry Truman holding up newspaper saying DEWEY DEFEATS TRUMAN

We can help you mitigate, or at least be aware of, the limitations of biased data. It may be possible to change your testing strategy to reduce bias, or to use statistical techniques to reduce the impact of bias.

Reach out today for help with biased web analytic data.

LET’S TALK

Trusted consultants to some of the world’s leading companies

Amazon, Facebook, Google, US Army Corp of Engineers, Amgen, Microsoft, Hitachi Data Systems