Tuesday, August 27, 2013

Same Data, Different Conclusions

Recently, a research firm called Canalys published a press release entitled, "Half of top iPad apps either unavailable or not optimized on Android". The press release was promptly picked up by a handful of major media outlets, including AllThingsD and even The Guardian.

What exactly did Canalys find? To quote their press release directly:
New Canalys’ App Interrogator research highlights one of the deficiencies of the Android ecosystem: limited availability of high-quality, tablet-optimized apps in the Google Play store. Of the top 50 paid and free iPad apps in Apple’s US App Store, based on aggregated daily rankings in the first half of 2013, 30% were absent from Google Play. A further 18% were available, but not optimized for tablet users, offering no more than a smart phone app blown up to the size of a tablet screen. 
Just 52% of apps had Android versions both available through Google Play and optimized (if only a little) for tablet use. ‘Quite simply, building high-quality app experiences for Android tablets has not been among many developers’ top priorities to date,’ said Canalys Senior Analyst Tim Shepherd.
First, note that the press release headline is not clear. It is somewhat easy for someone headline browsing to go away thinking, "Wow, only half the top apps on Android!"

Luckily, Canalys does clarify the headline in the full press release, but the finding is still portrayed dramatically. Just barely over half of the top paid iPad apps are available in full tablet form in the Google Play store? That's quite a deficiency, and certainly noteworthy. But a deeper look into the data reveals a slightly different story. You can find the full data here.

If you simply remove all the the Apple-published apps from the study, suddenly the results change. 59% of the top (now 44 instead of 50) paid apps are available in full versions from Google Play, and 70% are available, but not necessarily "optimized". (It is also worth noting that the published data never defines exactly what "optimized" means).

If we look at Canalys' free app data, the picture becomes even rosier for Android. Again discounting Apple apps, 87% of the top (now 45 instead of 50) free apps are available in full, "optimized" versions for Android tablets.

Combining both free and paid app data now gives us 73% of apps available in "optimized" versions on both platforms, and 79% percent available in some version.

It is also worth noting that Canalys chooses to frame their conclusion as "30% were absent from Google Play," instead of stating that, "70% were present on Google Play." While this is the same conclusion quantitatively, our minds perceive the two statements differently.

Is the data half-empty or half-full?

While here I've chosen to discuss this one Canalys study, the use of "massaged" data (the word "bias" has become too loaded) is commonplace in press releases, analysis reports, and the major media. This is not a revelation, but something many data professionals, and even many well-educated members of the public, have known for a long time.

But then again, who really wants to stare at tables all day except for a few Math teachers and analysts? Communications professionals know this, and they make sure to distill data down into more digestible, dramatic nuggets, lest they lose their share of attention. Can you really blame an organization for dumbing down their quantitative conclusions into a form that the average reader can appreciate? At the end of the day, like in this Onion article about the coverage of Miley Cyrus' VMA performance, it's all about how many eyeballs you can grab.