19 May
Innumeracy or Sensationalism?
Data butchery on the web generally takes one of two forms: lousy research, or lousy reporting of research. Today’s Mashable earns the latter prize with this header:
Pew Internet Finds Web Has Little Effect On Purchasing. Bold headline, but if you actually take the time to visit Pew’s site to read the report, the first sentence on the page is this:
The internet plays an important role in how people conduct research for purchases, but it is just one among a variety of sources people use and usually not the key factor in final purchasing decisions.
The Pew study dealt solely with the impact of the Internet upon purchasing music, cell phones and real estate, which hardly lends credence to Mashable’s first sentence that “a suprisingly small percentage of people research purchases online.” This article is riddled with errors—any one of which may seem minor, but in total add up to a savage piece of data butchery:
- Mashable: 83% of respondents said they still discovered new music by way of radio broadcasts.
The Actual Study: 83% find out about music from the radio, TV, or in a movie (BIG difference, that.) - Mashable: The study also goes on to say that of the people who purchased music in the past year, only 7% found online factored into their purchases, while 10% of cell phone buyers and 11% home buyers/renters shared the sentiment.
The Actual Study: These percentages are of consumers who said online information had a major impact upon their decision. - Mashable: In a similar vein, Steven Musil of CNET points out a study conducted by Parks Associates that found that 30% of people have never written or sent an email.
The Actual Study: Both the CNET article and the graph Mashable re-posts quote the same correct number from the Parks Associates study of 21%. The 30% figure is persons who have “never used a computer to create documents.”
As a researcher, though, I am most offended by this offhand remark:
Of course, seeing as the results are published on information pooled between August 3 and September 5, 2007 among a sample of 2400 adults age 18 or older, that specific data set may not be particularly accurate. But if we’re ballparking, Web streams and digital downloads from iTunes, Amazon MP3, eMusic, and others still clearly have much room to grow.
I encourage you to download the report, and note that on page 38 the Pew researchers detail the complete methodology and sampling regime for the study. This disclosure fully meets all eight standards for minimal disclosure from the American Association for Public Opinion Research and tells me that this was a quality survey conducted with pride by professionals. I am not sure what aspect of this sampling methodology the Mashable team felt were ‘inaccurate’ or ‘ballparking,’ but what I see makes me 95% confident that these responses are within a margin of error of plus or minus 2.3 to 2.7 points, depending on the question, and that’s a ballpark I feel comfortable playing in.
