Big Data, Big Ruse

by Stephen Few, Visual Business Intelligence Newsletter

For many years now we’ve been told that we’re living in the Information Age. As more data and the technologies that process it surround us, we’re told that we now work smarter and more efficiently, but experience says otherwise. We blame the failure on ourselves. “Why do I feel buried and confused when I’m supposed to feel empowered? Something must be wrong with me! I’m not smart enough. I’m technologically inept.” Vendors tap into this anxiety and milk it for all its worth. Big Data is just the latest carrot that they’re tauntingly dangling to keep us chasing a fantasy. While we breathlessly struggle to gain our feet, they count their money and laugh.

Every few years the BI revenue stream begins to dry up a bit and the marketing folks come up with a new promise of information nirvana. Some of these enticements are constantly recycled, like “self-service BI,” which gets trotted out with every new software release, hoping we’ll forget that the same promise was made but never fulfilled last time. Big Data is just the new rallying cry for the same old stuff BI companies have been producing all along. Yeah, I know that they’re enlarging storage capacities and improving processing times, but that’s hardly new. Data didn’t suddenly get big, I’ve been working in information technology for 30 years and data has always been big. …………….

Like many terms that have been coined to promote new interest in business intelligence (dashboards, analytics, business performance management, advanced data visualization, etc.), Big Data thrives on remaining ill defined and feeds on ignorance. If you perform a quick Web search on the term, all of the top links other than the Wikipedia entry are to BI vendors.    ………

……….. Matthew O’Kane recently described in my blog the following advantages of Big Data:
“The main driver of benefit is when predictive analytics is improved through the use of more varied and deeper data sets. It is this area where new techniques are required because the tried and tested regressions and decision trees won’t cut it any more.”

Is it true that predictive analytics have suffered from insufficient quantities of data? Perhaps in some cases, but I don’t think this is a fundamental problem plaguing predictive analytics. I think the bigger problem is that fact that few people have been sufficiently trained in statistics to build meaningful predictive models. This problem has existed all along…………………………….

Will new sources of data and our ability to store and interact with greater volumes of data lead to new insights? Perhaps. I certainly hope so. The point that I’m making, though, is that few if any of these new insights will emerge from anything that BI vendors are marketing as Big Data technologies. They will emerge from people using their brains to think smarter about data. They will emerge from effective interaction with data, rooted in statistical skill and supported by tools—especially visual analysis tools—that are well designed to augment human perception and cognition. In other words, they will emerge when people find and learn how to use the loom that’s needed to weave data into meaningful insights.  Read the Blog

DCL: This blog is a bit of a rant. But we’ve all heard a lot about “Big Data” recently, especially at business conferences, and a lot of us may have some sympathy with the writer.