The time and efforts we waste on big data

The issue with Big Data isn’t its hype – there are certainly enormous benefits in capturing the data being generated by a rapidly digitizing world. The issue has been and continues to be how we approach the problem of having too much data, accumulating too rapidly and in too many forms for organizations to handle. Big Data has been managed too much as a technical problem and addressed by attempting to build massive data warehouses and ever-larger Hadoop clusters managed by the rarest talent of all, the data scientist.

This is the modern day Tower of Babel, the biblical tale of humans attempting to reach God (perfect analytics?) by building a single, massive tower to the sky. For those who don’t know the story, God didn’t appreciate this act of ultimate hubris and caused the participants to suddenly speak many different languages and scattered them across the earth, rendering humans incapable of attempting anything like this again.

A data Tower of Babel

While the analogy isn’t perfect, there are lessons for those who need an approach that is easier and more cost effective than putting all data eggs in one Hadoop or data warehouse basket. For starters, why would we want to incur the cost of so many data movements and so much data governance if analytics can be done in place and decisions made locally where needed?

Secondly, how much collaboration around data can occur when the data is no longer near its source. Where is the room for human interaction with each other and with the data to give greater context and understanding to its meaning? Our newly digitized world isn’t only about analytics as human judgment remains a key piece of the puzzle.

Lastly, how does the business involve itself in data-driven decisions when the data is owned and managed by individuals who are too busy building the Tower and focused on algorithms to be part of the data’s use? Tower building is a full time job.

It’s about decision-making

The real goal of any organization is making excellent choices about what’s working and what’s not and, where to spend, where to cut, and how to achieve the most benefits with the fewest consequences. These are day-to-day distributed decisions that require access to distributed information and involve distributed human beings in distributed geographies. Before the digital age, we managed this distribution with organizational hierarchies and it’s too soon to throw that architecture away simply because we’ve developed the ability to move, store and crunch massive amounts of information.

This point matters because all decisions aren’t equal (ranging from trivial to critical, simple to complex/interdependent), many (but not all) can be automated, and distributed analytics have a powerful place in supporting how it all gets done. For an organization to manage such disparity in its need to decide, a decision architecture has to be part of the plan and involves a variety of approaches and tools. The organization needs first to know what decisions it makes and then decide what decision service supports each choice in the best way possible.

Gartner’s Roy Schulte, Teresa Jones and Lisa Kart describe the following decision services in their piece Find the Best Approach to Decision Management:

• Rule-Based
• Predictive-Based
• Optimization
• Complex-Event Processing

Human judgment is clearly also a service to be part factored into any decision architecture. Together, these technology approaches can be put to use across the distributed decision architecture to achieve both speed, accuracy and repeatability. Lastly, the decisions themselves need to be analyzed and improved as a constant technique.

Big Data Part II

In the post-big data era, the next level of organizational maturity involves coming up with the decision architecture that provides real business value. In this stage, the organization’s resources, whether that’s technology infrastructure or the human element, are deployed to align with distributed decision support. This is key to closing the chapter of the “lab experiment” and showing real results from the investments that have been made to date.

A decision architecture doesn’t put focus on data for data’s sake or collecting and storing the most data possible (regardless of its value), but instead on data as subservient to the needs of business processes. A decision architecture first asks the question, “What do I need to decide?” and then follows that decision down to its component information and the logistical elements (like time) that make the decision possible in an optimal way.

(Picture credit: Wikipedia – Pieter Bruegel the Elder – The Tower of Babel)