Making Sense of All That Data

In my last blog I mentioned that Big Data has progressed far beyond just being a business buzzword. There are entire industries being shaken to their core because a leading player finds a way to analyse their customers better than their rivals. Far from being a management trend, this is a strategy that will fundamentally change many industries.

But have you ever stopped to appreciate just how much data we are creating today?

Some excellent analysis in Business Insider recently explored this question. The problem is that people and companies are just creating so much data – it is increasing at an exponential rate. At the present rate we are doubling the amount of digital data that exists every two years.

But even though we are creating and storing these enormous amounts of data, only 0.5% of it is being analysed. There is so much data out there that companies, governments, and individuals feel swamped, unable to gain insights from it.

The Business Insider article features a comment that cuts to the heart of the Big Data issue: “You have to start with a question and not with the data,” says Andreas Weigend, former Chief Scientist of Amazon, now director of the Social Data Lab and lecturer at UC Berkeley.
Businesses need to start thinking about the insights they could get from their customers, to ask more ‘what-if’ questions. There are solutions out there in the data, but it is impossible to analyse every byte of data.

A typical analogy for the average person might be the difference between email and Twitter. You check every email, even if it is only long enough to decide that it should be deleted. However, you only check Twitter messages that are arriving as you are looking at the news stream, or you use intelligent filters and analysis to ensure that interesting messages are made visible.

Businesses need to start thinking of their Big Data strategy in the same way. How can insights be drawn out from the data they already have?