Data Revolution Is Changing How We Capture And Analyze Information

One of the key changes in the way that most companies function today is their use of data. The Internet of Things (IoT) is allowing sensors to monitor supply chains and vast pools of customer behavior data can now be analyzed to spot trends or predict how they will behave in future.

But in the recent past, many systems have focused on the human, or manual, capture of information. A customer service agent might key in the name and address of a customer or the customer has to fill in several fields when checking their orders. Data has never been as important and yet if we are going to capture enough to make sense when analyzing it then we need a more industrial machine-focused approach.

Deloitte recently published research indicating that they believe there will be a transformation over the next 18-24 months in the way that companies capture and then manipulate data. Deloitte argues: “For their Machine Learning (ML) strategies to succeed, they [the companies] will need to fundamentally disrupt the data management value chain from end to end.”

Traditional databases have always focused on an array of records and fields, but this needs to be redefined if the data being captured can be used to help teach ML systems and subsequently be used by an Artificial Intelligence (AI) system.

Deloitte suggests three initial steps for executives thinking about how to improve their data capture and analysis:

1. Capture and Store: search all your legacy systems and databases for unstructured data that is not being analyzed at present and load it into a cloud-based database.
2. Discover and Connect: use cognitive data steward technology to create connections and highlight insights in the pool of data.
3. Amplify ML Capabilities: how can you use 5G and edge computing to decrease latency and increase the real-time capture of data across your entire system?

To my mind, this advice indicates that there is a dramatic change in the type of data being captured by modern companies. A database used to require strict formatting and rules about what could be captured – customer number, address, phone number etc. Now we are exploring the concept that anything and everything about how the customer behaves can be captured and analyzed – even down to how they browse on a website or how they physically move around inside a retail environment.

There are clearly some trends emerging:

• Less clean data: it will be more common to capture anything even if it doesn’t fit inside a formatted database titled ‘customer behaviour.’ This will be largely unstructured data that is captured automatically, rather than entered by a person.
• More formats: browsing data, images, video, sensor feedback, social media comments… data no longer needs to conform to a structured view and therefore it will come in many different formats and file types.
• Real-time data: as the Deloitte research indicated, the use of edge computing and 5G increases the ability to capture vast amounts of data in real-time. IoT sensors can use 5G to constantly send information, rather than waiting for the status of a sensor to be checked.

Cloud computing becomes essential in this environment. Not just to offer flexible storage, but also to ensure that a ML system is not inhibited by any local network issues or unreliability. It is certainly true that this reflects an enormous change in the way that data is gathered and analyzed – it’s more automated and focused on real-time analysis, not just when a user sends a query to the database.

There is a revolution taking place in the capture, use, and analysis of data. It is being fueled by both the IoT and cloud and companies that don’t review their data architecture now may well miss out if the Deloitte predictions on timelines are correct.