What is Data Warehousing Integration?

In my last article here I talked about enterprise data lakes. Data is talked about a lot these days because we are all generating so much of it. Data really is everywhere because every action taken inside your company by employees, or with your products by customers, is recorded somewhere. The data mounts up and can be analyzed, but how do we make sense of it all?

As I explained in the earlier article about data lakes, data is usually siloed. It comes from many different sources and they are not usually connected. You might have data from your ERP system, your CRM, a customer database, a product database, and so on. These are all sources of data that are used inside your business, but they are not connected to each other.

It’s a bit like when you see grain silos at a farm – those huge steel tanks containing harvested grain. Or a brewery, where each different type of beer is being developed inside a different enormous tank. They are all separate and unconnected when in the silo.

Data integration means bringing it all together.

We need to break down the barriers and allow all the different data to mix. This is when we can create real value and insight – taking data from every corner of an organization and generating insight and connections.

When I talked about the EDL, it possibly gave the impression that this type of data analysis is only for very large companies. This is not the case for data integration. Companies of all sizes will have multiple different data sources. Each source of data is usually locked down – your accounting data never connect to your CRM data. This can create miscommunication and disconnection throughout your organization.

If your asset management, accounting, and CRM are all locked away in separate silos then there is no ability to generate the insight that is possible by connecting these disparate sources of data. You can connect all these various systems, but even add more unstructured data into the mix, such as folders full of Excel sheets.

Without this process of data integration it is extremely difficult to plan for any form of business decision-making with the best possible information about what is taking place inside the organization.

Think about the manufacturing industry – cars for example. Each individual vehicle requires thousands of parts to be delivered to a production line at exactly the right time to ensure that assembly can take place correctly. Auto brands build cars, but they also offer financing and new ownership models, such as subscriptions. They need to manage the details of millions of customers globally. Once you start connecting the various sources of data – even just at a high level – it can sound overwhelming.

Banking is similarly complex. Customers have loan products, deposits, cards, and various other financial instruments. Banks need to track all customer activities, prevent fraud, and calculate what customers might want in the future so they can anticipate future needs.

It’s clear that both these industrial scenarios would be impossible to manage without the ability to share data across silos, therefore the integration process is vital – it connects the dots and turns your data into information and insight.

The COVID-19 coronavirus pandemic has offered data analysts some great opportunities to study Big Data trends across the world. Read more about Big Data In A Time Of Pandemic