The Most Overlooked Benefit of a Data Warehousing Project

One of the benefits of a data warehousing or data integration project is that you get to see how good your data really is. 

When you open the hood to your source systems more than likely you are going to find a few surprises – not the good kind.  Shravan Miriyala gives examples of common problems with integrating customer data from multiple sources.  Here’s a partial list from what he has found:

• Unreliable data
• Different schema constraints produce different data
• Those fields counted on to be able to link records from multiple systems match muss less than expected
• Several stages of progressively fuzzy, and less reliable, passes through the data to attempt to get matches
• Contradictory data
• Business rules are broken

While it is a horrible mess, this should be expected whenever you are trying to integrate data from multiple systems.  It’s all part of a days work whether the project is create a data warehouse, operational data store, integrated pipes for your business applications or OLAP cubes for your favorite BI tool.

If you don’t run into at least some of these challeges you need to run out and play the lottery.  Right now!

Finding out about inconsitancies and problems in the source systems is a very good thing.  It puts these issues on the radar.  It allows for creating the processes and rules for how to manage this data.  It should also shed light on the source systems, find out why this is happening and hopefully fix the problem upstream.

That’s not to say resolving the issues will be easy.  It will be tough.  Some issues may not be resolvable at all.  A common example is that data that was expected to be in a record is, in fact, not being captured.  Or being captured so infrequently that it cannot be counted on to provide much value.  

So, while it may not be easy to address, if you don’t know about a problem you can’t work to resolve it.  The sooner you know, the sooner you can act.