Master Data makes your Big Data meaningful
Periodically, I meet with our keys customers to discuss the state of their MDM program and learn about how we can serve them better. Recently, I sat down with the CIO of McDermott International, Maurice Tayeh. Naturally, the topic of big data and analytics came up. Maurice pointed out something interesting. (Incidentally, Mr. Tayeh will be speaking on MDM, Big Data and Business Intelligence at the upcoming Gartner MDM Summit in Las Vegas April 2-4, 2014). Mr. Tayeh asserts that big data projects are unworkable without having master data management in place first.
Given the kinds of questions I hear around big data projects, Mr. Tayeh's assertion seems out of place. Most of the big data discussions I hear are focused on how to manage and search across very large data sets. Where do I physically put all my data? What platform (Cloudera? HortonWorks? etc?) should I use for my hadoop cluster? How do we tie shopping behavior to social feeds from Twitter/Facebook/LinkedIn/Google+?
But if your goals include gaining insight from all this data there's an additional challenge that has nothing to do with the volume of your data. You need to be concerned about the underlying sets of descriptive data the dimensions, attributes, hierarchies (or, master data) that makes big data meaningful.
The issue is one of consistency, without consistent dimensions, attributes and hierarchies (or, master data) you'll run into the apples to oranges comparison problem ramped up to petabyte-scale. Here's a couple questions you can use to figure out if you may have a master data management problem in your big data implementation:
Are your identifiers consistent across your source applications?
Are your dimensions consistent across your groups/departments/regions?
Are your dimensions consistent across time?
For example, imagine you're a multinational quick service restaurant (QSR) and franchisor and one objective of your big data program is to analyze point-of-sales transaction data to assess the performance of a new menu items and promotions.
Naturally, these questions are not exhaustive (what other questions would you ask?) Our key takeaway is that without managing your small data analyses on your big data might not make sense. That's how master data makes you big data meaningful.