TRILLIUM SOFTWARE INSIGHTS

Do Data Profiling Right: Shine Light on Data Quality in the Shadows

By Michele Goetz, Vice President of Product Marketing, Harte-Hanks Trillium Software

Data profiling is one of those activities that always seem to be leveraged for a data integration, system consolidation and MDM project.  The effort is conducted on disparate databases and data marts, and insight is supposed to be gleaned from the exercise.  Yet, what is really learned?

Did you really think about and map a framework for what you wanted to understand?  Do you find that you still have conflicts and implementation issues because there are things you didn’t discover during data profiling that now raise their ugly heads?

I find more and more that the biggest issue in data profiling exercises is that investigation and discovery fail to account for the three states of data:

• Creation
• Change
• Consumption

If you only point data profiling at a single source, you don’t have a full picture of how data flows and is utilized by an organization.  Data quality issues arise in the shadows and are difficult to address because there is no context of data’s value and use within the business.  So, you need to understand and compare the state of data from point-of-capture databases, within supporting staging areas, and within a consolidated or federated data warehouse.

A simple way to test this is to see what happens when a call from a business unit comes in to address a data issue.  Typically the triage goes something like this:

Step 1:  You look at the source system to validate the data quality issue
Step 2:  You look at the how data comes into the system
Step 3:  You look at the business process that generates or changes the data as it comes into the system.

Quite the archeological dig! 
Next time you undertake projects in data management or business process optimization, DO: 
-  Develop a data profiling practice that investigates data conditions across various data states,
-  Start from top-down of a business process rather than bottom-up from the data warehouse to assure data context,
-  Maintain data profiling rules utilized in investigation, add in data quality business rules, and monitor the three states of data to stay ahead of issues.

 

What do you think?  Good process?  Did I miss anything?

 

How is an Eraser like Data Quality? – It’s All About People

By Len Dubois, Sr. Vice President, Marketing & Sales Support, Harte Hanks Trillium Software

I realized today that shortly after the creation of the pencil, the data quality market developed.  And, at the time, leading the charge in the upper right “Leaders” quadrant of data quality solutions was the rubber eraser (remember those?).  Later, the bottle of Wite-Out® correction fluid swiped some market share, but only because folks stopped using a pencil and started using a typewriter. 

But in truth, the eraser stands as a pinnacle in data quality solutions.  The eraser allows you to instantly correct, at an economical cost, information or data that is deemed inappropriate for use by its intended audience. 

Now, in the larger scheme of information quality, you can appreciate that I am writing this tongue-  in-cheek.  But the eraser has one thing in common with every new, innovative data quality technology solution that has followed it, and that is, it allows for a human being to make a decision about the data based on the context and use of the intended data or information.

Data Quality – its meaning in business terms and its place in the market – have undergone enormous changes in the last few years.  But as with the eraser, one thing remains constant.  Effective data quality management is 51% human interpretation of the meaning of the data or information within a business context, and 49% an organization's ability to automate that interpretation into a business process.  So, no matter how much technology we throw at the problem, it is the person whose subject matter expertise on the business use of the data that makes the difference when it comes to understanding and deciding to take corrective action on data quality issues. 

Of course, I’d never advocate for an eraser with every notebook computer or iPad, but I am saying that when you take the person out of the equation, some data quality and remediation solutions just don't stack up.

Do you agree with my thoughts?  How do humans impact your data quality efforts?  Does anyone even use an eraser anymore?

It’s not about seeing the solution, It’s seeing the problem.

By Nigel Turner, Information Management Consultant, NHT Data Consultancy Ltd.

In this second blog I’ll talk about why BT decided to embark on an enterprise wide programme of activities to address data quality issues, an initiative that ultimately delivered more than $1 billion of benefits to the company.  I’ll do this by focusing on two main questions.  How did BT know it needed to act?  How did it get started?

The first step in addressing a problem is always to recognise it.  Initially we found ourselves in a position where, as the English writer GK Chesterton once wrote, “It isn't that they can't see the solution. It's that they can't see the problem.” 

At the outset we didn’t know we had a problem with data quality.  We knew we had business and IT issues around marketing & sales, operational efficiency, product launches, etc.  But we had not made the connection that these disciplines had a common thread – data. This wasn’t because we were sleepwalking, but in those very early days in the late 1990s, few people anywhere had viewed data quality as a focus in its own right. 

But gradually the mist started to clear, mainly by chance and not commission.

Despite massive investment in new IT, business managers complained that they still couldn’t do many of the things they hoped new IT projects would deliver.  These included getting a single customer view, faster and less manually-intensive processes, quicker launching of new products, cheaper IT support costs and so on.  At the time, one cynical senior executive was cited as saying, “All I’ve got from this new IT is the same wrong information but twice as fast as before.”  A less than ringing endorsement.  The CIO knew she needed to act fast.

So she ordered an investigation of the problems. Some of the conclusions were predictable, for instance closer collaboration between the business and IT was needed, particularly more disciplined requirements management.   But to many people’s surprise, a common theme kept emerging – data quality.  A common gripe was that the “T” in IT was working OK; it was the “I” that was the primary bugbear. 

Because of known and suspected shortcomings in data, processes kept falling over, and achieving a consolidated data view of customers and products was a major challenge as data was duplicated (but almost never exactly) across many of BT’s hundreds of core IT systems.  Moreover no-one in the company had any specific responsibility to manage data end-to-end. It soon became obvious that a specific focus on data quality was needed.

So data quality was tackled as a problem across the company for the first time. The CIO put together a small cross-functional team, containing people from both corporate business and IT.  Their brief was to do something to investigate and alleviate the problems.  BT’s journey had started.

So what’s the point of this story for others embarking on an enterprise data quality and data governance journey?  First, in an organisation which is just understanding the importance of data quality and its management, a key role of data quality and data governance specialists is to redefine the problems in data centric terms and to explain these to others in the organisation.  They need to inculcate a new way of thinking.  Data professionals therefore need to be effective agents of change.

Second, IT usually gets the blame for the problems, so it is critical to build business awareness of their responsibilities and commitment right from the outset.  BT’s CIO made it clear to her Board colleagues from day one that she could not address the problems alone but we required a concerted business and IT partnership.

Finally, prepare for a long haul.  In large organisations the scope, scale and complexity of problems caused by failings in data and its management take many years to develop.  They can therefore take years to resolve and can’t be fixed overnight.  Manage expectations carefully.   To paraphrase a much wiser man than I, the Dalai Lama, do not let expectations hinder your path.   

In my next blog in the series, I’ll relate how the small team established in BT started to tackle these challenges.

 

Data Quality at the Speed of Business

By John Nicoli, Managing Director, Harte Hanks Trillium Software

I meet with a select group of clients once a quarter as part of our Customer Advisory Board to learn about ideas, best practices, and interesting anecdotes that help expand our knowledge.  These are always active, informative and sometimes surprising discussions centered on a key topic. 

About a year ago, these folks identified a shift in philosophy in the level of responsibility required for data quality applications – back towards the business owners instead of IT – in order to speed up the necessary improvements required for their business performance.

Since then, we continue to hear with greater confidence from our customers that data quality and business intelligence – which were traditionally only “IT responsibilities” – are now being shifted, at least in part, to business owners.  These people either work in horizontal departments such as marketing or risk and compliance, or in specific lines of business driven by product managers who originally compiled or created the data in the first place.  Industry analysts are confirming that they are hearing similar trends. 

That just makes sense since business departments intimately understand the impact data quality has on their companies’ business performance, especially when the data is enriched and made more valuable.  IT is terrific at creating the infrastructure and automation that the business depends on to survive.  But, IT often lacks visibility into understanding the business value of the sea of data that they are charged with supporting.  This data quality accountability balance is a great win for both team, but requires collaboration between these groups at a much higher, deeper and more frequent rate than ever in the past.

But, an even more significant issue is that business is moving faster than ever with more pressure on cost containment and developing competitive advantage.  Business leaders are being forced to more quickly adapt to “speed of light” changes in the market in order to compete.  Even with rapid development methods, IT has traditionally not been set up to react to the speed that is required in today’s business climate.

The constant pressure of speed as well as vigilance for cost containment at all levels is a common theme that is top of mind for business leaders worldwide.  It is no secret that project budgets have been put on hold on a number of fronts on the IT side because of a number of reasons and complexities. 

Yet, data is often one of the most valuable assets in the business and is highly visible throughout numerous business processes.  It provides a path to growth, and at the same time, can be a hindrance to execution. Businesses are being choosy about which projects to take on, because of the need for rapid results, but data quality endeavors are one relatively fast, high-value way to drive positive business process performance results both A lot can be accomplished with a small dedicated team at a relatively low cost per value ratio.  

Business speed and adaptation as well as new regulatory compliance requirements are like a vice grip on our clients – squeezing resources and profits from business and IT teams.  But it’s becoming even clearer that by understanding the impact of data on the business, and collaborating closely with IT across every department, companies will very quickly realize huge cost and efficiency benefits that will impact every aspect of your organization.

 

When is the Right Time to go Enterprise Wide?

  By Len Dubois, Sr. Vice President, Marketing & Sales Support, Harte Hanks Trillium Software

Most organizations understand the value of having high quality information feeding their operational and analytical solutions.   But, data quality can be an often complex topic, and companies are barraged with detail about data quality, and there’s tons of stuff about the topic on the Web.

Type the words “data quality” into a search engine and you will find roughly 310,000,000 results.  Type in “data quality vendor” and the result is a mere 13,000,000.  So it’s safe to say that the practice and supply of data quality methodologies, software, and advice is well established. 

But, as I talk to companies and customers, one concern is not addressed very often, “When is a point solution for data quality not enough and when should we attempt to centralize our data quality practice?"  For example, one of our customers mentioned that they have point data quality solutions for marketing (CRM), sales (both sales automation solution and point-of-sale), Business intelligence for business planning and decision making, ticketing, customer call center, and their data warehouse. 

They knew the challenges that these disparate point solutions raised.  The business had to deal with inconsistent definition of important terms, ownership of relationships and data assets, definitions of what information could and or should be shared, who had permission to change data, and what metrics should be utilized to evaluate data quality.  Yet, in the face of these issues, they initially didn’t realize that a broader, enterprise solution would help them meet their business goals. 

Ultimately what they needed was an enterprise-wide strategy, organizational structure and roadmap – created in simple business speak.  This approach allowed them to create a phased approach to guarantee high levels of data quality for all internal stakeholders, and realize the benefits of data quality on an enterprise level.  They worked to create a “data quality firewall” for their organization, that ensured only clean data resides in its data warehouses, and reaches their business processes and systems around the world.

If this sounds like something your company could benefit from, perhaps this case study from one of the world's largest airlines will help you decide when an enterprise approach to data quality makes business sense for you.