TRILLIUM SOFTWARE INSIGHTS

Data Quality Becomes More Crucial for Business Intelligence

 By Michele Goetz, Vice President of Product Marketing, Harte-Hanks Trillium Software

Boris Evelson of Forrester Research provided his Top 10 Business Intelligence Predictions for 2012. 

There are two predictions that really stood out as key data quality pitfall areas:

“…No management or control is not acceptable, but too much control does not work. Finding the right win-win combination that combines the flexibility and agility that self-service brings with behind-the-scenes monitoring and adjusting will become the name of the game.

“Exploration (without preconceived notions, prebuilt fixed data models, or up-front specific questions in mind) will be the new bread and butter of BI suites in addition to reporting, querying, OLAP, and dashboards/data visualization.”

In each of these instances, analysts and consumers of data may not have the quality controls in place to gather and get the insight necessary to make informed decisions on what the data tells them. Data quality programs still have not reached an enterprise-wide state to bridge gaps across silos of data. Even when paired with master data management initiatives, quality is sporadic to highly-specific business requirements and business units. Open up the doors to data exploration across multiple and possible un-regulated data stores and you can see where consumers will come up against data reconciliation issues and poor data quality.

However, all is not lost. Data quality solution capabilities have come a long way from back office middleware for data warehouse integration. Trillium Software has specifically focused on quality controls at the point of exploration and consumption in environments that analysts and business consumers live in. Whether controls are introduced through a SOA deployment, or assessment and analysis are instituted within business intelligence activities, poor quality data introduced into self-service intelligence and data exploration don’t have to impede insight.

 

When Risk Becomes Reality – Even Weather Impacts Data Quality

By David Binkley, Senior Technical Account Manager, Harte-Hanks Trillium Software

As many of you may know, the weather here in the United States has been rather unique this summer.  It’s been extremely hot and dry in the South and Southwest and extremely rainy here in the Northeast.   Take those conditions into effect; add a random earthquake and a hurricane or two; and you’ve got a very challenging opportunity for many of our clients.

Well, I just happened to be at one of those clients recently, whose shipping operations were badly impacted by all of the above external conditions.  This particular customer happens to be a manufacturer of extremely expensive products that are required to be refrigerated from the point of manufacturing to the final point of use.

Taking into account the disruptions of power in the East from hurricanes like Irene, the client needed to be able to effectively synchronize product manufacturing and shipments to their customers in regions where the power grid was not impacted, or to hold the manufacturing of products for those areas where the power was likely to be out.   Because this client has excellent data quality control over their shipping information, the IT and shipping departments were able to quickly identify potentially impacted orders and have customer service representatives contact the customer to either reschedule the deliveries or confirm that delivery was still requested.

Now you might ask, how much of an impact did data quality really have on this situation?  Well, take the following into account:

To identify the orders that were impacted, the customer first needed to know what products required refrigeration, and that required accurate coding of the components of the order.  Second the order had to be evaluated as to the location of the actual shipping address, and that shipping address had to be evaluated as to its accurate location within the affected delivery areas.  Third, the customer had to be contacted via phone, e-mail etc. to verify that the order could still be sent. 

Finally, the shipper had to coordinate its impacted delivery areas with our customer’s shipments to make sure that the shipper’s distribution centers were also not impacted by the weather.  ALL OF THIS INFORMATION relies on the quality of data!!

Fortunately, for our customer and thousands of their customers, all of the above worked out.  Literally thousands of shipments were halted, re-routed or otherwise modified to make sure product would not be impacted by these events.  Production runs were modified to make up for halted orders and business resumed as normal after a week or so. 

In fact, the only problem I heard about was that some reports weren’t out at their usual time the day after manufacturing and shipping had set an all-time record for orders fulfilled in a single day as catch-up processing was in full bloom.  Well, I guess you just can’t win them all…
Think about it.  Would your data and its quality permit you to respond as well?

Data Quality Drives Process-driven MDM

By Eric Ecker, Vice President, Central Europe, Trillium Software

A few weeks ago, we announced an alliance deal with Germany-based Software AG, a leader in business process excellence and MDM software. This alliance makes so much sense because as you know, accurate, consistent and valid master data is the crucial foundation for any efficient business process.  As such, MDM is fast becoming an important corporate strategy for our shared customers, and is the reason why we’ve partnered with other MDM providers too.

But remember, centralizing master data is only half the equation.  The real key is implementing flexible data quality services within the MDM environment to truly create a single, consistent and trusted master data record - no matter what MDM environment you’re using.

Having a single, centralized, clean record will drive business value and improve operations and transactions results.  It will provide a clearer picture of customer needs and demand, and also mitigate opportunities for failure in data warehouse and business intelligence implementations.  That is, as data gets extracted, loaded and transformed many times from various sources, it tends to change and may be outdated or obsolete as new information is added to the master record.  So, as a result, reported data may not reflect intended expectations because of data discrepancies – and of course, in this time of regulatory compliance requirements, that would be a very bad thing.

In addition, you have to realize that many companies already have MDM projects in progress, so if you want to integrate data quality, you can’t simply rip out and replace the MDM solution in favor of a data quality solution that is not flexible enough to work with what you have.  In fact, large organizations may utilize multiple MDM or CRM solutions, or other types of existing data-driven applications and repositories that operate business processes.  So, you have to then integrate data quality solutions after-the-fact. – painstaking if you don’t have DQ software flexible enough to work directly with your current data environment. 

So, if companies are serious about governing their data, they have to talk about process-driven MDM and a solid data quality management strategy as part of their overarching information management process.  This is exactly what the Software AG alliance brings to the table and we’re excited about the opportunities to help our shared customers.

Are you a Software AG customer and if so, how do you ensure quality of data?

Rethinking Data Profiling – Data Quality BI

By Michele Goetz, Vice President of Product Marketing, Harte-Hanks Trillium Software

In a recent informal poll, we asked people about how data profiling fits into their enterprise data management efforts.  Responses indicate that data profiling is still primarily conducted at the front end of projects as a scoping exercise at a very tactical level.  There appears to be some penetration of profiling to monitor conditions by data stewards.  But real data governance support is low, as less than  10% utilize profiling to enforce the relationship and activities to ensure data meets business needs.

This is telling, as it shows data profiling, and in turn the results attained, are kept at the tactical level within IT services.  It points toward a reason why there continues to be disconnects between the business and IT in regards to data quality and data management.  Without a strong connection between data conditions and business objectives, data governance charters are paper tigers and lack the strength to gain business support and adoption.

Data profiling is a key tool for IT to get business attention on initiatives that will support business objectives.  Just as line of business managers have reports and dashboards to manage the performance of their departments and communicate results and impact to leadership, IT needs this same ability when working with business stakeholders.  Creating a data quality performance dashboard for management linked to a data stewardship report framework for data operations and activities creates alignment between the business and IT and shows the influence efforts have in terms the business understands.

Related resources:
Reach for the Top:  Selling Data Quality to Executives

A Strategic Approach to Data Profiling

 

QR Codes: Next Generation Data Quality Challenges

By Nelson Ruiz, Senior Product Manager, Harte-Hanks Trillium Software

Quick Response Codes, or QR Codes, are the next generation of barcodes. Wikipedia defines them as “a specific matrix barcode that is readable by dedicated QR barcode readers and camera telephones.  The code consists of black modules arranged in a square pattern on a white background.”  They are popular in Japan, South Korea and the Netherlands.

So, these barcodes contain data, and since I have been working in the data quality space for years, I, of course asked myself, “I wonder when we will read about the first big manufacturing mistake due to a QR code error?”

And that is when I learned about QC Codes Error Correction (http://www.denso-wave.com/qrcode/qrgene2-e.html). QR Code Error Correction is like a “backup system” within the QR Code that will allow the retrieval of the correct data from the code in case the QR Code is damaged in some way.

Now the “error code” methodology makes use of data redundancy and it takes double the space to store a retrievable word. Which means not all data on the QR Code can be “Error Corrected.”  In such cases, the business driver or analyst needs to actually decide which information on this QR Code is of the highest importance in order to include it in the Error Correction of the QR Code.  When deploying data quality in your enterprise, you may need to take the same approach.  And you create business rules that tell the business and the system what information, data, and processes are most critical – and what the business response should be to drive positive result. 

Business rules are created to automate the method of maintaining data quality. In recent years, people have been crazed about business rules for measuring data quality. In order to ensure enterprise data quality, companies are proactively building data quality business rules as a best practice for ensuring higher quality data, with the objective of using this EDQ solution to monitor and manage data while approaching higher data quality standards.

This approach will not only create better data quality, but ensure higher levels of ROI of your EDQ implementation and Master Data Management practice. This approach will allow you to centralize your EDQ efforts while providing cross-platform and cross-business-application support on a single DQ solution.

For example, when we have a centralized DQ rule for what is a valid product-code, we can ensure that the business rules that define an accurate product-code are consistently correct across the enterprise. If the creation and maintenance of business rules can be a challenge for a large enterprise, just think of how much it will slow down your business if you had multiple copies of these rules. For the same reason that companies deploy Single Customer View Solutions, you want to use an EDQ solution that allows you to deploy the same business rules across multiple applications and environments.

When you create your next RFP regarding EDQ solutions, ensure good data quality “Error Correction” capabilities. Ask yourself, Can it be deployed across applications and platforms supporting the same business rules? Does it support business-rules for business drivers and business rules for data quality? 

Your data stewards and data users will be eternally grateful to you for doing so.

Care to comment on your Error Correction processes for data quality?  Or, send me a Tweet at @_nelson_ruiz

 

Do Data Profiling Right: Shine Light on Data Quality in the Shadows

By Michele Goetz, Vice President of Product Marketing, Harte-Hanks Trillium Software

Data profiling is one of those activities that always seem to be leveraged for a data integration, system consolidation and MDM project.  The effort is conducted on disparate databases and data marts, and insight is supposed to be gleaned from the exercise.  Yet, what is really learned?

Did you really think about and map a framework for what you wanted to understand?  Do you find that you still have conflicts and implementation issues because there are things you didn’t discover during data profiling that now raise their ugly heads?

I find more and more that the biggest issue in data profiling exercises is that investigation and discovery fail to account for the three states of data:

• Creation
• Change
• Consumption

If you only point data profiling at a single source, you don’t have a full picture of how data flows and is utilized by an organization.  Data quality issues arise in the shadows and are difficult to address because there is no context of data’s value and use within the business.  So, you need to understand and compare the state of data from point-of-capture databases, within supporting staging areas, and within a consolidated or federated data warehouse.

A simple way to test this is to see what happens when a call from a business unit comes in to address a data issue.  Typically the triage goes something like this:

Step 1:  You look at the source system to validate the data quality issue
Step 2:  You look at the how data comes into the system
Step 3:  You look at the business process that generates or changes the data as it comes into the system.

Quite the archeological dig! 
Next time you undertake projects in data management or business process optimization, DO: 
-  Develop a data profiling practice that investigates data conditions across various data states,
-  Start from top-down of a business process rather than bottom-up from the data warehouse to assure data context,
-  Maintain data profiling rules utilized in investigation, add in data quality business rules, and monitor the three states of data to stay ahead of issues.

 

What do you think?  Good process?  Did I miss anything?

 

How is an Eraser like Data Quality? – It’s All About People

By Len Dubois, Sr. Vice President, Marketing & Sales Support, Harte Hanks Trillium Software

I realized today that shortly after the creation of the pencil, the data quality market developed.  And, at the time, leading the charge in the upper right “Leaders” quadrant of data quality solutions was the rubber eraser (remember those?).  Later, the bottle of Wite-Out® correction fluid swiped some market share, but only because folks stopped using a pencil and started using a typewriter. 

But in truth, the eraser stands as a pinnacle in data quality solutions.  The eraser allows you to instantly correct, at an economical cost, information or data that is deemed inappropriate for use by its intended audience. 

Now, in the larger scheme of information quality, you can appreciate that I am writing this tongue-  in-cheek.  But the eraser has one thing in common with every new, innovative data quality technology solution that has followed it, and that is, it allows for a human being to make a decision about the data based on the context and use of the intended data or information.

Data Quality – its meaning in business terms and its place in the market – have undergone enormous changes in the last few years.  But as with the eraser, one thing remains constant.  Effective data quality management is 51% human interpretation of the meaning of the data or information within a business context, and 49% an organization's ability to automate that interpretation into a business process.  So, no matter how much technology we throw at the problem, it is the person whose subject matter expertise on the business use of the data that makes the difference when it comes to understanding and deciding to take corrective action on data quality issues. 

Of course, I’d never advocate for an eraser with every notebook computer or iPad, but I am saying that when you take the person out of the equation, some data quality and remediation solutions just don't stack up.

Do you agree with my thoughts?  How do humans impact your data quality efforts?  Does anyone even use an eraser anymore?

Data Quality at the Speed of Business

By John Nicoli, Managing Director, Harte Hanks Trillium Software

I meet with a select group of clients once a quarter as part of our Customer Advisory Board to learn about ideas, best practices, and interesting anecdotes that help expand our knowledge.  These are always active, informative and sometimes surprising discussions centered on a key topic. 

About a year ago, these folks identified a shift in philosophy in the level of responsibility required for data quality applications – back towards the business owners instead of IT – in order to speed up the necessary improvements required for their business performance.

Since then, we continue to hear with greater confidence from our customers that data quality and business intelligence – which were traditionally only “IT responsibilities” – are now being shifted, at least in part, to business owners.  These people either work in horizontal departments such as marketing or risk and compliance, or in specific lines of business driven by product managers who originally compiled or created the data in the first place.  Industry analysts are confirming that they are hearing similar trends. 

That just makes sense since business departments intimately understand the impact data quality has on their companies’ business performance, especially when the data is enriched and made more valuable.  IT is terrific at creating the infrastructure and automation that the business depends on to survive.  But, IT often lacks visibility into understanding the business value of the sea of data that they are charged with supporting.  This data quality accountability balance is a great win for both team, but requires collaboration between these groups at a much higher, deeper and more frequent rate than ever in the past.

But, an even more significant issue is that business is moving faster than ever with more pressure on cost containment and developing competitive advantage.  Business leaders are being forced to more quickly adapt to “speed of light” changes in the market in order to compete.  Even with rapid development methods, IT has traditionally not been set up to react to the speed that is required in today’s business climate.

The constant pressure of speed as well as vigilance for cost containment at all levels is a common theme that is top of mind for business leaders worldwide.  It is no secret that project budgets have been put on hold on a number of fronts on the IT side because of a number of reasons and complexities. 

Yet, data is often one of the most valuable assets in the business and is highly visible throughout numerous business processes.  It provides a path to growth, and at the same time, can be a hindrance to execution. Businesses are being choosy about which projects to take on, because of the need for rapid results, but data quality endeavors are one relatively fast, high-value way to drive positive business process performance results both A lot can be accomplished with a small dedicated team at a relatively low cost per value ratio.  

Business speed and adaptation as well as new regulatory compliance requirements are like a vice grip on our clients – squeezing resources and profits from business and IT teams.  But it’s becoming even clearer that by understanding the impact of data on the business, and collaborating closely with IT across every department, companies will very quickly realize huge cost and efficiency benefits that will impact every aspect of your organization.

 

When is the Right Time to go Enterprise Wide?

  By Len Dubois, Sr. Vice President, Marketing & Sales Support, Harte Hanks Trillium Software

Most organizations understand the value of having high quality information feeding their operational and analytical solutions.   But, data quality can be an often complex topic, and companies are barraged with detail about data quality, and there’s tons of stuff about the topic on the Web.

Type the words “data quality” into a search engine and you will find roughly 310,000,000 results.  Type in “data quality vendor” and the result is a mere 13,000,000.  So it’s safe to say that the practice and supply of data quality methodologies, software, and advice is well established. 

But, as I talk to companies and customers, one concern is not addressed very often, “When is a point solution for data quality not enough and when should we attempt to centralize our data quality practice?"  For example, one of our customers mentioned that they have point data quality solutions for marketing (CRM), sales (both sales automation solution and point-of-sale), Business intelligence for business planning and decision making, ticketing, customer call center, and their data warehouse. 

They knew the challenges that these disparate point solutions raised.  The business had to deal with inconsistent definition of important terms, ownership of relationships and data assets, definitions of what information could and or should be shared, who had permission to change data, and what metrics should be utilized to evaluate data quality.  Yet, in the face of these issues, they initially didn’t realize that a broader, enterprise solution would help them meet their business goals. 

Ultimately what they needed was an enterprise-wide strategy, organizational structure and roadmap – created in simple business speak.  This approach allowed them to create a phased approach to guarantee high levels of data quality for all internal stakeholders, and realize the benefits of data quality on an enterprise level.  They worked to create a “data quality firewall” for their organization, that ensured only clean data resides in its data warehouses, and reaches their business processes and systems around the world.

If this sounds like something your company could benefit from, perhaps this case study from one of the world's largest airlines will help you decide when an enterprise approach to data quality makes business sense for you.

Rules and Standards Everywhere – But How Do You Govern?

By Michele Goetz, Vice President of Product Marketing, Harte-Hanks Trillium Software

Great discussion on DM Radio on March 31.  Aaron Zornes, coming back from an MDM/Data Governance conference in London, provided some interesting insight on where organizations think MDM solutions are falling short.  Top of the list: centralized management of rules and standards.  The request from attendees was to have a work center application for workflow and repository management.

Attendees felt that after years of outsourcing IT, there was a distressing lack of management tools as they bring things back in house.  And, in the development of MDM and enterprise data infrastructure, rules and standards were dispersed throughout a variety of technology tools and scripts. Aaron provided reasons for why these tools were dispersed, but his responses begged the question “is that really the primary problem with why MDM solutions are falling short?”

First, the outsourcing of IT wasn’t the issue so much as the fact that the business was not taking accountability and ownership of the data.  Why care about an easy-to-use console or application when all you need is processing capabilities and coding?  The challenge today is not IT so much as data stewards in the business who require easier ways to interact and manage data.

Second, the issue of rules and standards dispersed across the infrastructure is more a failure of best practices and the enablement of development silos.  The reality is, by better aligning data to the business, we are discovering this old way of doing things just isn’t going to cut it anymore.

My mantra has always been, just because you can, doesn’t mean you should. I think that is why we are in the position we’re currently in when managing our data and its quality. It is in the nature of people to solve problems the easiest way they know how, not necessarily the most effective way.  I agree, we need to centralize rules and standards and have personally felt that pain as a data steward.  But, is it that we need to add yet another tool or workbench to the stack, or is it that we need to look back into our practice and determine where it makes most sense to centrally manage? 

Data quality solutions with robust profiling capabilities, rules engines, and stewardship management tools provide that central platform for defining and enforcing data requirements. It is what they were built to do.  When thinking about aligning your data to the business process and need, consider:

•  What was the intended purpose of the technology?  Data movement, data quality, data hub, repository?

•  What is the breadth of capabilities in the technology that can simplify and reduce the overall cost and time to implement while improving the chances for success?

•  Is the technology aligned to best practices?

•  Has the technology taken into account the variety of skills and users that will need to utilize the capabilities?

•  Does the technology help the business and IT collaborate and speak the same language in the same environment?

If you truly want to centrally manage rules and standards within your technology platform, the first step is deciding where the best place is to centrally govern the data.  That will give you true data quality.