How to Separate Noise from Meaning in Big Data

How to Separate Noise from Meaning in Big Data When we rely on data to make decisions, how do we tell what is a meaningful signal and what is merely noise? Data is neither, in and of itself, as Stephen Few reminds us in his latest book: “Signal: Understanding What Matters in a World of Noise.”

Tweet: How to Separate Noise from Meaning in Big Data

Few has written a series of books about harnessing visualization to aid in analysis. In “Signal,” he takes a broader look at analysis, focusing on the idea of “sensemaking” – that is, deriving meaning from data that can be used to empower decision makers. This is especially relevant for data sets that are large and unfamiliar. Here’s how you can apply some of these techniques to help you understand a set of data and what it might be telling you.

The word “signal” is a metaphor for the patterns and meaning that are hiding in data. In electronics, signals must be separated from noise to be useful.  In the age of big – and ever-growing data – more data means more noise and bigger challenges in isolating the signals.

Go back to basics

Few is one of the experts I introduced earlier in this Practical Analysis series.  In “Signal,” he suggests a somewhat back-to-basics approach, emphasizing techniques that proved effective in the days of smaller data. He argues that these are just as relevant to big data due to their potential to amplify signals. The book covers many of the same techniques I discussed in my posts on tools and concepts, though with a greater emphasis on using visualization to both explore and explain.

Explore the data

One important premise of the book is that you need to do a certain amount of analysis just to understand the data to begin with. This is known as exploratory data analysis, and the author likens it to an explorer getting his bearings in a newly discovered land. “When we survey the land, we begin to understand its norms…. This sense of normality – what’s routine – can then serve as a backdrop against which signals – often departures from the norm – with stand out.” Few writes that he usually begins his statistical journeys by examining variation within important categories. For example, a list of products does not tell us much until we add numbers showing which items generate the most sales revenue.

Celebrate 3S’s

While big data is usually associated with 3Vs – volume, velocity, and variety – Few emphasizes the virtues of 3S’s – small, slow, and sure. He argues that only a small amount of data will ever function as signals and while data now comes in many varieties, only a few of these new choices are sure. Few also urges analysts to work slowly and deliberately. He writes, “We must take our time to understand information and act upon it wisely. Speed will, in most cases, lead to mistakes.”

Statistical Process Control

Few’s book introduces readers to the useful practices of statistical process control (SPC), a bit of a departure from pure statistics. SPC helps analysts separate routine variation from exceptional variation. (These are also known as common cause and special cause variation.) Not all outliers are signals, and sometimes the “noise” in big data can manifest in variation. SPC is an easy-to-apply tool to discern signals within variation over time. Here is an example that shows the results of an initiative to reduce hospital mortality in England.


The book ends with a bit of a pep talk emphasizing the responsibility of analysts as stewards of not just the data, but the truth. Those who have the skills and knowledge to effectively organize, analyze, and interpret data have a great responsibility to seek and defend the truth to ensure the best possible use of data toward better informed decisions. It actually makes you want to be an analyst!

“Signal” covers a lot of possibilities for exploring data sets to discover interesting patterns. And the practical examples help shed light on how you would do these things with your own data. Overall, “Signal” helps give readers a comprehensive understanding of data and practical ways to apply it to understand real-world problems.

Note: This blog is part of my Practical Analysis series, in which I explore three topics integral to understanding information: analysis, interpretation, and communication. You can find my initial post here. Next, we will investigate some of the most compelling ways to present and share your analysis. Stay tuned.

How Program Management Can Help Capture More Revenue


What is one of the biggest challenges in the beverage alcohol industry? Program management – especially without the use of program tracking capabilities. Before technology, beverage alcohol organizations used spreadsheets as their tool-of-choice to manage programs. However, as programs increased in number and complexity, organizations spent more resources on program tracking and devoted less time to generating revenue. However, as technology has developed, program tracking can be done much easier and more efficiently than before.

Tweet: How Program Management Can Help Capture More Revenue

Now, it’s OND (October-November-December), a busy time of year for those in the beverage alcohol business. How can beverage alcohol organizations boost productivity, efficiency, and revenue? Here are 5 ways program management can help.


In Baseball and in Business: “Adapt or Die”

November 1, 2017, may someday be remembered as the date that analytics irrevocably took over baseball. That’s when the Houston Astros won the World Series. Theirs is the latest championship clinched by a team relying on modern analytics in a sport filled with revered old traditions.

Tweet: In Baseball and in Business: “Adapt or Die”

Baseball has changed significantly in the past 15 years, with analytics upending 150 years of conventional wisdom. So far in this Practical Analysis series, we have focused on statistical tools and concepts.  Baseball provides a compelling example of how these can be applied, sometimes with astounding results. The rise of analytics in baseball also offers a cautionary tale about embracing – or ignoring – empirical analysis. In the words of one baseball insider, organizations must “adapt or die.” (more…)

Back to Basics: 5 Things to Think about When You Create a Dashboard

Your organization has invested a lot of time and money collecting data and wants to get some useful information out of what you’ve collected. In the early days of BI, usually only a few people in the company could make sense of the data. But today, with dashboards, more and more employees are able to access the data and use it to make better decisions.

Tweet: Back to Basics: 5 Things to Think about When You Create a Dashboard

What does this mean to you, the dashboard designer? It means you are probably creating dashboards for more than just one or two individuals. And because you have so many end users, creating dashboards that are used or not can severely impact what the company experiences in terms of ROI in the dashboards themselves. The bottom line: you want your dashboards to be understood by every end user. How can you do it? Get back to basics and follow these 5 concepts.


5 Reasons to Meet with Us at the Gartner Data & Analytics Summit

Gartner Data & Analytics Summit in Frankfurt, Germany Later this month, Dimensional Insight is attending the Gartner Data & Analytics Summit in Frankfurt, Germany. We are very much looking forward to attending the educational sessions with Gartner analysts, learning from our peers all across Europe (and the globe!), and of course, meeting with attendees.

Tweet: 5 Reasons to Meet with Us at the Gartner Data & Analytics Summit

Are you attending the summit? If so, here are 5 reasons to swing by our booth and meet with us.


3 Lessons in Implementing Healthcare Data Governance

Do you ever procrastinate or delay doing something you know is good for you? Of course, you do. We all do. It appears healthcare organizations do, too, when it comes to instituting data governance. A recent survey finds many hospitals are struggling to implement such systems.

Tweet: 3 lessons in implementing healthcare data governance

But just as a nudge from a friend can help us conquer personal procrastination, organizations that are stalled can learn from peers. Today we look at lessons from Covenant Healthcare, a healthcare system that recently set up its data governance. (more…)

Practical Analysis: Our Statistical Toolbox

practical analysis, graphs on iPadIn Part I of my Practical Analysis series on analysis and statistics, I talked about statistical concepts. Understanding the concepts is essential, but we also need to know the tools to put them to work. Here we discuss tools for applying statistical concepts and the importance of using the right tool at the right time.

There are two sides to what we do with these tools. As analysts, we can think of ourselves as members of a team competing to prove the truth. We’ll call this “the game.” The defensive side is working to defend our claims, using our tools and data. On offense, we work to refute the opposing team’s claims – at least when they differ from ours. The theories of Statistics provide the rules and boundaries of the game.

Tweet: Practical Analysis: Our Statistical Toolbox

Using Deborah Rumsey’s books “Statistics for Dummies I & II” for background, let’s look at a few particularly useful tools in an analyst’s toolbox.


3 Ways Measure Factory Will Enable Data-Driven Decision Making

This week, we announced the availability of Measure Factory, our new automation engine that is an add-on to Diver Platform. Our goal with Measure Factory is to help organizations make better data-driven decisions that ultimately benefit their finances, operations, patient care (in the case of hospitals), and more.

Tweet: 3 Ways Measure Factory Will Enable Data-Driven Decision Making

Want to learn more about Measure Factory and how it can benefit your organization? Here are 3 things Measure Factory is designed to accomplish.


How to Increase Sales and Drive Revenue for OND

increase sales ond, woman in a wine storeDid you know that 60% of wine/spirits sales take place in OND (October-November-December)? For most of us, this time of year is all about upcoming holidays – Halloween, Thanksgiving, and the holiday season. However, for those in the beverage alcohol industry, this time of year is extremely busy and important for driving revenue.

Tweet: How to Increase Sales and Drive Revenue for OND

Your organization will need all the right strategies and tools to capitalize on opportunities in Q4. Here are some tips on how to increase sales and drive revenue for OND.


Practical Analysis: How Statistics Help Us See the “Gray”

Practicals analysis-statisticsWhere do statistics fit into the world of analysis? Well, analysis allows us to make objective arguments to discover the truth based on actual observations – that would be data – as opposed to subjective claims based purely on intuition or potentially biased points of view.

Tweet: Practical Analysis: How Statistics Help Us See the “Gray”

But observed data still comes with an inherent degree of uncertainty. Where math tends to be “black and white” precise, statistics deals in the realm of “gray.” Statistics allow us to factor in uncertainty so we can draw meaningful conclusions with confidence. In this blog post, we will take a look at statistics’ fundamental concepts and tools. (more…)

How to Use DI-Production Extensions

di-productionDI-Production is a visual tool for developing production scripts. Production scripts help you automate processes. The scripts can automate and schedule simple to very complex processes such as extract, transform, and load (ETL) related tasks like loading data, moving files, and building data models.

Twitter: How to Use DI-Production Extensions

But what if you have a task to automate that isn’t part of the standard set of process and control functions? What if you want to ensure that everyone on your development team uses the same solution for a given challenge? That’s the perfect time to create and use DI-Production extensions for standardized, reusable solutions. (more…)

A Cool Tool for Experimenting with Dashboard and Portal Color Schemes

Year after year, our web site visitors have shown a great deal of interest in an article I wrote about a tool that can be used to explore color combinations online – Adobe Color. Since I have never gone into detail about how to use the tool, I thought people might benefit from a short tutorial. So I am updating my original blog post for this month’s dashboard design article.

Tweet: A Cool Tool for Experimenting with Dashboard and Portal Color Schemes

For DivePort users who want to create a beautiful user interface that is both highly functional and visually appealing, color choice is an important factor. If selecting colors is not your forte and you need something other than what is available on your company’s color palette, you can use Adobe Color. Let’s take a look at how to use Adobe Color Creative Suite to find colors for your dashboards. (more…)

National Health IT Week: A Salute to Healthcare IT

Salute to Healthcare ITSince this is National Health IT Week, we’d like to celebrate the role of health IT in healthcare delivery and recognize all who work in the field, whether at the institutional or vendor level. Health IT has played a major role in the transformation of healthcare delivery, from the introduction of the electronic medical record (EMR) under the Obama administration, to today, when it allows organizations to view data and generate actionable reports to improve outcomes and efficiency.


Tweet: National Health IT Week: A Salute to Healthcare IT


Practical Analysis: Making Analysis Accessible to Everyone

man juggling objects, practical analysisIn the first post of the “Practical Analysis” blog series, I suggested that numeracy will become an increasingly essential skill set in the 21st century, and that everyone, in one way or another, will become an analyst. But what does it actually mean to be analyst? There is no one answer. The possibilities range from someone who uses a working knowledge of analysis in their day-to-day work to hotshot data scientists at companies like Amazon and Google.

Tweet: Practical Analysis: Learning from the Experts


Survey Says: Hospitals Struggle with Data Governance

survey hospitals data governanceData integrity. It – or a lack thereof – can make or break a business intelligence implementation. How can analysts make better decisions when there’s no consistency between data definitions across different departments and data sources?

Tweet: Survey Says: Hospitals Struggle with Data Governance

Many hospitals are turning to data governance processes to help them manage their assets and ensure consistency between data. However, as a recent Dimensional Insight survey found, healthcare organizations are struggling to implement data governance. What are their most pressing concerns and challenges? Following are a few highlights from the survey. (more…)

How Wineries Can Invest their IT Dollars for Better Analytics and Job Creation

man working on laptop and a glass of wine on the tableWhen it comes to investing in technology in the wine industry, there’s a huge discrepancy in terms of money spent. A recent WBM Technology Survey by Wine Business Monthly magazine shows that while 90% of small wineries spend less than $50,000 per year on IT, more than half (54%) of large wineries spend more than $1 million.

Tweet: How Wineries Can Invest their IT Dollars for Better Analytics and Job Creation

This obviously makes sense – wineries do what they can with the resources they have. What was striking, though, is that the majority of medium and large wineries had the same top IT issue, which is “providing better data to our trade sales team.” Better data can indeed produce a myriad of positive results – providing insight into areas such as finance and sales, and spurring company growth. In this blog post, we’ll look at how business intelligence solutions can help wineries – no matter what their revenue or the size of their IT budget – make better use of their data and gain the insights they need.


Honoring Women in the Wine & Spirits Industry

Wine and spirits wholesalers in the U.S. account for 74,000 jobs and $6.5 billion in wages. While it is a male-dominated industry, women are increasingly making their mark.

Tweet: Honoring Women in the Wine & Spirits Industry

To promote and connect women in the industry, the Wine and Spirits Wholesalers Association (WSWA) created the Women’s Leadership Council (WLC) last year. This year, Dimensional Insight was proud to sponsor the group’s conference in Washington, DC. I attended on behalf of the company. Here are a few observations from the event.


Numeracy: Analytical Skills for the 21st Century

woman with laptop on numeracy data backgroundIf you’re involved in any way with analyzing information, you undoubtedly are acquainted with the term “big data” by now.  There is certainly a proliferation of electronic data becoming available in increasing abundance. But just because there’s more of it, is it really that much different from the data we’ve had right along? I’d argue that the basics are still very much the same but with the addition of both rich possibilities for finding meaning and daunting challenges for filtering out the noise.

Tweet: Numeracy: Analytical Skills for the 21st Century

In this blog series, which I’m calling Practical Analysis, I’ll explore the emerging role of the analyst in helping to answer some questions that I’ll discuss below and in building a body of knowledge to take advantage of the data deluge. Hopefully by the end, you’ll be completely jazzed about your future as an analyst!


Learn about Chart Portlet and the Future of Dimensional Insight Visualizations

chartlet portal on desktopDo you need to compare periodic sales data across different regions or different products to inform and support your sales team? You have the data, but you need to quickly turn it into easily consumable, visual presentations. Enter DivePort’s powerful chart portlet where you can produce and edit a multitude of visualizations without context switching as you customize your dashboard.

Tweet: Learn about Chart Portlet and the Future of Dimensional Insight Visualizations

The chart portlet is one of DivePort’s newest features. It was designed to be the main visualization tool for Spectre data in DivePort 7. However, you don’t need Spectre data to take advantage of the new chart portlet. Most of the features are compatible with models and marker files, too.


Meet Dimensional Insight’s Top Dashboard Designer, Zhenggang Hu

Dimensional Insight’s Top Dashboard Designer, Zhenggang HuMany of our customers have commented recently on how beautiful our new application interface designs look. But do they know who is responsible for these visually stunning displays? It is Zhenggang Hu, our user interface (UI) designer. Zhenggang has been the designer of many of Dimensional Insight’s software; he currently resides in Boston, Massachusetts and works out of the Dimensional Insight Burlington office.

Tweet: Meet Dimensional Insight’s Top Dashboard Designer, Zhenggang Hu

I sat down with Zhenggang recently to get an idea of how he works and thought our readers might also be interested in learning more about him. Read on to learn more about Zhenggang’s role at Dimensional Insight, his creative process, and the project he’s most proud of. (more…)

What to Consider as You’re Starting to Measure Population Health

What to Consider as You’re Starting to Measure Population Health | paper cut peopleTake any population – any group of people. Let’s say you have a group of healthcare professionals in one room. They have in common their area of work. But if you were to ask that group of people what “population” they belong to, you might get a variety of different answers. Red Sox fans, for instance – or Cubs fans. Or something else entirely, such as gender, age, residency, political leanings, etc.

Tweet: What to Consider as You’re Starting to Measure Population Health

That’s the thing with populations. There are myriad ways to define them. Sometimes they are easy to define, but most times they are not. There is a lot of overlap in populations, especially when it comes to the healthcare environment. In this blog post, we’ll talk about some of the biggest challenges when it comes to dealing with populations within the healthcare environment – what’s known as population health management. We’ll also look at some examples of how you can use population health information and apply it to solve real-life problems. (more…)

5 Ways Your BI Platform Should Optimize Your Distributed Sales Channel

Cost-conscious consumers take advantage of instant access to pricing and item availability on their smartphones, often choosing the best bang for their buck over brand loyalty. To win and retain customers, companies are launching significantly more promotions and incentive programs. So how do you measure and analyze the effectiveness of these promotions?

Tweet: 5 ways your BI platform should optimize your distributed sales channel

Distributed sales channels, such as those in the beverage alcohol industry, are more challenging than direct-to-consumer sales models because of the limited visibility suppliers have downstream. As sales models become more complex, the number of data sources that need to be integrated and analyzed exponentially increases. With the right business intelligence (BI) platform, you can overcome these hurdles. Here are five ways your BI platform should help you optimize your distributed sales channel. (more…)

How to Use Map Portlet to Visualize Geographic Data

color coded map of the worldEveryone knows the old adage, “A picture is worth 1,000 words.” How true this is, too, for data represented geographically! Maps provide an intuitive way to compare data across various regions turning tabular data into easily interpreted visual information.

Tweet: How to use map portlet to visualize geographic data

Available in Dimensional Insight’s Diver Platform 7.0.26 and later, the map portlet is redesigned with a configuration wizard, graphical user interface, and significant new functionality to make it easy to integrate geographically represented data on your dashboards. (more…)

The Downside of Data Science

The Downside of Data ScienceA few weeks ago, I attended the Real Business Intelligence Conference, which was hosted by Dresner Advisory Services on the campus of MIT. The show was packed with great presentations by speakers such as Dr. Theresa Johnson of Airbnb, Professor Thomas Malone of the MIT Sloan School of Management, and David Dadoun of the Aldo Group.

          Tweet: The downside of data science

One of the more thought-provoking presentations was given by Cathy O’Neil, author of the book, “Weapons of Math Destruction”, in which she argues that mathematical models and algorithms give the illusion of being impartial, but in actuality, many of them end up perpetuating stereotypes and inequality. Following Cathy’s presentation, I sat down to read her book. Here are some of my thoughts.


Spotlight on our Distributors: Andy Dennis of eProgram

Andy Dennis, eProgramOver the last several months, we have profiled a number of our distributor partners on the blog. Today, the spotlight is on Andy Dennis of eProgram, based in Australia.

Tweet: Spotlight on our Distributors: Andy Dennis of eProgram

eProgram has been a Dimensional Insight distributor since 2002. In this blog post, Andy tells us a little bit about his company, the types of customers his organization works with, and the trends he is observing in technology and business intelligence.


Congratulations to our Most Wired Customers

Congratulations to Dimensional Insight customers who were named the Most WiredHospitals & Health Networks (H&HN) magazine recently announced its 19th annual Most Wired survey honorees, and I’m happy to report that several of our customers made the list! Congratulations to our 20 healthcare customers that were honored.

          Tweet: Congratulations to our Most Wired customers

The annual Most Wired survey measures the level of IT adoption within hospitals and health systems. In addition to providing a list of hospitals and health systems that meet the criteria, H&HN also provides some interesting data with the survey. Here are a couple of points that stood out. (more…)

How to Best Support Your Salespeople

The role of a salesperson is difficult. Salespeople are often challenged with less than perfect working environments, yet they are expected to excel without the tools office workers have at their disposal. In the office, workers have all the resources they need readily available: presentations, files, data, and experienced staff to help with department-specific answers to questions.

Tweet: How to best support your salespeople

But what about those whose jobs require them to “live” on the road? How can they most effectively do their jobs and gain access to the same tools that in-house office workers have? That’s what we’ll be examining in this blog post. (more…)

3 Healthcare Challenges and How Analytics Can Help

Healthcare is inarguably facing some of the most daunting challenges today, with the potential repeal of the Affordable Care Act (ACA), the uncertainty of the American Health Care Act (AHCA), numerous other regulations with which providers must comply, new payment models, and much more.

Tweet: 3 healthcare challenges and how analytics can help

At our recent Dimensional Insight Users Conference (DIUC17), Julie Beard from KLAS Research spoke at a couple of our breakout sessions about some of the biggest healthcare challenges providers indicate they are facing today and where the industry is headed. (more…)

Visual Standards in Dashboard Design

People often confuse design with style. But this is misguided. They do not mean the same thing. Design principles do not change. Yet style is changing constantly.

Tweet: Visual standards in dashboard design

In this article, I am going to take a look at visual standards in dashboard design. We’ll examine how you can use design principles to determine the best dashboard – both in style and in functionality – for your users. (more…)

3 Business Intelligence Trends and How They Impact Your Organization

At our recent Dimensional Insight Users Conference (DIUC17), industry analyst Howard Dresner, who coined the term “business intelligence” back in 1989, joined our CEO Fred Powers on stage to talk about some of the trends in business intelligence today. The timing was serendipitous, as Howard had recently released the results of Dresner Advisory Services’ 2017 Wisdom of Crowds Business Intelligence Market Study.

Tweet: 3 business intelligence trends and how they impact your organization

Here are 3 trends that Fred and Howard discussed at the conference. (more…)