In my first post on exploratory data analysis (EDA), I discussed why it’s important to get a “sense” for the data you’re working with before applying analytical techniques that may come with critical assumptions and prerequisites.
In this post, I’ll expand on the concepts I discussed in that post and also examine some other exploratory techniques. The goal is to make you familiar with various forms of data analysis so you can use them to make the right decisions for your organization.
We humans like pictures. We’re wired to process images quickly and effectively.
The evolution of those capabilities over thousands of years has allowed us to survive by detecting and evading predators, finding forage, and excelling as predators ourselves. To support these activities and others, the image processing capabilities of our brains have become especially adept at distinguishing basic shapes, such as lines, circles and various polygons, as a precursor to fully comprehending what the actual imagery represents.
We are also capable of understanding complex concepts, abstractions and relationships. But not all of us are created equal in that respect – at least judging from the distribution of scores on the math component of college prep exams! But even the savants among us reach a limit as the volume of data and complexity of problems become intractable to the human mind.
And though we may differ in our analytical abilities, we do seem to share one thing: our ability to reason visually plays an important role in our problem solving. Let’s examine this in the latest “Practical Analysis” blog post. (more…)
Too much data and not enough useful information is one of the great paradoxes of the era of large scale, pervasive computing. Improving that balance is a key challenge for the modern-day analyst. That requires becoming familiar with the most appropriate tools for generating meaningful insights. And there are some fairly basic yet extremely powerful ones that you need to know about.
This next chapter of the Practical Analysis blog series, beginning with this post, is dedicated to endowing you with that knowledge.
The original idea for the “Practical Analysis” blog series came from a seemingly simple question: “What is analysis?” Answering that question took me on a fascinating journey from Florence Nightingale’s work to improve public health in the mid-19th century to the most recent developments in the fields of machine learning and artificial intelligence.
Though I’d found some interesting anecdotes and instructive examples, it felt that something was still missing. Like the answer to the original question. Maybe I needed to ask some different questions that would yield more useful answers such as: What are the essential tools that 21st century analysts should have in their toolbox? Where did they come from? What was the thinking behind them? And ultimately, why do they matter — today?
As an analyst, you can perform the most sophisticated analysis and draw the most compelling conclusions, but without a way to share these with others, your hard work stays with you. So how to best communicate when it comes to quantitative data? In a nutshell: words and pictures.
Tweet: How to harness the power of pictures
In this blog, we offer practical suggestions on how to tell compelling stories through data visualization. (more…)
When we rely on data to make decisions, how do we tell what is a meaningful signal and what is merely noise? Data is neither, in and of itself, as Stephen Few reminds us in his latest book: “Signal: Understanding What Matters in a World of Noise.”
Tweet: How to Separate Noise from Meaning in Big Data
Few has written a series of books about harnessing visualization to aid in analysis. In “Signal,” he takes a broader look at analysis, focusing on the idea of “sensemaking” – that is, deriving meaning from data that can be used to empower decision makers. This is especially relevant for data sets that are large and unfamiliar. Here’s how you can apply some of these techniques to help you understand a set of data and what it might be telling you.
November 1, 2017, may someday be remembered as the date that analytics irrevocably took over baseball. That’s when the Houston Astros won the World Series. Theirs is the latest championship clinched by a team relying on modern analytics in a sport filled with revered old traditions.
Tweet: In Baseball and in Business: “Adapt or Die”
Baseball has changed significantly in the past 15 years, with analytics upending 150 years of conventional wisdom. So far in this Practical Analysis series, we have focused on statistical tools and concepts. Baseball provides a compelling example of how these can be applied, sometimes with astounding results. The rise of analytics in baseball also offers a cautionary tale about embracing – or ignoring – empirical analysis. In the words of one baseball insider, organizations must “adapt or die.” (more…)
In Part I of my Practical Analysis series on analysis and statistics, I talked about statistical concepts. Understanding the concepts is essential, but we also need to know the tools to put them to work. Here we discuss tools for applying statistical concepts and the importance of using the right tool at the right time.
There are two sides to what we do with these tools. As analysts, we can think of ourselves as members of a team competing to prove the truth. We’ll call this “the game.” The defensive side is working to defend our claims, using our tools and data. On offense, we work to refute the opposing team’s claims – at least when they differ from ours. The theories of Statistics provide the rules and boundaries of the game.
Tweet: Practical Analysis: Our Statistical Toolbox
Using Deborah Rumsey’s books “Statistics for Dummies I & II” for background, let’s look at a few particularly useful tools in an analyst’s toolbox.
Where do statistics fit into the world of analysis? Well, analysis allows us to make objective arguments to discover the truth based on actual observations – that would be data – as opposed to subjective claims based purely on intuition or potentially biased points of view.
Tweet: Practical Analysis: How Statistics Help Us See the “Gray”
But observed data still comes with an inherent degree of uncertainty. Where math tends to be “black and white” precise, statistics deals in the realm of “gray.” Statistics allow us to factor in uncertainty so we can draw meaningful conclusions with confidence. In this blog post, we will take a look at statistics’ fundamental concepts and tools. (more…)
In the first post of the “Practical Analysis” blog series, I suggested that numeracy will become an increasingly essential skill set in the 21st century, and that everyone, in one way or another, will become an analyst. But what does it actually mean to be analyst? There is no one answer. The possibilities range from someone who uses a working knowledge of analysis in their day-to-day work to hotshot data scientists at companies like Amazon and Google.
Tweet: Practical Analysis: Learning from the Experts