Metric sales

Better Data Preparation and the Importance of Metric Grain – The New Stack

Data readiness is an issue that plagues nearly every organization when it comes to understanding and uncovering actionable insights from business-critical data. If you or your organization have been through this, the good news is that you are not alone.

However, for organizations to use business and decision intelligence tools to comprehensively understand and extract their data, that data must first be carefully prepared.

When done correctly, data preparation can yield many business benefits, such as enabling advanced analytics and business intelligence tools to perform quick and in-depth analysis to drive more informed decision-making. smart.

Start with your metrics

Joel T. McKelvey

Joel is VP of Product and Partner Marketing at Sisu, the AI ​​and ML-based business intelligence engine that analyzes machine-scale data. A former product manager at Google and head of product marketing at Looker, he has extensive experience in data and analytics, including business intelligence, databases and data warehousing, and data deployment models. ‘to analyse.

Before beginning data preparation, the data team should consider the objectives, particularly what the business is looking to measure and the types of information that might be needed. To start, focus on defining the metrics and selecting the correct metric grain. This empowers your data team to successfully analyze complex data quickly, uncover key drivers of change, and turn that insight into actionable business decisions.

Data structure often proves to be a challenge for data teams. Using traditional business intelligence tools, they are accustomed to manually sifting through millions of rows and columns of complex data, requiring analysts to spend exorbitant amounts of time searching for information.

Thanks to advances in the modern data stack, there are now many tools available that leverage artificial intelligence, machine learning, and natural language processing against these datasets to convert raw data, unstructured and siled into structured and well-defined data. Tools such as AI/ML-based analytics can provide programmatically defined measures of data relevance, based on metrics defined by the data team.

In May, Sisu launched its first integration with dbt Labs to help our customers with metric definitions in the dbt Cloud metrics metadata layer. Data governance has traditionally been a static process, cut off from the ever-changing reality of a business, which can negatively affect the efficiency, reliability, and impact of data initiatives.

Metric configurations should be consistent and up-to-date across the organization to ensure accurate and in-depth analysis that meets business needs. However, this typically manual process takes valuable time from data teams and leaves room for human error. With Sisu’s dbt integration, customers can leverage metrics they’ve already created in dbt for more complete, efficient, and faster execution. Machine learning-based analytics directly in Sisu’s business intelligence engine.

The challenge of metric granularity

The grain metric refers to what each row of data represents in a set, so if a data team does not choose the correct input when analyzing the data, their analyzes will show inaccurate results. It’s what we like to call GIGO: trash in, trash out. Selecting the right grain for your metric can be tricky, especially when trying to ensure you’ve captured all relevant dimensions for accurate decision making.

The granularity issue can often become a roadblock for businesses, as manually determining the granularity requirements of an analysis depends on the metric you are exploring, and the dataset must also be at the right grain size for that metric.

Say you are a store owner and you are tracking your sales performance data week by week, and your smallest grain size is one week. This means that a week is as accurate as your data gets (your data is pre-aggregated to the weekly grain), making it impossible to analyze the impact of hourly and daily performance on weekly performance. Sure, you can easily find each day’s average sales from your weekly sales data, but you’ll never be able to look at individual day’s performance, which will limit what you can learn and do with your data.

For grain, finer is usually better when doing full scans. Ensuring your organization has analysis-ready datasets that retain as much grain as possible protects against unexpected or unpredictable future analysis. To learn more about best practices for selecting the best metric grain, see Sisu’s fact sheet on Choosing the Right Grain for Your KPI Metric.

Once you understand the grain of your dataset, create the right metric, and define your datasets to support the necessary metric grain, you can then focus on quickly iterating through datasets and existing metrics to determine the root cause for changing key metrics, such as weekly sales performance. This is where Sisu’s metric grain feature can come in handy.

With Sisu’s metric grain feature, data teams can discover the best key findings faster and adjust the granularity of metrics based on new questions being asked – a manual process that would take analysts weeks with traditional BI tools. . Sisu’s ML algorithms can also detect, validate, and model dataset primary keys and present them as your grain in the metric configuration, allowing analysts to avoid preparing datasets at any level. in advance. Users can also adjust the granularity to produce new aggregated datasets in the metrics setup, eliminating the hassle of manually scaling your data based on independent metrics needs.

For many of our customers, metric grain has been a key feature in combating a common problem: data distraction. For example, an online marketplace started with 5.5 billion transaction records and several thousand columns of data. By using Sisu, they were able to reduce that initial amount to the most relevant 250 million records, eliminating hundreds of columns and a significant amount of unnecessary noise.

They quickly identified the key factors affecting their most important metrics and saved considerable time and resources. This, in turn, helped their data team avoid hundreds of hours of manually searching and searching for data, and made insights affecting their metric change all the more relevant due to the speed at which they could be identified. We’ve seen clients, such as Overstock and HomeToGo, report up to 80% acceleration in data investigation and insight speed.

By using tools such as Sisu and dbt to thoughtfully prepare and understand the underlying causes affecting key business indicators, data analysts can answer important business questions from stakeholders in greater detail and, in some cases, uncover key drivers of change that might not have been visible in the business context. only. In this new era of leveraging data to make informed business decisions, using tools that make your data work for you will be what sets your business apart from the competition.

The owner of TNS, Insight Partners, is an investor in: HomeToGo.

Feature image via Pixabay.