4 Reasons Why You’re Still Waiting for Insights from Your Data Scientist

Simple things we can do to get actionable insights faster.

Have you heard conversations similar to these?

“Do we have a Q2 customer acquisition projection? No, we don’t? We’ve been waiting on this for weeks. Isn’t that just something we should just … have?”

— A Chief Marketing Officer

“We still don’t have reliable results from our last A/B test. We need to confirm whether the events have been collected accurately.”

— A Product Manager

“We don’t pull the SaaS data into the database right now. We’d have to add the request to the data engineering team’s backlog. After that, we can start analyzing it.”

— A Business Analyst

If you’ve heard comments such as these, you more not alone. A study from Sigma Computing found that 63% of employees say they can’t get insights from their data solutions in the right timeframe.

Overview

In this article, we identify 4 reasons why insights are delayed in organizations, as well as ways we can mitigate these issues.

1. The project scope isn’t well defined
2. There are problems with the data
3. Insights aren’t delivered iteratively
4. A lack of analytical templates

We can use the metric time to insight to track the speed of insight delivery. Though teams rarely track this metric, it’s useful to think of this metric (even conceptually) as we talk through the following issues. 55% of business domain experts find that the average time to delivery is 1–4 weeks, and 71% of domain experts have unmet expectations from their analytics team.

1. The project scope isn’t well defined

It’s difficult for data experts and domain experts to understand each other. The prior has spent much of their career acquiring a set of skills the latter isn’t expected to. In fact, one-third of business domain experts are not confident in their ability to articulate data questions or needs to their data teams (Sigma Computing).

By creating a research brief, we can set expectations before any work begins and we create a lingua franca between data and domain experts.

Project briefs are one to two-page documents that provide an overview of a project. To many, this may seem like a superfluous step, especially when working on projects internally. However, I would argue that project briefs are critical for the following reasons.

• It’s a contract between technical and non-technical stakeholders that sets expectations of what will be delivered. The analyst will always have the destination in sight, even while they dig through the data forest.
• Provides a centralized, consistent reference throughout the duration of the project. Any newcomers to the project can use this document to quickly onboard. We make our project briefs living documents, providing updates in the document throughout the duration of the analysis.
• It’s a “lean” mode of communication. Research briefs contain the most important information and the most poignant insights related to a particular business topic. It’s a good way to disseminate actionable insights across the business — a single page provides everything an employee needs to know about the given project.
• It’s a persistent artifact that lives in the business’ collective knowledge. When you have a consistent way of documenting analysis, the learnings from that analysis can continue to impact the organization long after its creators leave.

You can access sample research briefs from theDataStrategist here.

A sample research brief from theDataStrategist.com.

2. Issues with data

The not-so-old adage says that “data scientists spend 80% of their time cleaning data.” Even if it’s not 80%, there are always issues with data.

Given the advent of the modern data stack, there are two technical approaches to addressing data issues:

1. Solve the issues upstream. There are many ELT (extract, load, transform) services that simplify pulling data from your product, database, or SaaS partners. Examples of these platforms include Fivetran, Stitch Data, and Airbyte. A major positive to using these platforms is that they produce consistent data outputs. If you’re pulling information from Amplitude, Stripe, Klaviyo, Google Analytics, or other common SaaS platforms, there are likely open-source code repositories that provide all the necessary transformations your data requires.
2. Identify the issues downstream. Even with high-quality data coming from the source, analysts should always sense-check their data before proceeding with the analysis. If there are fundamental issues with the data (e.g. events are getting tracked incorrectly), we may not be able to expedite the analysis. But the faster we identify bad data, the faster to can make good decisions in the organization.

Analysts can use the following checklist to quickly QA their data before using it in an analysis:

• Data availability. We have the data required to complete the analysis (ie no missing columns or datasets).
• Data completeness. There’s no missing data (ie missing rows) in the dataset. If there are, we have discussed how to address the missing values with the business stakeholders.
• Data triangulation. When our datasets are aggregated (for instance, by calculating total users and revenue), high-level metrics align with our team’s key business reports.
• Data semantics. Metrics and dimensions are defined in the data in the same way that domain experts discuss them.
• Data bias. We are aware of outliers and potential biases in our data. Here are two excellent articles on data bias (one and two).

3. Insights aren’t delivered iteratively

Some data science teams wait until the end of their analysis to present findings. This may be weeks after the start of the analysis. And any misalignments that arise between the analyst and business stakeholders during the duration of the project may impact the project’s efficacy.

A better approach is to iteratively release results. Drip-releasing insights throughout the analysis not only engage our audience but also allows us to pivot work based on the feedback we receive.

During your next analysis, take the following approach to release insights. See whether this expedites time to insight and reduces unmet expectations.

1. Identify the goals of the analysis and add them to your project brief. Treat the project brief as your first deliverable in the analysis. It sets the tone and approach for the rest of your work.
2. Create a presentation outline as soon as possible. This is similar to an outline of a story, with the headline to each slide taking us one step towards our final conclusions. Of course, we do not have any conclusions at this point, but we can populate the presentation as we receive new information and insights.
3. Continue to update the presentation. As you move further into the analysis, your story becomes increasingly clear. Providing your clients with an unpolished view of the analysis will help them understand where you’re going. And they will be able to provide feedback that will help the analyst answer the most important business questions.
4. Get early feedback on preliminary insights. As mentioned, drip-releasing insights on an ongoing basis allow domain experts to give continuous feedback. You can also caveat that any early insights may change (however they shouldn’t change so significantly that they change the interpretation of the results).
5. Give updates on a daily basis. Even if you don’t have results yet, it’s good practice to provide daily updates on the project. This can be a short Slack message or email containing Updates, Next Steps, and Blockers.
6. Finalize insights with domain experts. Before your final presentation, all insights and conclusions should be reviewed by our primary business stakeholders. Doing so will make them champions of your work as it’s distributed throughout the organization.

Taking this approach creates an ongoing feedback loop between data and business domain experts. When you have periodic touch bases, you’ll be able to course correct more quickly whenever there are misalignments between stakeholders.

4. Lacking templates

Talented analysts can generally answer any business question given the right data and enough time. Yet there are things that we can do to help analysts work faster, regardless of their talent level.

Templates are one of the easiest ways to expedite analysts’ work, especially when using a method that has been used elsewhere in the organization. As a former manager once told me: “The first time we analyze something, just do it. The second time, streamline it. The third, automate it.”

Even if we don’t go as far as automating analyses, templates go a far way to streamline analysts’ work. And analysts always have the option to augment or deviate from the templates as they see fit. And as our analytics team scales, templates are a good way for individuals to contribute to the company’s data practice.

Our company can use both strategic and technical frameworks to reduce our time to insight.

Strategic frameworks help expedite processes by creating a consistent approach to operations. Rather than reinventing an approach to doing something, the analyst can focus on optimizing the end results. Common types of strategic frameworks include:

• Metric selection frameworks. These are consistent approaches to identifying metrics, such as the Google HEART metrics framework.
• Presentation templates. Consulting teams are known for using template presentation decks. Internal teams can leverage presentation templates as well — including emails, notebooks, presentation decks, and reports.

Technical frameworks are code repositories that give analysts or data scientists 80% of the code needed to complete a given task. Since analysts don’t have to recreate that 80% of the code (or effort), they can focus on deeper, more valuable business questions from the outset. Examples of these frameworks include:

• A/B testing. Especially when a team runs several tests in a single domain (e.g. product improvements or marketing optimization), we can templatize A/B tests and common outputs related to the test (e.g. metric box plots for the test vs. control groups). In this case, we merely need to change a set of experiment parameters — such as, the success metrics, time periods, and test flags.
• Forecasts. Forecasts are generally run on a monthly basis and they use the same approach each time. This makes is an excellent candidate for templatization.

Examples of Jupyter Notebook templates.

In Conclusion

Most companies experience bottlenecks in the insights production process. By closing the gap between data and business domain experts, we can produce more valuable insights faster. We’ve discussed the following ways to close that gap:

1. Using a project scope to define the scope of the project.
2. Manage data issues upstream (using ELT tools) or downstream (with data sense checks).
3. Delivered insights iteratively, getting continuous feedback from business stakeholders.
4. Leverage frameworks that expedite often complete analytical operations.