The One Thing Every Product Manager Should Do With Their Data Scientist
We all love to see things moving up and to the right and to check things off our to-do lists. For the product manager, this often means looking at simple topline or “vanity” metrics instead of asking the hard questions about what specific customer behaviors drive valuable outcomes for their company. For the data scientist, this often means optimizing the accuracy of a high-profile prediction instead of working on a more actionable but less sexy problem.
How do product managers help create focus on the analytics that are really important — the ones that can answer valuable questions and spur significant improvement?
As a product manager, one of your most important jobs is to identify and prioritize what matters and then apply that focus to facilitating productive work with your interdisciplinary collaborators. With your data scientist, the one thing this means above all others is “framing” the dependent variable. This will help the product manager focus their point of view on what matters and help the data scientist do better work.
What’s a Dependent Variable?
Let’s say you run an ice cream shop and you want to predict your revenue over time. The dependent variable would be revenue and the independent variables would be factors that you think are tied to revenue like time of year, temperature, precipitation and holidays, to name a few.
You might think that these are the same as cause and effect. In fact, they’re not. When you identify a dependent variable (DV), you’re just framing it as the measurement that you want to observe and likely forecast as a function of the independent variables (the IVs). The relationship between the DV and the IVs might be causal or it might just be correlation.
Why Is Framing a Dependent Variable Important?
Doing better work on identifying the right DV and ensuring that its measurement is actionable, what we call framing, will help with analytical clarity and actionability. In the ice cream store example, it might seem like a reasonable and standard thing to predict sales. The data scientist you’re working with has probably done this kind of time series forecasting many times and will happily dive into the analysis. You might see some great, colorful charts and graphs showing histories and forecasts of revenue per day over many, many months.
But what if the big driver of profitability for your shop is labor and what if the real point of doing the forecast is to decide how many employees to staff each hour? A great question to ask about actionability is “What’s the relevant intervention here? What are we going to do differently based on observing this DV?” Suppose the intervention is deciding how many employees to schedule for each hour next week. From there, another obvious question is “Who intervenes?” If, in the ice cream store example, it’s a shift manager, then they might not find the charts and graphs your data scientist produces particularly actionable. They might prefer the shorter-term forecast at higher granularity. Or they’d likely just prefer to see a number of employees scheduled to start each hour over the coming week — something that looks like the spreadsheet on their shop computer or even the schedule pinned up in the break room of the ice cream shop.
Beyond the ice cream shop, framing the DV (e.g., sales by day over months or employees by hour over the coming week) can become more nuanced and all the more critical. For example, let’s say you work at a retail bank and your charter is to transition customer interactions to self-service, digital experiences. In a situation like this, we often see teams look at aggregate metrics like monthly active users, daily active users and time on the site. However, if, as a customer, I’m trying to transfer money between accounts, none of those will offer you a useful view of how well things went for me and how satisfied I am. If anything, even though it may be counterintuitive, less time on site may actually be better. The customer wants a less time-consuming experience. How might we define the DV from their perspective? Instead, a DV that deals, for example, with how easily the user was able to do what they intended is probably a clearer, more actionable indicator for the relevant team.
Why doesn’t everyone already do this?
It’s natural to focus on project completion and work that shows well internally. But high-performing innovation teams need to do more to define and prioritize the customer outcomes they want. By asking the hard, often inconvenient questions about how to frame a dependent variable, the product manager and data scientist can complement each other’s work and improve outcomes for their company.
