Mastering the metrics: How to become a data-driven PM

Aida Tabak, Senior Director, Analytics Product at ServiceNow, gave this presentation at the Product-Led Summit.

I've always been a bit of a data nerd. I love data. As early as my studies, I knew that I wanted to build analytics products. So that’s what I've been doing for many years now, and the company I'm currently doing it at is ServiceNow.

Why should you use data?

We’ve heard a lot about value and how to bring on different roles, and in this case, it's always about metrics. It's about how you show the value. How do you bring everyone to the same alignment? That’s really what it means to be a data-driven product manager (PM).

So why should you use data?

Prioritize

One of the key things for a PM is to use it during prioritization. Oftentimes, you have requests from sales and different stakeholders, and they're saying, “This is the most critical thing.” Or if you have a legacy product and you want to rebuild it, it's so useful if you can see the data and identify the key things that you should bring over.

It helps you to have conversations across different roles when you're identifying the prioritization of your roadmap.

Understand customers

The second part is to understand customers. With a lot of the data, you see how the customers are using your product. All of us go and talk to our customers on a regular basis, but it's really helpful if you spend some time understanding your customer based on product usage before you go to them.

Do this before you go and conduct any type of usability testing, any type of interviews, or even roadmap sessions.

Trigger enablement

The third reason is to trigger enablement. There's customer success, maybe outbound product management, marketing, etc. So it's really interesting to use the data and identify where we can help our customers by providing additional enablement material like videos, trainings, and specific sessions. And if you have partners, you can build up the ecosystem around it.

The ultimate goal is for the customer to get the value they want out of the product.

See impact

The fourth one, which is my favorite one, is that we want to see the impact that we're creating for the customers.

If you have thousands of customers, you can’t go to every single one of them and see the impact you’ve created for them. Of course, that’d be amazing. Instantly seeing the value and the impact that you're creating for customers really motivates you to go beyond.

Turning data into action: Overcoming key challenges

One of the problems that I've encountered during this journey is that our company had a lot of data.

But when we started the journey, we were trying to identify why we weren’t successful in using that data. We look at the data, we report on the data, and we send it to our management. But what do we do with that data?

I love this quote:

“Not everything that counts can be counted, and not everything that can be counted counts.” - Albert Einstein

What happens is that people say, “I need to have metrics. I need to have data.” So you have lots and lots of data, but it can actually be useless types of data.

It's also critical to understand that metrics and data aren’t enough. A lot of things can’t be counted. You have to go and talk to the customers and understand the market.

The second problem that we’ve seen a lot is that we start on this journey, we try to identify some type of template or training that’ll help the product organization to identify the metrics that are important.

But then you start hearing things like, “I’m not sure I trust the data.” “Who’s created this metric?” “What’s the data underneath?” “What does this dashboard show?” So the conversation just continues and we're not taking any action on the data.

The third problem is knowing when to use the data. Sometimes, because we have all of that data, we use the data to make a point and we start hiding behind it. So it prevents us from innovation because we're stating, “Look, the customers aren’t using this.” Maybe it's a different reason why they're not using it, so don’t hide behind the data.

Sometimes, as product managers, we just want everyone to stay away from us, all the different stakeholders that we have, and we can hide behind the data. So let's not do that.

And the last problem is that anything related to data is analysis paralysis by continuously drilling down into the data.

I love to analyze. I spend every day looking at how our products are being used. And sometimes I just spend hours on it and don't do anything with the data because I'm thinking, Well, what about this? What about that? What if I conduct this type of research?

So you need to have a breaking point when you're trying to act on the data.

Focus on the right metrics with a KPI tree

What I'm now going to show you is one way that helped us to connect the dots. This is one of the first steps that we took to solve the problems that were identified before.

It’s a KPI tree, and it’s something that helped us a lot.

In my current role, we tried so many times to be successful in a data-driven product organization, but we were constantly failing. Why? Because some exec would come along and say, “It's a mandate. You all need to have 100 metrics.”

So we’d do it and checkmark it. Who cares what metrics are? And that was it.

The other problem was that someone else would need to create a dashboard and we wouldn’t have access to the data. They’d put specific tools on us. And we were constantly finding reasons not to use the data.

So we said, “How are we going to help the PMs? We're going to train and coach them and provide a white glove service to all of them by helping them create a KPI tree.”

A KPI tree enables you to have the conversation to align across different teams on what the value of your product is. What are the key features that your product’s providing to your customers so that they can get the value? It's not related to revenue. It's not related to any of the profits. It's related to product usage.

If you want to be a product-led organization, you need to think in this way. That’s my opinion at least. It really helps.

The KPI tree aligns teams by connecting customer behavior, product metrics, and company goals. In many companies, you have a goal, but then you really need to think about what your strategy is to achieve that goal. What’s the North Star metric that you need to trigger?

The second part is that as soon as you have this, track the progress toward your strategy by monitoring relevant metrics and connecting them to your overall goals.

I mentioned that one of the key problems is that you have too much data so you don't know where you're focusing. And in this case, you can constantly track the progress of how you're executing. It helps you to maintain control and stay aligned.

And the last part is to coach product managers. You have a lot of churn and new product managers joining, and then everything starts all over again. What’s the value? Where's it written? All the documentation, etc. But having this KPI tree and building dashboards on top of it really enables them to have conversations and understand the customers and the product much better.

Harnessing the KPI tree: Pedal case study

In the Netherlands, cycling is a big thing, and now everyone’s buying an electric bike.

There's a very imaginative person who’s the Head of Product at a bike-sharing app called Pedal. In Amsterdam, we share everything because there's a lack of parking, a lack of everything. We have sharing apps for anything you can imagine.

This company, Pedal, really wants to increase ridership among customers. So they have a goal, they have an objective, and that’s what they want to achieve in the next two years.

Imagine that you’re responsible for improving the app’s user experience and increasing engagement with the platform. How are you going to do that? You're going to ideate and you're going to do customer research.

I'm now going to show you what happens both with and without a KPI tree.

What happens in companies is that they have the metrics and a dashboard, and it contains a dozen metrics that you're tracking. It's very time-consuming, everything’s scattered all over the place, and you don’t understand if you're on a good path or not. You have the data, things like monthly active users, and that's it.

But once you do this exercise, and trust me, it takes time and it takes conversations across different roles, what happens is that you’ll end up with a dashboard that’s based on the KPI tree. The team will have a well-defined understanding of what success means for the project.

So how do we believe that we're going to achieve this goal?

You're not sure. It's a hypothesis. But at least you have something. You’ve done the research and you’ve set specific goals on how you're going to achieve it.

The dashboard provides a more concise and relevant view of their work. It's one view that’s tailored to your needs and the goals of the product.

And for the last one, we’ve heard a lot about alignment with different roles. Once you do this exercise, my advice is to bring the designers, bring the engineers, and bring the sales, depending on your organization. Bring all of them into the room while you're defining the KPI tree so it becomes a shared document across various departments.

It really enables us to get alignment and become a product-led organization.

So this is an example of a KPI tree. The key thing and the most difficult one is to pick one metric, and that’s your North Star metric. PMs constantly prioritize, and this is the same thing. Pick one metric.

In this case, it's the daily active riders (DAR) that we're looking at, because we’re really interested in the fact that the bikes are being used on a daily basis.

And then you start to build your KPI tree.

There's a concept of leading and lagging KPIs. They're contributing KPIs that’ll enable you to achieve the specific North Star KPI. So of course, one of the critical things that we have to be aware of is how many bikes are available. Do we need 20? Do we need to scale up based on the number of users? So it's all in a positive relationship with the daily active riders.

With the two big additional KPIs that are important to identify, do we need to get more bikes? What’s the average wait time? Once you reserve, you get a time slot and sometimes it might be that it's too busy and you need to wait until you can get the second bike. And there’s also the bike utilization rate.

The second one is the user satisfaction rating. I think all of us do it in some way or another.

And then I've put the third one with dotted lines because it's a hypothesis. It's something that we want to achieve. And we’ve identified or we think that one of the other critical KPIs is the average ride duration. How much time are you going to spend on the bike? Are you using it for short rides or long rides? So the contributing KPIs are averaged distance traveled and frequency of use.

Of course, this is just a very basic example. But a KPI tree needs to be simple and concise.

How to gather more input

Don't be scared. You don't have all the answers, so it's good to make assumptions and start a hypothesis.

What can you do to actually prove those hypotheses? You’ll start to conduct user interviews. We constantly do that.

You’ll develop a customer journey map to understand the entire journey of users better.

And at the end, although it's a hypothesis, we’ll analyze the product usage data to pinpoint the potential activation models.

Essential reminders for your data-driven journey

There are things that you can’t forget on this journey. What happens is that as soon as you have this visible data, you’ll start to ask different questions. You’ll want to see more data and you’ll want to change your KPIs. These KPIs should help you.

When we did this the first time and we showed it to a PM, he said, “No, the data isn’t right,” because the usage was so low.

And we said, “Low usage data isn’t bad. It means that you can do something about it because now you see the data.”

So don't be afraid of your data.

And as you see the data and conduct additional user interviews, evolve your KPI tree.

Combine quantitative and qualitative. Always combine and don't just rely on your data. You don't have all the data, you don't have all the insights from the customers from their organization and their way of working. Every customer is different, so do continue to conduct additional analysis on them.

Talk cross-discipline. Involve the other roles while you're on this journey and bring alignment across all these different roles. It's a tricky journey, but it really, really helps.

And embrace the culture. We also mentioned how to create change in the organization. This didn't happen from the top down. This happened because we were tired of hearing everyone complaining about needing usage data. “Oh, we have some but I can’t use it.” No one was solving it.

So we said, “Let's create a small group and do it in our own time. And let's show the value and the impact we can create in the organization.” And now we’ve transformed an entire organization. And trust me, ServiceNow is huge. Doing this with a small team is so powerful, and you can make the change in your company.

So embrace the culture, and have fun with your data.