I’ve written about metrics a number of times, since a lot of my career has been in tools to help people measure things. DURSLEy and CAPS talks about operational and business metrics, Product Sales Metrics is about measuring if your product is moving. There’s posts about measuring product quality, presenting metrics to leadership, building good reports, and of course general observability. All of that said, it’s useful to recognize when the tools aren’t appropriate. Some things aren’t measurable, or measured with tilted scales, or cost too much to measure.

It’s easy to think of obvious examples of problems; what’s tougher is when you’re working with a domain that seems like it should make measurement sense, but for some reason that measurement never happens or is really low quality and rarely used. It’s easy to tell that product or engineering leader that they should get busy and use the tools you’re providing… but remember, people usually have reasons for their behavior. Let’s explore a non-exhaustive list of possible reasons why a business might not be using measurement.

Measurement won’t affect outcomes

The most dramatic and hard to argue with reason for avoiding metrics? Because the organization is not currently data driven. If this is what you’re finding, it’s worth investigating if leadership is happy with that state of affairs. If they are satisfied with intuition and anecdata, then yeah, there’s not a lot of point collecting hard metrics. On the other hand, this reason can be used as an excuse to stay trapped in a poor status quo. If leadership would like to change the way things are done, the best time to start is now, one project at a time.

The data is too hard to work with

This one is also extremely common in my experience; there’s some willingness to try, but first level investigation finds that you need to gather more data or change the data collection in some way. For instance, maybe you want to calculate MAU (monthly active users) and DAU (daily active users), but your logs don’t disambiguate sessions from users. You can count sessions, but you don’t know how many unique users are represented in that number. Because this means the next step is no longer a managers’ side project, but rather a code change requirement, the next step is never prioritized and nothing happens. While this looks like a tactical blockage, it is in fact a reflection of strategic goals: visibility isn’t as important as the other things that are being done instead. Again, the only way around this is to work the process with people who can reprioritize. One technique that might help is to work with the data that already exists and highlight its limitations. There is a risk that this will empower anti-metrics voices though, since your results will not hold up well to scrutiny.

The tools are wrong

Sometimes I am a customer of data tools, and sometimes I am a vendor. I’m not going to pitch my products or trash others’ in this space, but I have to acknowledge a frequently raised objection to product metrics: “the data you want to measure is not what our monitoring toolset of choice is good at”. This can be frustrating for a product or engineering lead, because it means that you’re not going to get help or infrastructure from the existing toolset. It’s tempting (but inadvisable) to go rogue at this point and implement another system using shadow IT patterns. That’s not a terrible choice for doing analysis of internal metrics, but it’s easy (particularly in SaaS) to bump into the line of compliance-controlled data and create a problem that you’ll need help to clean up. The better choice is to work with IT to get access to a system that will meet the need. That means you’ll need prioritization, which means getting leadership help.

The revealed answer might not be what we hoped for

A difficult situation can arise when the product team’s intuition is that the answer metrics provide won’t be what leadership is wanting to see. Maybe the ideal scenario for a product’s design is not actually how customers use it; maybe the hope is daily use but reality is infrequent emergency response; maybe there’s dreams of being the single pane of glass instead of feeding the product that actually does that role. The reason this can be tough is that it requires leadership. Let’s say that the organization’s goal is to change those customer realities. It’s not ideal to ignore the present reality, but you also don’t want to crush hope with mountains of data explaining every detail of how far is left to go. This measurement blocker takes a creative, open approach to solve.

Selling the dream, and time boxing

It’s important to recognize that these reasons for pushback against product measurement are logical, even if they’re not ideal. In order to get progress towards data driven product development, you will need to deal with the logical reasons for that pushback. A lot of that is probably going to be in selling the potential benefits of product measurement. Personally, I wouldn’t pitch that as an ask for a top down initiative, but rather an ask for approval to seek incremental progress, buy tools if needed, and chase the low hanging fruit. A top down initiative to change product culture has to be fully committed as the most important thing for the organization to do… and that might well not be the case. Instead, improving product metrics collection should be encouraged as the result of feedback loops. You might need to kick start that with engineering help, but doing so should be done in a strictly time-boxed and results-driven way, like a proof of concept with a vendor.