Product quality is a subjective measure, so you have to find ways to objectively track effort without creating a Goodhart’s Law problem. Since you’re not going to get an easy and accurate metric, you’re left to verifying proxy metrics.

1. Establish common processes and track that they are followed. That’s design and development style books, in-house library management (did someone say SBOM?), architecture and design reviews, unit tests, automated and manual functional tests, code reviews on every PR, and so much more. Most of all, it means rigorous definitions of done and explicit staffing for the task of ensuring process is followed (note that this is totally out of style, but is in my opinion necessary for any of the subsequent ideas to function).
2. Dashboard objective signals of customer dissatisfaction, such as support cases and ERs and churn by cohort. You may not know how much quality is enough, but you can tell when you don’t have enough to justify your cost. Be cautious that you don’t mistake a market fit problem for a quality problem though; fixing all the bugs won’t bring the Betamax back.
3. Dashboard objective signals of team throughput, such as tickets created versus resolved by type. Again, this is pointless if teams work from different definitions of done, and if you didn’t think about Goodhart’s Law before, here’s a good time to review it. These signals are very easy to game and can become a brake on the business, even if you’re not intending to use them in a bad way. Your dashboard might be used by others without your knowledge and guidance, or even outlive you at this company. Instead, you could try to dashboard some objective elements of quality design or implementation, like demonstrations that common libraries are used instead of custom approaches, but I honestly think it’s less effective than looking at customer satisfaction.
4. Ask questions about the resulting dashboards, uncover which teams are moving fast and breaking things, decide what you want to do about that because it’s very likely a conscious trade off being made for good reasons somewhere in the org.

So, that’s a lot of processes and dashboards to track that they’re getting used and dashboards to track their results. Without explicit staffing for QA and PjM (project management), maintaining those processes and nagging people into following them are added to the tasks of the “empowered squad” or whatever it’s called in your org. Say you want a result like “everything gets tested before we give it to a customer”… well, your EM and PM are now trying to make that happen, in addition to the jobs they were hired for, so it’s one of the things that falls off their plates under load. This is especially true because the testing won’t uncover bugs from complex interactions and scale, which is why you see figures like Charity Majors saying “just test it in production”. An experience EM or PM will do the same analysis when deciding if it really matters that everyone forgot to click the Verify button in Jira, or log their hours, our put tickets into sprints.

The natural outcome of this is that quality issues, like security, only become really important when something goes loudly and publicly wrong and an executive yells about it, and then only until the yelling stops.

In short, if it’s important to measure product quality, make someone own it.

“But wait”, you might be thinking, “haven’t I read a bunch by product management luminary Marty Cagan that process people are bad?” Indeed! That post and its follow up and another about scaling process and another about process, as well as the books, paint a nuanced picture of people using process to make products. I’m not going to summarize that effectively in a few words, but I will say that it’s useful to question if measuring subjective values is worth the effort. Objective ones like revenues, pipeline effectiveness, and churn rate are hard enough to track. An organization large enough to lose legibility begins to have consistency challenges, which leads to process and plans and people to enforce them, great. But “if you hire some designers, you’re going to get a lot of designs.” People need to keep their jobs, and the process engineering work of product operations never approaches done.