At Amplitude, I facilitate lots of workshops. Many of these workshops center around deciding what to measure. This starts, of course, with asking good questions. Good questions help teams focus their measurement/telemetry efforts, and increase the likelihood that what they measure will enable valuable insights. A couple of well chosen events and event properties beat out a firehose of data and/or an autotrack solution any day.
In this post, I’d like to share some helpful approaches to eliciting impactful questions so that your team can make better decisions about what to measure. This takes practice, but better question asking is within any team’s reach.
Before we jump into the nitty gritty, I wanted to share the most important thing I’ve learned:
You need to make it safe to ask “dumb” and less fully-formed questions. And you can’t rush it. When the people brainstorming are worried about looking silly, they’ll shut down. If they feel rushed, they’ll stick with surface-level questions. Great questions spring from less-great questions which spring from “bad” questions. It takes time and multiple cycles of divergence and convergence to come up with the highest impact questions.
So make the time and make it safe.
OK, with that out of the way, let’s get started.
For a long time, I would start my question brainstorming activities with the question “where do we need to reduce uncertainty?” That was it. Some teams loved the ambiguity. They ran with it. But for other teams it was too open-ended. It was intimidating. I’ve since adapted my approach.
When it comes to being “data informed”, people generally have one of three related needs:
- I need to make a decision, and I need data to inform that decision.
- I want to reduce uncertainty related to an assumption.
- I want to understand performance and impact. I want to know if something is working. I want to prove something is working (or not working) or will work (or will not work).
I say these are related because decisions involve assumptions. Assumptions guide decisions. And we typically want to know if something is working so that we can make a decision of some sort (even if that decision is to do nothing).
But it can help to divide these out when trying to elicit questions. Why? I’m not exactly sure, but it feels like different people gravitate to different perspectives. Just using one approach (e.g. just using a lean canvas filled with assumptions) seems to limit teams. I also see teams stress out about benchmarks and “standard” metrics, with no real sense of what decisions they hope to inform and/or assumptions they hope to validate. Maybe this provides more flexibility?
My next realization was that the resolution of the question—the level—matters. When brainstorming, refining, and prioritizing questions, it helps to try going up and down a level (or two). An open-ended question is helpful to inspire more specific questions. And a specific question is helpful to inspire more open-ended questions. Why is this important? You can engage everyone in the activity, regardless of how general or specific their questions tend to be. Also, when you leave ten minutes to brainstorm questions, teams stick to a single level instead of exploring other options.
To make this point, I share a table that looks like this (this is from the actual board we use in Miro):
The table has three columns: one for decisions, one for assumptions, and one for areas of performance and impact. For each column I give sample questions/assumptions along a spectrum of specificity.
For example, we might be deciding whether to even continue working on a product. Or we might be deciding on button placement. Our assumption might be foundational to our whole business (“demand will increase over a decade”), or it might be about an assumption related to button placement (“that type of button always goes on the right”). We might be wondering about the efficacy of our whole strategy, or about the efficacy of a small workflow tweak.
To get warmed up, I ask participants to brainstorm three examples for each column.
- Example decisions
- Example assumptions
- Example is-it-working type questions
But I add a twist. “Make sure to give me one super specific example, one super broad example, and one medium-specificity example.” Hopefully you can see what I’m up to. This is like an active stretching routine before a workout.
Completed for a DIY app pairing builders with kit designers it looks like this:
With some examples on the board, we flow immediately to the next step.
“OK. Now pick one decision, one assumption, and one is-it-working question to explore further. Brainstorm three sub questions for each. Where must you reduce uncertainty? What questions would—if answered—help you unlock this puzzle? Or at least increase your confidence?”
I also remind them of their options: why, who, what, when, where, which, how many, how, how long, do, are, will, have, should, and is.
This two step process—exploring the categories and levels, and then brainstorming sub-questions—gets people thinking more laterally, and more willing to go up and down question levels. It is much better than just jumping into questions.
If a team is having trouble, or if they just want more practice, I bring out these trusty fill-in-the-blanks.
- How many users ______ ed in the last 30d?
- Where are new users dropping in in the ______ funnel ?
- Does ______ and ______ impact long term retention for ______s?
- How well do ______ s retain compared to ______ s?
- Are users who ______ more likely to go on to ______?
- What’s the average number of _______ per ______?
- Where do customers go after ______, and do they end up _______ing?
- What unique customer behavior predicts ____________?
- When _________ did we adversely _______ ?
- Are people actually ________ ing, or are they just ________ ing?
- Where/when are customers having trouble when attempting to _______?
- Are our efforts to ________ resulting in ________ ?
- Is what we released causing ________ , or is that just ________ ?
- Is there any low hanging work that would let us ________ ?
- Are we on track to ________ ?
Together, these activities leave people a bit more confident about brainstorming questions.
For the remainder of the workshop we do the following:
- Brainstorm additional decisions (“should we…”), assumptions, and is-it-working type questions.
- Share and discuss. Tweak. Repeat.
- Dot vote or place Monopoly money to “pay” for valuable questions.
- Brainstorm sub-questions individually. Shoot for high volume.
- Review sub-questions as a group, and refine in pairs. Prioritize.
- Rinse and repeat until time runs out.
By the end of the workshop, we typically have lots of questions and sub-questions, but we also have some sense of what questions are valuable. We prioritize where it will be valuable to learn more. Perhaps even more importantly, we learn what “class” of questions are valuable. By that, I mean we learn the nouns, verbs, workflows, and goals that are most important.
This is one of the secrets to a smart instrumentation approach.
No, you can’t (and shouldn’t) instrument everything. No, you shouldn’t record every click. No, you can’t predict every question in advance. But you can, guided by good questions, instrument the things that unlock what we call the “long tail of insights”.
There is a misconception that the relationship between what is instrumented and the insights you can unlock is linear. Instead, a handful of intelligently instrumented events can enable a wide range of insights and metrics. Baseball is a great example of this. There is only so much you can do in baseball—swing, hit, run, steal, catch, pitch, etc.—but the world of baseball stats and analysis is endless.
Good questions guide the way. Once you’ve prioritized important questions, the next step is to ask “what can we measure to reduce uncertainty related to this question?”
And join us for Amplify, our annual conference for product nerds. October 14-16.