Finding your best experiment method
Do you like to cook?
For a beginner like me, I rely on recipes. I am not ready to open the fridge, scan food items, and quickly decide on a meal.
I encountered a similar problem with product experimentation. I was not ready to pick up an experiment method without comparing it to other options. When going from idea to experiment method I longed for a “recipe”. Sometimes I would pick usual suspects like a creating a landing page, write a blog, or attend an in person event to network.
With a combination of market research and years of experience working with product teams, I learned that there is no online repository of experiment methods from idea to scale. Experiment methods at the idea stage tend not to rely on technology whereas experiment methods at scale are built with technology.
My solution to finding the best experiment method from idea to scale was two-fold. I needed to curate a list of experiment methods and design a search criteria compatible with situational context scenarios.
My first experiment was curating a list in Google Sheets and sharing with my network. After positive feedback, I focused on iterating different search criteria.
The latest iteration of search criteria is divided into three groups:
Group 1: cost, speed, speed + direction
Group 2: qualitative, quantitative
Group 3: macro, details
The purpose of group 1 is to highlight important constraints applicable at different stages from idea to scale. At the beginning, running cheap and fast experiments is fundamental to extend your runway and avoid a late disappointment. As you progress towards product-market fit, speed and direction is the fuel to check you are on the right track.
The second group emphasises the need to make data informed decisions. Aside from gut feel or years of industry expertise, data is very important to avoid cognitive biases that can influence what decisions we make.
The third group applies a big picture and detail perspective to everything we do. For example, early on we want to emphasise the market (macro) versus a specific feature (detail).
The power of the search criteria manifests when you combine search criteria. For example:
-
“I want to test my idea cheaply”
“cost” + “qualitative” + “forest” -
“I want to validate a solution with the help of angel investor funding”
“speed” + “qualitative” + “trees” -
“I want to scale my product to find product-market fit”
“speed + direction” + “quantitative” + “trees”
In the online search tool called MethodKit expect 45+ experiment methods from idea to scale as diverse as “mashup" and “feature flags”.
Any feedback is greatly appreciated to keep improving the product experience.
Happy experimenting!
