brainstorm-experiments-existing

Product Experiment Design Assistant — Validating Hypotheses for Existing Products

Skill Overview

Helps product teams design low-investment experiments for existing products. Through methods such as prototype testing, A/B testing, and fake door testing, teams can quickly validate hypotheses and avoid unnecessary development costs.

Applicable Scenarios

Validating New Feature Hypotheses

Before investing in development, use low-cost experiments to test whether users genuinely need a feature. For example, use a fake door test to validate the authenticity of demand before deciding whether to proceed with full development.

Testing Product Ideas

When a team has multiple feature ideas but limited resources, use methods such as prototype testing and Wizard of Oz to quickly identify the most valuable direction and avoid developing features that no one will use.

Planning Product Experiments

Develop a complete experiment plan for an existing product, including selecting appropriate validation methods, defining success metrics, and mitigating risks to ensure the experiment produces actionable insights.

Core Functions

Multi-Method Experiment Design

Recommend the most appropriate validation method based on the type of hypothesis and the product stage:
- Prototype Testing: Use clickable prototypes to test task completion and measure whether users can successfully complete the core flow
- Fake Door Testing: Place an entry point for an unimplemented feature in the product and use click-through rates to validate the authenticity of demand
- Wizard of Oz: Manually simulate an automated feature behind the scenes to test users’ responses
- Technical Spike: Use minimal technical investment to validate technical feasibility
- A/B Testing: Compare different solutions in a production environment, with risk mitigation strategies in place
- Behavioral Surveys: Design questionnaires based on actual behavior rather than opinion-based research

Behavior-Oriented Metric Design

Emphasize measuring actual user behavior rather than stated opinions. For each experiment, define:
- A clear hypothesis statement (“We believe…”)
- A specific experiment plan (“We will…”)
- Quantifiable success metrics (“What will we measure…”)
- A clear success threshold (“If the hypothesis is true, we should see…”)

Responsible Testing and Risk Mitigation

Consider the following when designing experiments:
- Do not inconvenience or mislead users
- Do not cause irreversible damage to the business
- Establish clear risk mitigation measures for production experiments such as A/B tests
- Maximize learning value with minimal investment

Frequently Asked Questions

Where should product experiment design begin?

Start by clarifying the hypothesis you want to validate. This skill guides you to first clarify the product idea and the core hypothesis that needs to be tested, then recommends suitable experiment methods for each hypothesis. The key is to begin with “What do we believe?” rather than “What method should we use?”

How can an idea be tested without developing the full feature?

There are many ways to validate an idea with little or no code: use a clickable prototype to test the user flow; add a button in the product that leads to a “Feature in Development” page to test the strength of demand; or manually handle user requests to simulate the feature through a Wizard of Oz experiment. This skill recommends the most appropriate low-investment method based on your specific situation.

How can A/B testing be conducted safely on an existing product?

A/B testing requires particular caution, especially for products that are already live. When designing an A/B test, this skill includes risk mitigation strategies such as starting with a small percentage of traffic, setting clear stop-loss conditions, ensuring the core user experience is not affected, and defining the data analysis plan in advance. It is recommended to first use prototype or fake door testing to validate the basic hypothesis before considering A/B testing.

How can experiments be designed to truly validate user behavior?

Avoid relying on what users say; observe what they do. This skill emphasizes designing experiments that measure real behavior: prototype testing examines whether users can complete a task rather than asking whether they think it is easy to use; fake door testing examines whether users click a feature entry point rather than asking whether they would use it; and A/B testing examines actual changes in conversion rates. Each experiment clearly specifies the behavioral metrics and success thresholds to be measured.

When should prototype testing be used instead of A/B testing?

Prototype testing is suitable for early-stage validation, such as determining whether an interaction design is reasonable, whether users understand the feature, and whether the flow is smooth. A/B testing is more suitable for later-stage optimization, such as determining which of two solutions performs better and measuring the actual impact of a change on key metrics. In general, first use prototypes and fake door tests to validate the basic hypothesis and direction, then consider A/B testing to optimize details. This skill provides recommendations based on your product stage and validation objectives.