Eliminating Bias Through Binary Constraint
Why the 1-10 scale is fundamentally flawed, and how forced A/B decisions reveal true user preference.
The Central Tendency Problem
When presented with a Likert scale (e.g., "Rate from 1 to 5"), humans experience cognitive friction. Making absolute judgments requires energy. To conserve energy, respondents frequently select the middle option (3) or a "safe" positive (4). This is documented as central tendency bias.
The Illusion of Granularity
Product managers often believe a 1-10 scale provides more nuanced data than a binary choice. In reality, the difference between a user rating a feature a "6" versus a "7" is statistical noise, heavily influenced by their mood, the time of day, and anchor bias.
Forcing the Trade-off
Real-world decisions are rarely made on a sliding scale; they are made via trade-offs. You cannot spend the same dollar twice. You cannot build two features with the same engineering hours.
Binary A/B polling replicates this economic reality. By forcing a choice between two mutually exclusive options, you extract a signal of relative utility.
Example: The Pricing Page Conundrum
Consider asking users about a new feature:
- Bad (Likert): "How valuable is Single Sign-On to you? (1-5)" -> Result: Average 4.2. Action taken: Built SSO. Nobody upgraded.
- Good (Binary): "Would you rather have SSO, or faster report generation?" -> Result: 82% chose Reports. Action taken: Built reports. Revenue increased.
Next Steps
Ready to structure your own binary prompts? Review our Survey Design Guide or test your sample sizes with our Statistical Calculator.