Interpreting Statistical Significance
Knowing when to stop a poll and trust the data.
The Null Hypothesis
In a binary poll, the null hypothesis assumes that the true population preference is exactly 50/50. Any deviation from 50/50 in your sample is assumed to be due to random chance. Statistical significance is the probability that you would see your results (or more extreme) if the null hypothesis were true.
The 95% Confidence Standard
Industry standard dictates a 95% confidence level (p < 0.05). This means there is only a 5% chance that your observed results are a fluke. For high-stakes decisions (e.g., medical device interface design), 99% is required. For low-stakes decisions (e.g., blog post titles), 90% is often sufficient.
The "Peeking" Problem
Do not check your poll results every hour and stop it the moment it crosses the 95% threshold. This is called "p-hacking" or the "peeking problem." Because early data is highly volatile, you are likely to capture a false positive.
Always set your target sample size before launching the poll, and do not make a decision until that threshold is reached.
Use our Significance Tester to evaluate your final results before acting on them.