Survey significance calculator

How does the survey significance calculator work?

It runs a two-sided, two-proportion z-test on two independent groups. Under the null hypothesis that both groups have the same true percentage, the best estimate of that percentage pools the two samples:

pooled p = (x₁ + x₂) ÷ (n₁ + n₂)SE = √(pooled p × (1 − pooled p) × (1 ÷ n₁ + 1 ÷ n₂))z = (p₁ − p₂) ÷ SEp-value = 2 × (1 − Φ(|z|))

Here p₁ and p₂ are the two percentages as fractions, x₁ and x₂ the numbers answering, n₁ and n₂ the bases, and Φ the standard normal cumulative distribution function. The p-value is the probability, if the two populations really had the same percentage, of seeing a gap at least this large in either direction from random samples of these sizes.

The confidence interval for the difference uses the unpooled standard error, √(p₁(1 − p₁) ÷ n₁ + p₂(1 − p₂) ÷ n₂), and equals (p₁ − p₂) ± z* × that value. The test uses the pooled error and the interval the unpooled one, so in borderline cases the two can differ slightly.

The test assumes two independent random samples and enough respondents in all four cells (answering and not answering, in each group). A common rule of thumb is at least 10 in each; the calculator warns when a cell has fewer.

What is a worked example?

Segment A is 48% of 400 respondents (192 people) and segment B is 40% of 350 (140 people).

pooled p = (192 + 140) ÷ (400 + 350) = 0.4427SE = √(0.4427 × 0.5573 × (1 ÷ 400 + 1 ÷ 350)) = 0.03635z = (0.48 − 0.40) ÷ 0.03635 = 2.20p-value = 2 × (1 − Φ(2.20)) = 0.028

The 95% confidence interval for the difference runs from 0.9 to 15.1 points. The p-value is below 0.05, so the gap is statistically significant for a single planned comparison.

The line between significant and not is thin. A 45% against 38% gap with the same bases (7 points) gives p = 0.052, just over 0.05. And the same 8-point gap from ten times the sample (4,000 against 3,500) gives z = 7.0. Treat p-values as a measure of evidence, not as a switch.

How should you read the result?

• p-value. The probability of a gap at least this large if there were truly no difference. A small value means the data are hard to square with no difference. It is not the probability that the difference is real or that the result will replicate.
• Confidence interval. The range of true differences consistent with the data. It shows both the direction and the likely size, so it is more informative than the verdict alone.
• Not significant. This means the evidence is not strong enough, not that the groups are equal. A wide interval that includes both zero and large differences calls for more data.
• Significant. This means sampling noise is an unlikely explanation. Bias, weighting or how groups were formed can still produce a real gap that is not caused by what you think.

Is a significant difference an important one?

Not necessarily. Statistical significance depends on both the size of the gap and the size of the samples. With large samples, very small gaps become significant: with enough respondents even a 1-point gap passes the test. With small samples, a large gap may fail to reach significance.

Judge practical significance separately. Ask what size of difference would change a decision, then compare that with the confidence interval. If the whole interval lies below the size that matters, the result is significant but unimportant. If it spans sizes that matter and sizes that do not, collect more data before acting.

What about multiple comparisons?

Each test at the 5% level has a 5% chance of a false positive when there is no real difference. Run many tests, across segments, questions or waves, and some will look significant by chance. For k independent comparisons, the chance of at least one false positive is 1 − (1 − α)ᵏ.

The Bonferroni adjustment divides the significance level by the number of comparisons. It is simple and safe but conservative. Holm's step-down method is never less powerful and equally simple, and the Benjamini-Hochberg procedure controls the false discovery rate when many tests are run. Enter the number of comparisons in the calculator to apply Bonferroni to the verdict and the interval.

Decide which comparisons you will make before looking at the data. Testing every segment and reporting only those that look significant makes the adjustment meaningless.

When should you not use this calculator?

• The same people are in both groups. Before-and-after results from the same respondents, or two questions answered by the same people, are paired. Use McNemar's test.
• The groups overlap. Comparing a segment with the total, which includes it, breaks independence. Compare the segment with everyone else.
• The data are weighted or clustered. The test treats the bases as simple random samples. Weighting and clustering reduce precision, so use effective bases (base divided by the design effect) or the survey software's own test.
• The sample is not random. P-values assume random sampling. For opt-in panels and convenience samples they describe the sample, not the population, and should be read with caution.
• Counts are small. If any of the four cells has fewer than about 10 people, use Fisher's exact test.
• The measure is not a percentage. Use a t-test for means such as average scores, ANOVA for means across more than two groups, and a chi-square test for percentages across more than two groups.

Frequently asked questions

Cite or link this tool

You are welcome to link to this tool or cite it in a report, article, course or blog post. Copy the HTML to link to it with attribution, or use the plain-text citation.

Embed this calculator

Put this calculator on your own website, course page or blog post. It is free to embed, on one condition: keep the credit link under the calculator. The credit link is the licence for free use; if you remove it, please remove the embed as well.

The embed is a compact version of this calculator that works inside an iframe. It runs in the visitor's browser, and what they type is never sent to PulseLake or anyone else. Change the height if your layout needs it.

Related tools

• Survey sample size calculatorHow many completed responses do you need? Set the confidence level, margin of error, expected proportion and, if you know it, the population.
• Margin of error calculatorWhat margin of error does your sample give you? Enter the sample size, confidence level and proportion, and optionally the population.
• Van Westendorp price sensitivity calculatorPaste or load answers to the four Van Westendorp price questions and get the four price points, an acceptable range and a chart.
• Gabor-Granger price calculatorEnter the share of respondents who would buy at each price and get the demand curve, the revenue-maximising price and, with a unit cost, the profit-maximising price.
• Survey question bankStandard, widely used survey questions grouped by purpose, with exact wording, scale, when to use each, a common bias to avoid and its source.
• Research brief templateTurn a research request into a clear brief: objective, decision, audience, method, timeline and deliverables, ready to copy.
• All free survey and research calculatorsThe full list, with the formulas each one uses.

Formulas and references

The standard sources behind the formulas on this page:

1. Agresti, A. (2013). Categorical Data Analysis (3rd ed.). Hoboken, NJ: John Wiley and Sons. Two-proportion z-test, confidence interval for a difference of proportions and the chi-square equivalence.
2. Bonferroni, C. E. (1936). Teoria statistica delle classi e calcolo delle probabilità. Pubblicazioni del R. Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8, 3-62.
3. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65-70.
4. Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate. Journal of the Royal Statistical Society B, 57(1), 289-300.
5. Wasserstein, R. L. and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129-133.
6. Abramowitz, M. and Stegun, I. A. (1964). Handbook of Mathematical Functions. Normal distribution functions (series 26.2.10 and the continued fraction 26.2.14, used to compute the normal distribution).
7. Acklam, P. J. (2003). An algorithm for computing the inverse normal cumulative distribution function, with one step of Halley's method (used to find the critical value z).