Customer survey designer and analyzer

Designing the questions

A survey measures what its questions ask, not what the writer meant. Most bad survey data comes from a handful of wording faults, which the tool looks for:

• Double-barreled: two things in one question ("speed and cost"). A customer happy with one and not the other cannot answer.
• Leading: the question suggests the answer ("our friendly technicians…", "don't you agree…").
• Loaded or assumptive: emotionally charged words, or an assumption ("Do you still have problems…").
• Absolutes: always, never, every. Few people can agree with an absolute.
• Negatives and jargon: disagreeing with a negative statement is a double negative; an acronym the customer does not know gets a guess.
• Unbalanced scale: more positive than negative choices pushes answers upward.

The checks match words, so they raise false alarms and miss subtler faults. Pilot every survey with a few real respondents.

Summarizing rating answers

Likert and rating scales are ordinal: the points are in order, but the gap between "agree" and "strongly agree" is not known to equal the gap between "neither" and "agree". So the tool reports three kinds of summary:

• Top-box and bottom-box: the percentage choosing the top (or bottom) one, two or three points. This is the most robust summary and the easiest to explain. Its margin of error is z√(p(1−p)/n).
• Median: the middle answer, valid for ordinal data.
• Mean and standard deviation: widely reported, but they treat the scale as equally spaced. A large standard deviation means opinions are split and the mean describes almost nobody.

The diverging bar chart centers each question on the neutral point, with lower ratings to the left and higher to the right, so the size of the unhappy group is visible at a glance.

Net promoter score

NPS asks how likely the customer is to recommend you, on 0 to 10. Promoters answer 9 or 10, passives 7 or 8, detractors 0 to 6. NPS = % promoters − % detractors, from −100 to +100. Scoring each respondent +1, 0 or −1, the standard error is √((p_P + p_D − NPS²)/n), and the margin of error is z times that, in points. Because NPS is a difference of two percentages, its margin of error is wider than either one: with 200 answers it is often ±10 points or more. The tool tests whether the gap between the best and worst segment is larger than chance would explain.

Sample size and response rate

For a percentage near 50%, the sample needed for a margin of error E is n = z²(0.25)/E², about 385 for ±5 points at 95% confidence, reduced by the finite-population correction n/(1 + (n−1)/N) for a small population. A low response rate raises the risk of non-response bias: those who answer are often the most pleased and the most annoyed.

On the exam

CQPA IV.B covers the tools for gathering customer feedback, including surveys. CQE I.G asks you to define, apply and analyze the results of customer satisfaction surveys, and CQE VI.A.2 covers measurement scales (nominal, ordinal, interval, ratio). CSSBB IV.A.2 asks you to select data collection methods such as surveys and to review them for validity and reliability, and CSSBB II.C.1 lists customer loyalty metrics. Expect to spot a double-barreled or leading question, choose a summary for ordinal data, and interpret a net promoter score.

Other tools and templates