Data from surveys that use multiple choice questions should give readers pause. For an example, see my post of July 21, 2005 about findings from BTI.
It is very difficult to craft questions that have accurate, comprehensive and mutually exclusive choices. Multiple-choice questions with more than seven answers risk confusing respondents. Multiple-choice questions also run the risk of omitting a choice that respondents would like to select. One test for this, however, is the number of clients who mark the ubiquitous “Other.” The fewer the number of “Other” responses, the more accurate and comprehensive were your choices.
The order of choices makes a difference. A surveyor can skew replies by listing first the most desirable choice, the one that the surveyor wants to find.
Worst of all, a question can ask respondents to “check all that apply.” No one can make much sense out of the jumbled data that results.