Law Department Management Blog
A frequent kind of question on surveys is the multiple-choice question, even though they are beset with methodological traps. Especially egregious are those questions that invite respondents to “choose all that apply” (See my posts of July 21, 2005 about that instruction; Dec. 20, 2005 that criticizes such a methodology; and March 13, 2006 on “choose more than one.”).
An improvement is to ask respondents to pick their top three or four of a large set (See my posts of Aug. 14, 2005 on this variation as well as Nov. 5, 2006; July 4, 2006 on “pick the top 5”; Nov. 5, 2006 on “pick your top three from ten.”).
Even better is to ask respondents to rank their choices (See my posts of Dec. 20, 2005 on putting priorities on the choices; and March 31, 2007 on alignment with clients.). One problem with this step is that even undesirable choices will be ranked. They can also misread the instructions and reverse their ranking from what you wanted.
Best of all for multiple-choice questions is to ask the survey respondents to allocate 100 points among the options (See my post of July 4, 2006 on this sophisticated technique.). They can give as many points as they want to their most favored choice and sprinkle fewer points among the remainder as they see fit. The downside of doing so is that respondents must think more and take more time, which discourages participation. Still, this survey technique yields the clearest understanding of the respondent’s views.