A better way for surveys to ask respondents to rank elements of a set

A survey for law departments asks respondents to “rank from ‘1’ to ‘9’ in order of frequency of use the billing arrangements listed below.” A typical list follows, not in alphabetical order I note (See my post of Dec. 20, 2005 with methodological warnings.).

What I like was the survey’s recognition that not every law department has experience with every method, so they cannot meaningfully rank them all. To that point, the instructions say: “Note: if you do not employ some of the billing arrangements listed, please leave ranking blank. For example, if you only utilize four the items below, rank the four you to use from 1 to 4.”

Without this methodological improvement, some amount of the data will not be based on reality. Respondents will rank even those billing practices where they know nothing about effectiveness ((See my post of May 17, 2006 on the methodology of a survey on judicial hellholes.).