Metrics and benchmarks provide opportunities for sleight of hand, for tricks that fool some people, for willful ambiguities, and for dumb decisions. Here are eight candidates for booby prizes other than samples that are not representative and questions that are not neutral.
Frame the findings with “save up to” or “reduce up to” language (See my post of June 2, 2011: misleading to tout extremes.).
Use indefinite numbers like “lots of” (See my post of May 24, 2011: Kraft’s law department clarifies estimative language.).
Emphasize averages instead of medians, especially with smaller sets of numbers (See my post of Nov. 30, 2005: suggests using the average of the middle.).
Report unweighted, un-normalized numbers like absolute numbers of lawyers rather than lawyers per billion of revenue (See my post of Nov. 10, 2009: weighted metrics with 12 references.).
Pick a low or high starting point or date, depending on what change you want to promote, such as the year of the hugely expensive lawsuit as the baseline for showing a reduction in spending (See my post of Sept. 12, 2011: baseline from which you measure makes a huge difference.).
Choose assumptions that help make your point. If you want to estimate the savings from some initiative, new software for example, in the absence of data you assume that lawyers spend at least five minutes per day doing something and that the software will reduce that time. You have baked in your conclusion to some degree.
Fail to explain your margin of error (See my post of Oct. 31, 2007: formula for margin of error and two references; and Dec. 9, 2005: margin of error and sample size.).
Conceal an obvious interest in the findings (See my post of Sept. 13, 2011: potential for service-provider influence on data.).