Disappointed recently with sloppy benchmark efforts by several purveyors, I realized that the law department world lacks third-party assessments of benchmarking efforts. Journalists from time to time go a round or two with data from benchmark surveys, mostly compensation, but they never lay a glove on methodology. Any law firm, vendor, academic, consultant, or trade group can mush together a grab-bag of data, stir it around, and pronounce “findings.” No one vouches for the appropriateness of procedures, the absence of bias, or the plausibility of findings. No one articulates standards of good benchmarking.
Dubious benchmarks thereby become urban legends of management. Loose and unreliable numbers float around. The data debris that results from poorly-executed benchmarking suggests a variation on Shakespeare: “The evil that bad benchmarks do lives after them; the good is oft interred with their bones.”