Published on:

This skeptical blogger, the null hypothesis, and proof of management effectiveness

“I conclude that I’m a skeptic not because I do not want to believe but because I want to know.” The quote, with the italics, is from Michael Shermer, the columnist in Scientific Am., Vol. 301, July 2009 at 33.

As a kindred skeptic who wants ever so much to know what works for legal department operations, I mistrust many claims about the efficacy of legal department practices. I march arm in arm with Shermer, who writes that “Science begins with the null hypothesis, which assumes that the claim under investigation is not true until demonstrated otherwise.” For instance, people claim that partnering benefits a law department. Show us metrics that compare departments that partner with those not coupled. Convergence is claimed to be best? Where are the metrics that demonstrate the null hypothesis is untrue, that show total legal spending declines in step with declines in the number of firms retained.

Shermer makes the point that the “burden of proof is on the person asserting a positive claim, not on the skeptics to disprove it.” You say that document management software has saved bundles? Such a positive claim needs support, or others can only say – as with all the practices that are asserted as beneficial – that they may well work but we do not have proof. Anecdotes and assertions do not prove a claim; at best they depict it. Benchmarks need to show correlations we skeptics can trust (See my post of March 11, 2009: correlations, missing in action.) or other forms of demonstration need to be adduced.

Posted in:
Published on:
Updated:

One response to “This skeptical blogger, the null hypothesis, and proof of management effectiveness”

  1. Mr. Rees, this is a peculiar post. I can’t quite accuse you of knocking document management software, because as a member of the community, I think people are right to be skeptical. Nevertheless, perhaps software is like a law firm partner, sometimes it’s not the right fit, and it costs the firm too much aggravation and money. So, I guess the rule is, be careful about picking your law firm partners, and just as careful selecting a document management software.