Ian Ayres, Super-Crunchers: Why Thinking-By-Numbers is the New Way to be Smart (Bantam 2007) describes in Chapter 2 how business uses randomized tests to learn more about their markets. For one group of consumers, picked at random, they do something and for the rest they do something different, then pore over any changes in outcome. It made me think of ways that general counsel could harness the power of randomized studies (See my post of March 25, 2008: send sample matters to both external and internal counsel to see differences in cost.).
As one instance, a law department might randomly choose matters to have their invoices carefully reviewed and the law firms advised of write-downs. Three months later look to see whether the effective billing rates or cost structure of those matters were any different than matters for which bills were not carefully reviewed. More precisely they could test whether the reductions end up with a cost saving net of the time it took to achieve them. Any change would be an example of what statisticians call the “treatment effect.”
A second test would assign similar cases at random to two capable law firms and then track the performance differences. Or use a disciplined ECA procedure on every other case for six months. If the two groups are similar in every other dimension, the department can be confident that any change in the two groups’ outcome was caused by their different treatment.