Articles Posted in Benchmarks

The general counsel of Alfa Laval co-authored a solid article in the ACC Docket, Nov. 2011 at 39. The authors discuss a common method to describe possible outcomes: expected value. Each outcome that has a monetary result is expressed as the percentage of the particular outcome multiplied by the money.

Try this technique on the odds you estimate that a regulatory agency will impose a fine. A 50 percent chance of paying a $1 million fine has an expected value of $500,000; a 30 percent chance of a $2 million fine has a value of $600,000; a 20 percent chance of a $3 million fine has a value of $600,000. Together, the overall expected value of a penalty is $1.7 million, the total of each of the three odds-adjusted outcomes. It is all the results weighted by their likelihoods (See my post of July 15, 2005: three ways to reach expected value more realistically.).

Beware, caution the authors, that you don’t just tell clients the overall number. You need to convey the range of outcomes-with-likelihoods that constitute it.

“[M]aking a prediction in terms of probability simply does not fit well with predicting an event that will happen only once.” A lawyer who tries to give a client the likelihood as to whether something will happen – for example, the odds that the Department of Justice will oppose a proposed merger – by stating a percentage (“75% chance this will get by”) is really saying that if this identical deal happened 100 times, for 25 of them the DOJ would step in and stop it. A good article in the ACC Docket, Nov. 2011 at 40, makes this point.

The authors recommend that lawyers use “natural frequencies” to express probabilities, such as “in 25 out of a 100 instances like this, anti-trust objections will kill the deal” (See my post of May 5, 2011: natural frequencies compared to percentages or decimals.). Otherwise, if the likelihood given the client is more than 50 percent, the client tends to remember that as a prediction that this time it will happen.

An interview published this month of John Oviatt, chief legal officer of the Mayo Clinic, who oversees 35 lawyers and 45 other legal staff, covered his views of the importance of various benchmark numbers.

Oviatt’s view is that “the second most important of all metrics is the metric of leverage, the ratio of nonlawyers to lawyers, and particularly paralegals and contract managers.” He uses that benchmark “in demonstrating value to the C-suite that the legal department is focused on delegation of work to the least expensive competent level, and that implies increased utilization of paralegals and increased use of contract managers. We’ve actually decreased our total lawyer head count, but we’ve transferred some of those saved dollars into increased legal assistants and contract managers in our department.”

The GC Metrics benchmark survey found this year that for 303 North American law departments, the median ratio was slightly more than one lawyer for every non-lawyer (55% of total legal staff were lawyers) (See my post of Oct. 27, 2009: one-to-one ratio of lawyers to support staff with 9 references.).

For a legal department its ratio of legal expense to revenue needs to be one of the top metrics, in the view of John Oviatt, chief legal officer of the Mayo Clinic. Here are his five reasons from an interview on Law.com, which I have numbered for reference. [1] “It’s something that CFOs and CEOs understand. [2] It’s a metric that is generally used across shared service organizations …. [3] It’s one that is easily understood. [4] It’s easily tracked over time. [5] The information is accurate. At least your CFO will know if it’s accurate, because the information is provided and can easily be confirmed internally.”

Oviatt is right on all counts although realists can quarrel with 4 and 5. Let me gild the lily. [6] This regnant metric is comprehensive, which means it covers nearly all legal expenses (usually not settlements and judgments) and therefore not only can’t be gamed as much as can other partial metrics but also it embraces all kinds of structures and management choices. [7] It is also a normalized ratio, which means law departments of all sizes can match themselves to others. And [8] it is easy to calculate.

If you’d like to become more comfortable with logarithms and exponential functions, in the context of running a law department and understanding its metrics, you might have a hankering to read my InsideCounsel column on those mathematical relations. My Morrison on Metrics column compares linear and exponential functions and offers some insights on numbers raised to powers. All good stuff for cocktail party chatter!

Two websites are particularly well known for analyzing politician’s statements for accuracy, FactCheck and PolitiFact. Reading about them in the Economist, Nov. 26, 2011 at 43, I found myself wishing there were equivalents for articles about law department management (or blogs, for that matter). In some measure I have cast myself in that role. When facts or benchmarks regarding legal departments come to my attention, one of my first reactions questions the believability and accuracy of whatever is asserted. Hardly credulous, more like a pain-in-the neck quant pedant, it troubles me when numbers are tossed around carelessly. Even if a number sounds right, was the methodology for arriving at it sound?

Surveys by interested parties leak the most, but other times writers seize on a number and don’t bother to confirm it against other sources or to poke at it for even surface plausibility. A vivid and disturbing example is all the guesstimation of the size of the U.S. legal market.

The article recommends crowdsourcing tools, comment boxes for online articles, retractions and corrections by the publication, as well as “Standardisation – of data sources, measures of factual reliability, and platforms for sharing information.” I’m all for that and I hope this blog contributes to clarity and reliability in the facts about the legal industry.

When you hear of a statistical finding, you should want to understand that number’s reliability. If the research that produced the number were repeated several times, how much would the results vary?

Consider an example. Let’s make the simplifying assumption that the participants in the GC Metrics benchmark survey make up a reasonably random and representative sample of at least U.S. law departments. The margin of error for findings from a set of normal numbers shrinks in proportion to the square root of the size of the set. Hence, a benchmark finding based on 200 law departments – the participant base of the HBR Consulting (nee HildebrandtBaker Robbins) report – has a margin of error of 14.1. A finding from the GC Metrics report, based on 800 law departments, four times as many, has a margin of error of 28.3. That means the margin of error shrinks in half from the smaller to the four-times-larger survey.

A close approximation of the margin of error is 0.98/√n where n is the sample size. With 800 law departments (n=800), the margin of error calculates to approximately 3.5 percent. A finding based on that group could vary up or down by 3.5 percent and be just as reliable or likely as the finding given. For 200 law departments, the swing is 6.9 percent — four times more participants cuts in half the confidence interval so the results from the larger set is more precise and reliable (See my post of Dec. 9, 2005: margin of error and sample size; Aug. 30, 2006; sampling error; April 22, 2007: error; and Oct. 31, 2007: formula for margin of error.). With benchmarks, respondent size matters.

Metrics from small law departments exhibit much more variability than the same metrics from large law departments. For example, from one year to the next, outside counsel spending per lawyer will swing higher or lower for law departments with one-to-three lawyers than for departments with 20+ lawyers. The explanation, drawn from Daniel Kahneman, Thinking, Fast and Slow (Farrar, Straus & Giroux 2011) Chapter 10, results from what he calls the “Law of Small Numbers.” Kahneman explains that “extreme outcomes (both high and low) are more likely to be found in small than in large samples.” Think of it this way. Small law departments operate on a smaller sample of incoming invoices than do larger departments, so the variability (the standard of deviation in the annual sets of invoices) is greater.

As a second illustration, year-over-year variability will tend to be greater from smaller benchmark surveys than from larger ones. If 150 companies take part in back-to-back years, it is less reliable to state something like “a 2% increase in total legal spending” than if 850 law departments take part each year.
Sadly, Kahneman notes, “We pay more attention to the content of messages than to information about their reliability” (at 118).

To lessen the influence of outliers, a set of numbers can be Winsorized. Named after Charles Winsor, to Winsorize data, tail values at the extremes are set equal to some specified percentile of the data, such as plus and minus four standard deviations. Here’s how it’s done. For a 90 percent Winsorization, the bottom 5 percent of the values are set equal to the value corresponding to the 5th percentile while the upper 5 percent of the values are set equal to the value corresponding to the 95th percentile. This adjustment is not equivalent to throwing some of the extreme data away, which happens with trimmed means (See my post of May 28, 2007: five percent trimmed mean.). Once you Winsorize your data, your medians will not change but your average will.

Using the data from the 652 participants so far in the GC Metrics benchmark survey, I Winsorized the data for number of lawyers. Using a 90 percent process, at the small end meant changing 9 of them to 1 lawyer, which was the 5th percentile value from the bottom (the other 23 were already 1). At the high end, I changed the 32 with the most lawyers to the 95th percentile figure, 110 lawyers. After Winsorization, the median of 6 lawyers stayed the same but the average dropped from 25.96 lawyers to 18.03 – a decline of 30 percent because several very large law departments were drastically Winsorized.

Many posts on this blog have dipped into the well of the annual Fulbright & Jaworski surveys of litigation data. Each year the firm polls a few more than 400 law departments in the United States and the United Kingdom.

If the responding group year after year remains significantly similar might not the results reflect some degree of self-reference? If one year’s results show a decline in international arbitrations, to imagine one possibility, might the respondents read that and be influenced in their behavior during the following year, or just in how they remember and answer the next year? They hear the echo of their own voices and answer questions to some degree influenced by their own reported behavior.

My question ranges more broadly than F&J’s admirable investigations. We could surmise the same reverberation and influence from the annual ACC/Serengeti surveys. The echo chamber should have little effect on quantities that are easily counted, such as budget figures or staffing numbers. Its risk rises substantially on qualitative assessments, such as regarding the perceived effectiveness of alternative billing arrangements. When respondents have heard their own, collective assessments one year, the bandwagon effect the next year may reinforce beliefs and thus answers in the same direction.