Articles Posted in Benchmarks

I have written frequently about correlations (See my post of Feb.13, 2008: correlations with 16 references.). What I haven’t explained is how to find out whether a correlation is one that you can rely on.

The statistician’s term is “statistically significant,” a standard that has three components. To explain them let’s start with a correlation, such as between median partner hourly rates and median associate hourly rates. You collect that data for 20 law firms, enter it into a spreadsheet, and use a built-in function to calculate the correlation between the two sets of figures. The correlation is 0.45. How confident can you be that the correlation really means something and isn’t just some chance finding? I found a very clear explanation online of statistical significance and tinkered with it.

The easiest way to find out is to look in a statistics book that has a table of critical values of r (the correlation figure, here 0.45). You need to decide on a significance level, which is commonly called alpha and set at .05. This means that the odds that the correlation is a chance occurrence are no more than 5 out of 100.

Northeast Utilities is a utility holding company whose units serve about 1.9 million customers, generate revenues of about $5.8 billion, and employ about 6,200 workers. General Counsel Gregory Butler heads a team of 40 lawyers and 18 staff. In the article from the Nat. L.J., Oct. 22, 2009 where this background is given, he estimates that “Nearly 60% of the utility’s legal work is handled in-house with the rest performed by outside counsel.” That article is studded with benchmarks and points about benchmarks.

The ratio of lawyers to staff is much higher than the norm of one to one. Northeast Utilities has a bit more than two lawyers for each non-lawyer staff person. (See my post of April 18, 2009: lawyers as percentage of total legal staff; March 28, 2006: EMC has high ratio of lawyers; May 10, 2006: the US Department of State and its 160 lawyers and 140 support staff; Jan. 25, 2007: GM with its 107 attorneys and 109 support staff; Dec. 23, 2005: the ratio of one-to-one in prosecuting attorney’s offices; Nov. 28, 2007: Cummins, Cisco and the declining numbers of support staff; June 17, 2008: Starbucks and one-to-two ratio; June 18, 2009: Aetna figures; and Aug. 10, 2009: Abbott is very close to one to one.).

The ratio of one lawyer for every 155 employees may be meaningful to some general counsel, but I doubt its usefulness (See my post of April 18, 2009: lawyers per 1,000 employees with 6 references.).

Email me if you would like a copy of the 2009 Europe, Middle East and Africa (EMEA) benchmark survey or the Asia-Pacific (APAC) survey. Compiled by Laurence Simons, legal recruiters, and Rees Morrison, they cover 123 EMEA legal departments and 57 APAC departments, respectively.

Total legal spend in APAC per APAC lawyer came in at a median of $473,787. For EMEA, using an exchange rate of 1.3 dollars per Euro, the median was $646,787 (€497,529). Thus, inside plus external costs in APAC were approximately three-quarters of those in EMEA.

If your legal department in one of those two regions has a figure higher than the median figure, it could suggest:

Recent benchmark surveys by Laurence Simons,legal recruiters,and Rees Morrison report on 123 Europe, Middle East and Africa (EMEA) legal departments and 57 Asia-Pacific (APAC) legal departments, respectively. Email me if you would like a free copy and specify which one.

The median revenue per APAC lawyer was $250 million; the corresponding figure for EMEA was $204 million (€157,142,857 at 1.3 dollars to the Euro). Based on medians, the EMEA legal departments supported less revenue in their region than did their Asian counterparts. Averages told a different story. The APAC average was $274 million whereas the EMEA average was much higher at $317 million (€244,358,702). The difference between median and average figures says that some EMEA-based companies were much larger than the largest group of Asian companies supported much more revenue.

One part of the benchmark reports suggests eight ways that a general counsel might increase revenue per lawyer:

This post is for the mathematically intrepid.

Present skewed data on log scales. My benchmark projects always lead me to think of normalizing data by company revenue: lawyers per billion, total legal spend per billion. From a recent article, however, I realize that perhaps it is possible to normalize data by the logarithm of the revenue (See my post of Jan. 14, 2007: explains log scales and log-log scales.). When data falls along a very wide range, notably revenue, you can cope by using log scales.

Subtract an industry average from a company’s figure. Since profitability (considered as return on equity) varies widely by industry, benchmark researchers can subtract from profitability of each company the average return on equity for all other firms in the same industry. This would turn lawyers per billion, for example, into comparable metrics across industries (See my post of Jan. 12, 2009: divide by the industry average.).

Let’s start with how to create a heat map. Take all the litigation pending against your company and assign each lawsuit a percentage from 1-100 on the likelihood that you will pay some non-trivial amount in settlement or judgment. Then use the same scale of 1-100 to estimate the significance (amount) of that payment. With those two coordinates for each case you can plot them on a scattergram and create a heat map, according to a short item in the Harv. Bus. Rev., Vol. 87, Oct. 2009 at 76.

Put a mark for each case on a chart where the vertical axis shows a case’s significance and the horizontal axis its likelihood. Cases in the lower left — the green portion of the heat map — are some combination of relatively unlikely to result in a payout and even if you do it won’t be all that much. In the yellow zone, a stripe from the upper left corner to the lower right, are the cases of some financial risk and some chance of turning against you. The upper right triangle is red hot, where cases have significant financial implications and a relatively high probability of a bad outcome.

The heat map adds a quickly understood color coding to the more familiar scattergram.

Various posts on this blog discuss metrics and take inflation into account (See my post of April 27, 2006: effective hourly rates of the plaintiffs’ bar; May 31, 2005: growth of civil legal spending by Canadian businesses; Dec. 5, 2007: median revenue of a survey population 15 years later; April 1, 2009: median damages in patent litigation; June 24, 2009: the money illusion; and Sept. 28, 2009: legal fees and inflation over time.).

Almost always, when general counsel present figures that are more than a few years old, they should adjust those historical figures for the distortions of inflation (See my post of Dec. 31, 2006: adjust figures for inflation; and March 12, 2006: nominal and inflation-adjusted figures.). Otherwise, to say something like “we paid rates to our best law firm of $200 an hour in 1995” misleads the casual listener or reader unless you add “which in 2009 dollars is $310 [or whatever the inflation-adjustment says].” Benchmark surveys should also normalize past financial figures to current inflation-adjusted amounts.

According to the Law of Large Numbers, “you can have a high degree of confidence in the average value of a sample if the sample includes a very large number of observations.” As explained in the NY Rev. of Books, Oct. 8, 2009 at 30, therefore, the more legal departments contribute data to a benchmark study, the more you can rely on the resulting metrics (See my post of Feb. 7, 2008: crowdsourcing depends on the Law of Large Numbers.).

The problem arises when surveyors stop way short of a sufficient number of observations but show high confidence in the results anyway. The facetious invocation of the Law of Small Numbers refers to our tendency to put too much faith in too small a slice of data.

Even while I so much desire even shreds of benchmarks, I bark all the time at sample size (See my post of Dec. 9, 2005: margin of error and sample size; Oct. 31, 2007: formula for confidence levels; April 22, 2007: power tests and sample size; March 28, 2005: number of respondents; Dec. 19, 2007: few participants; and April 9, 2005: few respondents from a large invitee pool.). Still, in a data desert, better Small Numbers than no numbers.

Disappointed recently with sloppy benchmark efforts by several purveyors, I realized that the law department world lacks third-party assessments of benchmarking efforts. Journalists from time to time go a round or two with data from benchmark surveys, mostly compensation, but they never lay a glove on methodology. Any law firm, vendor, academic, consultant, or trade group can mush together a grab-bag of data, stir it around, and pronounce “findings.” No one vouches for the appropriateness of procedures, the absence of bias, or the plausibility of findings. No one articulates standards of good benchmarking.

Dubious benchmarks thereby become urban legends of management. Loose and unreliable numbers float around. The data debris that results from poorly-executed benchmarking suggests a variation on Shakespeare: “The evil that bad benchmarks do lives after them; the good is oft interred with their bones.”