Articles Posted in Benchmarks

Scientists call artifacts “observations in their research that are produced entirely by some aspect of the method of research,” as explained by Daniel Kahneman, Thinking, Fast and Slow (Farrar, Straus & Giroux 2011) at 110. Artifacts, weeds in the gardens of benchmarks, crop up when how the data is collected or prepared for analysis distorts the findings in some way. This risk of skewed irregularity goes to methodology, not to sample size or analysis (See my post of Feb. 19, 2010: representativeness of survey respondents; May 20, 2010: four methodological bumps; May 25, 2010: effective response rates; and June 13, 2010: example of well-described methodology.).

Many artifacts lurk around benchmark surveys of law departments:

Delivery: it might make a difference whether the survey was mailed, available only online, or asked by telephone or in person (See my post of Feb. 12, 2009: included telephone solicitation.).

In addition to General Counsel Metrics (GC Metrics), three U.S. organizations collect data from law departments and produce benchmark metrics for all to purchase. ACC/Empsight, American Lawyer Media (ALM), and HBR Consulting set submission deadlines in the late summer and produce their reports a month or two later. On that schedule data will not be included during the last quarter of each calendar year.

Perhaps that limited window for data collection skews the results they publish. Does the truncated season of the other three make a material difference in the benchmark numbers they produce? By contrast, GC Metrics welcomes participation into early January of the following year.

An idea, possibly daft, occurred to me regarding disclosures mandated by the SEC. Would that agency conclude that investors would benefit from having comparative data on legal spend? Would that information materially help the equity markets? If so, would it have the power to require listed U.S. companies to state their total legal spending during their fiscal years reported on? Since legal spend amounts to approximately half of a cent for every dollar of revenue, it may equal something like the compensation granted the five most highly paid officers, which must be disclosed. The impetus for that disclosure goes far beyond simple totals. I thought of this as I read Sylvia Nasar, Grand Pursuit: the story of economic genius (Simon & Schuster 2011) at 305.

After all, companies have to describe material legal liabilities. In 2006 the SEC mandated that companies where boards use peer groups to set compensation must disclose the peer firms. So, might the SEC step into law department benchmarks?

In a variation on the tort doctrine of the thin-skull plaintiff, those who care about metrics and measurement have to accept that most people, including in-house lawyers, become befuddled around very large numbers. Our cognitive capabilities didn’t evolve to let us toss off millions and billions with confidence and intuitive understanding. Inside our thin skulls, we don’t deal with large numbers very well.

For more on this limitation, my most recent column for InsideCounsel online explores some ramifications.

One question on a recent Mitratch survey regarding law department technology asked respondents “Do you currently invest in” and listed six classes of law department software. In order of declining percentages, the responses checked off by respondents were matter management systems (72% said they were invested in it), e-billing (52%), governance, risk and compliance (41%), e-discovery (41%), contract management (37%), and entity management (25%).

What seems noteworthy to me from this prioritization finding is the governance, risk and compliance showing. If that class includes online training software for the likes of FCPA, discrimination, antitrust and other common concerns, I can understand the relatively high ranking. In fact, most large corporations have some required training modules. Other than those applications, however, I am at a loss for what kinds of software applications the respondents might have invested in.

The pace of participation has picked up and there is a decent chance that this year’s final report will embrace 1,000 departments. As of today, there were 600. They reported more than 26,000 lawyers, $13.5 billion of legal spend, and $43 trillion of revenue. In addition to 22 basic industries, by the way, the December report will include at least four segments of industries: automotive suppliers, national laboratories, manufacturers who belong to MAPI, and semiconductor companies. More industry segments will appear if there are at least six companies and they will get quartiles and means as greater detail in the report.

This year the report tilts a bit more than last year to English-speaking countries, but still with 155 from Europe. The data on matter management system vendors has enriched steadily and will form the basis of an innovative analytic report late this year to be sent to all participants.

Learn from the thousand. No cost to submit data or get the 75-page PDF report in December. Click for the quick, secure survey and enter your six staffing and spending figures.

About a year ago I used the data from the 2010 General Counsel Metrics benchmark survey to calculate how 20 industries ranked against each other on four key metrics (See my post of Dec. 1, 2011: legal intensity; and Nov. 30, 2010: eight countries.). This year, at the 600 participant mark, I looked again at 17 of those industries (not having had Insurance last year and dropping Not for Profit). Note that about half the participating law departments are new to this year’s survey, which strengthens the conclusions because they greatly broaden the data set.

The fundamental finding involves stability: the first five industries retained exactly the same standings. The least legally intense industries, by which I mean they had overall the lowest lawyers and lowest staff per billion dollars of revenue and the lowest internal and lowest external spending per billion are Retail; Food & Beverage; Manufacturing (that is a surprise, with all the presumed litigation); Extractive, Mining and Chemicals; and Utilities (surprising, with all the regulation). They have 20-30 participants in each industry, although Manufacturing had more than 100.

As last year, the industries with the highest level of legal intensity included Business Services, Financial Services, and Technology. Only Business Services should raise an eyebrow among that group.

To proclaim changes in broad metrics, such as total legal spending from last year to this, a survey needs to have a fairly consistent core of respondents answering similar questions over the period of time. If there is churn – last year 100 took part but this year 50 of them along with 50 new ones – or if the questions and their definitions shift – last year asked about worldwide lawyers and this year about total or domestic lawyers – then year-over-year comparisons of the results are silly.

Hence, touted “continuity” of a benchmark survey may carry no water. Yes, 25 years ago Equitable’s legal department commissioned Price Waterhouse to do a consulting project, from which emerged the distant ancestor of one of today’s staffing and spending surveys. Later, the survey trundled over to Hildebrandt, continued when Thomson Reuters acquired it, and kept plugging away when BakerRobbins merged in. Now, a survey that places great stock in its lineage perches with a fourth company, the fledgling HBR Consulting (not to be confused with the Harvard Business Review – HBR). Continuity of a survey doesn’t matter; continuity of participants does, so findings ought to disclose turnover in the ranks.

Well, thankfully, now you can. My latest InsideCounsel column plunged into the insights you can gain when you understand, calculate, and properly apply the statistical tool called the coefficient of variation.

The calculation, easy to do with Excel, lets you compare the relative dispersion of two sets of numbers that don’t have the same base, such as to compare a set of numbers in dollars to a set in months. Click above to increase the statistical odds of some insightful variation in your life.

No sooner than I published about two impossible-to-reconcile figures on corporate legal spending, and the disappointing lack of support for either figure, I read another (See my post of Oct. 25, 3011 #2: $60 billion global and $200 billion U.S. litigation figures.). In Corp. Counsel, Oct. 2011 at 55, Mark Harris, the CEO of Axiom, starts his article with a reference to the “half-a-trillion dollar global legal services industry.”

Later Harris states the future of the legal industry rests in the wallets of the “largest 5 percent of corporations, who among them control more than half of the $100 billion U.S. corporate legal services market.” He later infers that the “market” includes in-house legal costs. That means he sees the United States as accounting for about 20 percent of the worldwide legal services expenses. I would be surprised if it is that low.

One final number from Harris: “80 percent of the [U.S. corporate legal services] market is controlled by a mere 200 GCs.” If we crudely assume the Fortune 200 to be the biggest spenders, then the smallest company had a bit over $12 billion in revenue while the median was $25 billion – all 500 recorded $10.8 trillion. Guessing that the top 40 percent of the list (200 companies) accounted for 80 percent of the total revenue, and positing that something like one half percent of their revenues went for legal services, that would approximately $45 billion (.8 times $10.8 trillion times .5%). Fairly close to Harris’ estimate of half the U.S. corporate legal services market.