Collective evaluations of law firms will inevitably bend to serve the rating organization’s goals

Moves are afoot to rate law firms (See my post of Dec. 14, 2009: good ideas from an ADR rating system; and Nov. 5, 2009: collective assessments by law departments of their firms with 10 references.). It’s a laudable goal, but naturally there will be some slips twixt cup and lip.

One slip that hasn’t been mentioned is the neutrality of rating classification systems. An excellent article in Admin. Sciences Q’tly, Vol. 54, Dec. 2009 at 555, discusses how US brokerage firms rate and classify producers’ securities. This stock is a “buy,” that one a “hold.” For law firms, for example, classification schemes might be “A 5-star firm, a 4-star firm …,” “a practice area champion,” “Top 25 in the country,” etc.

The main point of the article is that raters use their classification schemes –– to further their own organizational purposes. Not surprisingly, “Although rating systems facilitate exchange between producers and audiences, they also have particular value for rating organizations themselves.” Rather than seeking an objective, transparent formula, the raters alter aspects of their work to serve themselves. No one is disinterested, everyone wants an edge, and as we enter a period where there are multiple systems striving to evaluate the performance of outside counsel, we should recognize that those systems compete and that competition will influence how they rate, whom they rate, and the rating system itself.