Ranks of states on quality and fairness of judiciary may have been shoddy and unfair

In a previous post, I summarized an assessment by the U.S. Chamber of Commerce of the best and worst state legal systems in America (See my post of May 17, 2006 on the results.). Afterwards, I read a persuasive critique of that assessment, and feel that I should give those points equal play.

Bias toward corporate views. Only in-house counsel at companies with $100 million or more in annual revenues were surveyed. Their impressions of judicial quality and fairness sharply diverge, according to the critic, from the impressions of consumers who believe they have been wronged and seek redress.

Lack of quantitative proof of conclusions. The polling firm that conducted the survey admitted that “there is no hard data that can be used to measure the perceived fairness of a state’s legal system.” Subjective impressions, by a select group, created the rankings.

Many respondents had impressions based on no knowledge. Of the 1,437 lawyers who completed the survey, many had no knowledge of several of the state’s legal systems. For example, around 107 said they had direct knowledge of West Virginia’s judiciary.

I draw three lessons from this rebuttal. Always scrutinize the methodology and vested interests of surveyors. Always assume at least two sides to anything controversial. And don’t suggest applying an index to cases handled by law departments if the index metrics might be biased, purely subjective, and flawed.