Norman Bradburn, a Senior Fellow at the National Opinion Research Center, explains in an article (Legalaffairs, Nov.-Dec. 2005 at 24) differences between ranking and rating. (See my posts of April 27, 2005 on ranking knowledge management techniques, May 4, 2005 on forced ranking of employees, and Aug. 27, 2005 on ranking constraints to growth.)
Take a law department that decides to rank the top 50 firms it uses on such attributes as cost effectiveness, quality of legal work, responsiveness, diversity, knowledge of the company, inter-personal appeal, and technology prowess. The department’s lawyers assess the firms they have worked with using a score of 1 (low) to 5 (high) on each factor. Adding all the scores for all the firms lets the department rank them, with the highest ranking firm having the most total points.
Four problems crop up. The ranking suggests that the distance, for example, between the top ranked firm and the second is the same as the distance between the second and third ranked firms, when in fact the scores may be further apart or closer together.
The second problem with rankings is that they have a flat distribution – all 50 law firms will be evenly spread from the ranking of the first firm to the ranking of the one hundredth firm. In the real world, the performance of the set of firms probably follows more of a bell-curve distribution.
Third, firms will have different numbers of lawyers assessing them, ranging from one lawyer to many. A ranking list does cannot convey the representativeness of its scores nor the breadth of familiarity within a department of a law firm.
Finally, lawyers differ in their interpretation of the attributes, the importance they attach to a law firm’s performance on an attribute (See my post of Aug. 28, 2005 on importance in client satisfaction surveys.), and how easy or hard to grade. Unlike objective measures, such as the number of timekeepers who served a department in a year or the fees paid the firm, evaluations like these can’t escape subjectivity.
Even with these caveats, I recommend that law departments evaluate at least the law firms that they pay 50 percent of their total external spend.