Law Department Management Blog
In mid-2003, the Legal Technology Institute at the Univ. of Florida’s College of Law collected survey responses from 348 corporate law departments and outside law firms. The article summary mentioned document management and e-mail search applications as examples of KM [www.merrillcorp.com/solutions/lawfirms/article_legalstudy/htm]
The article summary exclaimed that “[o]ne of the more surprising findings was that …63 percent of law department respondents who reported having KM systems in place indicated that they use those systems either ‘frequently’ or ‘all the time.’” (emphasis added). What we don’t know is the number or percentage of departments that reported no KM systems in place.
Another example of self-selection undermining the statistical conclusion followed. “Of the survey respondents who reported a positive ROI from their KM program, 78% of the law departments indicated that their KM program either meets or exceeds their targeted ROI.” (emphasis added)
Let’s cool off a moment. We have no way of knowing the departments’ target ROIs; we are only dealing with the subset that reported a return, not the departments that lost money on their KM investment; and even of the plus-side group – however that was calculated – a quarter of them fell short of their target return on investment.
The metrics and their reporting suffer from acute slanting. [See my post of March 6, 2005 regarding problematic use of metrics.] A critic could read into both findings very pessimistic conclusions. One-third of the departments with “KM systems,” which encompasses such basic applications as document management and an e-mail find function, infrequently or never use them! Then, the ballyhooed ROI figures come only from those reporting a net return, without explanation of methodology, and a quarter fell below their ROI goal.