Try having your responsible in-house lawyers rate the performance of your major law firms, using a scale of 1 (Unacceptable), 2 (Mediocre), 3 (Good), 4 (Very Good) and 5 (Excellent) on six attributes. These come from FMC Technologies and you can view the form on Bruce McEwans thoughtful blog.
Understand goals
Expertise
Efficiency (cost, I assume)
Responsiveness
Predictive accuracy (how well budgets match actuals, I assume)
Effectiveness
It feels to me like “effectiveness” overlaps with knowledge of our business (Understand goals), knows the law (Expertise), and works productively (Efficiency). Otherwise, as attributes that in-house counsel should evaluate, these are solid (See my post of Nov. 16, 2005: evaluations of law firms with 9 references.).