What if several matter management systems allowed in-house users to rate their law firms on a handful of common criteria? What if there were an industry standard for what those criteria mean and the scale on which to rate them? What if periodically each law department stripped out any confidential information but shared the evaluations through a third party?
A third party could pool the evaluations, keeping confidential which law department gave which firm a particular rating, and everyone could learn from the consolidated results (See my post of July 21, 2005 on an effort to do something like this on a website.). The logical next step would be evaluations of individual partners.