A drawback of allocating points instead of ranking

I have advocated allocating points as an evaluation method that serves better than a simple ranking of 1, 2, 3 (See my post of Aug. 14, 2005: better to ask respondents to allocate points; July 4, 2006: sophisticated technique; July 3, 2007: most sophisticated method; July 20, 2008: a downside is that respondents must think more; and Jan. 11, 2009: decisions by teams and weighted preferences.).

To illustrate, if ten law firms have submitted proposals, I thought it was better to have the reviewers of their proposals distribute 100 points among the ten. That way, they could put 50 points on the firm that they thought was by far the best, for example. To merely give that firm 10 points and the next best firm 9 points and so on down does not sufficiently differentiate them.

Quite pleased with myself, I was rocked when someone pointed out that three firms might be equally very good, so the reviewer would have to allocate them 33 points each. If you then total all points allocated by all reviewers to all the firms, those firms won’t fare as well. Hmmmm…..

Steven B. Levy says:

May 3, 2010 at 1:28 pm

I don’t think trying to fine-grain it works in evaluating proposals. What if two voters split 70:30 and 35:65 trying to apportion their votes; does the 105:95 final score really indicate much?
Consider approval voting, where reviewers give a thumbs-up to one or more firms they believe can do the job well. (Defined here, with the usual Wikipedia author-to-author sniping, unfortunately: http://en.wikipedia.org/wiki/Approval_voting.)
Applying mathematical analysis to an RFP is like your family applying it to choosing where you want to go on vacation.
— Steven B. Levy, author of Legal Project Management: Control Costs, Meet Schedules, Manage Risks, and Maintain Sanity