Selection bias distorts a statistical analysis because of how the data was collected (See my post of Dec. 1, 2006 for an introduction.). Let’s consider some examples in the context of law departments.
Self-selection bias. Law department lawyers with strong opinions, deep interests, or substantial knowledge may be more willing to spend time answering a survey than those lawyers who don’t have any of those characteristics. If a survey asks about pro bono, for example, those who couldn’t care less will probably not bother to respond (See my post of May 14, 2005 suspecting this on a knowledge management survey; and Aug. 26, 2006 generally.).
Selected end-point bias. For example, to maximise a claimed trend of lower outside counsel costs, a law department could start measuring the drop from an unusually high year. Or a compensation survey draws its data from law departments that are generally much larger or much smaller than the surveying department is suspect.
Selecting non-representative respondents. Surveys of law department IT support staff will typically find more technology presence in departments than surveys of all law department staff (See my post of Aug. 27, 2005 about a survey of IT respondents.). Or a client satisfaction survey that goes out only to clients identified by members of the law department might not canvass the waterfront of opinion.
Partitioning data. All surveys make decisions about where to draw various lines, such as between “small,” “medium,” and “large” law departments; or according to “industry.” Or a law department divides its cases, with premeditation, into “small,” “medium,” and “large.”
Rejecting “bad” data on arbitrary grounds. If a department drops from its total legal spending the costs of one hugely expensive class action – “because it was so anomalous” – selection bias has gnawed away at statistical accuracy. Another instance of bias is where a law department study drops from analysis the cases where ECA was not done or budgets not submitted or status reports not completed.