Law Department Management Blog
Two years ago, you surveyed your 50 key law firms for their satisfaction with you (bet you hadn’t thought of doing that!), and the median rating was 4.7. This year, with the same 50 firms, that rating rose to 4.8, an increase of 2.1 percent. Can you pat yourself on the back, or does the improvement of 0.1 mean nothing statistically?
The margin of error in a survey, which is the amount by which a particular score could be plus or minus the result found because of non-methodological variation, depends on the number of people who respond. (Note: it is critical that respondents be chosen randomly – and take part without any unrepresentative patterns – so that the survey results can be generalized to the whole population.)
How well the sample represents the population is gauged by two important statistics – the survey’s margin of error and confidence level. They tell us how well the sample represents the entire population. For example, a survey may have a margin of error of plus or minus 3 percent at a 95 percent level of confidence. This means that if the survey were conducted 100 times, the data would be within three percentage points above or below the percentage reported in 95 of the 100 surveys.
For law departments, here are some survey response numbers and the margin of error percent at a 95% level of confidence: 300 (6%), 200 (7%), 100 (10%), and 50 (14%). A 95 percent level of confidence is an industry standard. The size of the population (the group being surveyed) does not matter. A few web sites calculate the sample size needed to obtain a specific margin of error at different confidence levels. One free site is available. Note that the margin of error only takes into account sampling error. It does not take into account other potential sources of error such as bias in the questions, bias due to excluding groups who could not be contacted, people refusing to respond or lying (selection bias), or miscounts and miscalculations.
Anyway, with our example having only 50 respondents, the margin of error is 14 percent at a 95% confidence level, much greater than the observed 2 percent shift, so you can’t rely on the improvement as reflecting a real-life change.