Kolmogorov-Smirnov test to see whether benchmark respondents are representative

Conscientious benchmark analysts should be concerned about whether their sample of respondents fair represents the entire population. My General Counsel Metrics benchmark survey, to take one example, has nearly 100 members of the Fortune 500 but are they sufficiently similar to the departments that have not responded from the Fortune group to make generalizations?

A statistical tool called the Kolmogorov-Smirnov two-sample test checks for patterns of bias in non-responders compared to respondents. It assesses whether the distribution of respondents is different from that of non-respondents for each of the variables measured with factual data. For example, does my set of Fortune companies have significantly fewer employees than the non-respondents? Are they materially larger or smaller on average in terms of revenue? Does more or less of their revenue come from domestic sales or market-to book value? This statistical test can tell whether there is consistent evidence that sample selection bias threatens the validity of benchmark findings.