A recent survey on diversity in law departments, InsideCounsel, Oct. 2006 at 56 et seq., states that “377 in-house attorneys responded to this survey. 19 percent of them were general counsel. 70 percent of respondents identified themselves as white; 14 percent as black; 7 percent as Hispanic; and 7 percent as Asian.” From these statements, ponder four points about surveys.
For a survey to have added credibility, it is important to disclose how many people were invited to complete the survey, and how that population was decided. For example, interpretation of the results differs if all readers of InsideCounsel were sent a survey or if only members of affiliation organizations, such as the Minority Corporate Counsel Association, were urge to complete an online survey.
Second, the article refers repeatedly to the percentage of companies or law departments who responded a certain way, yet the respondents were individuals. Do we know that only one person from any one law department responded?
Third, the assumption behind most surveys is that every respondent’s views hold equal weight and they have reasonably similar experience. With this one, however, the views expressed could be those of the general counsel of a 200 lawyer department or the most-recently hired third-year associate at a four-lawyer department.
Self-selection bias is my fourth observation on methodology. Readers can glean from the information provided that the respondents were disproportionately members of minority groups. Since about 30 percent were non-white, and that is a higher percentage of non-white attorneys than law departments in the United States typically have, some degree of skew is present. Put differently, to speak on attitudes of US law departments as a whole, the survey should have had proportionate representation by each group’s current demographic mix.