How accurately to law department surveys describe law departments (reliability of single respondents)?

Two methodological quirks of surveys came to my attention courtesy of Cal. Mgt. Rev., Vol. 48, Winter 2006 at 32-33. According to research on surveys, two lawyers in the same law department may very well differ on how they characterize its practices, because “when two respondents at the same organization describe the workplace, their responses are only moderately correlated.” So, if a survey has one response from a law department that assesses its leadership, who knows if a second peer would give a decidedly different assessment.

Toss in different views by levels – “when a manager responds about the workplace practices in the organization, that response is only weakly related to the practices that employees report – and accurate portrayal sags further. A general counsel gives one answer to a survey question, such as “To what degree do you reward creativity?” while a lawyer or administrator quite probably gives a different answer.