Practical significance is a conceptual framework for evaluating discrimination cases developed primarily on statistical evidence that is the subject of increasing interest and discussion by some in the equal employment opportunity (EEO) field.
- What is practical significance?
- How is practical significance related to statistical significance?
- How is practical significance measured?
- Do the Uniform Guidelines on Employee Selection Procedures address practical significance?
- Does OFCCP consider practical significance during a compliance review?
What is practical significance?
In the EEO context, practical significance refers to whether an observed disparity in employment opportunities or outcomes reflects meaningful harm to the disfavored group. The concept focuses on the contextual impact or importance of the disparity rather than its likelihood of occurring by chance.
How is practical significance related to statistical significance?
Since the "importance" of a disparity is influenced by the magnitude of the impact, the notions of practical and statistical significance are related. Statistical significance is a function of multiple factors, including the magnitude of the disparity, the number of observations in the analysis, and the power of the statistical test used. The purpose of a statistical test is to assess the likelihood that random or legitimate, nondiscriminatory factors rather than discriminatory factors produced an observed disparity. Under certain conditions, a virtually unnoticeable disparity in, for instance, selection rates, may nevertheless be statistically significant due to the size of the data set. OFCCP is mindful that enforcement efforts to eliminate small impacts may prove counterproductive due to the difficulty of an alternative practice improving upon an already small effect. OFCCP is also aware that the choice of the appropriate statistical test reduces the likelihood of a finding that lacks practical import.
Conversely, just because a disparity is not statistically significant does not necessarily mean that discrimination did not occur. Discrimination cases need not be founded upon a statistically significant disparity. A disparity that does not meet the usual statistical significance standard may nevertheless be compelling if, for example, there is a glaring selection shortfall supplemented with strong anecdotal evidence, or red flags in the company’s EEO compliance.
How is practical significance measured?
There is no universally accepted measure of practical significance in the EEO field. What is considered practically significant depends on the employment opportunity at issue and the specific facts of the case. There are a variety of measures and techniques discussed in the literature and case law, but no superior or consistently reliable measure has emerged. 1 Some of the measures used in employment discrimination cases include the impact ratio, the odds ratio, the flip-flop rule, the Apsley v. Boeing ratio, the standardized difference between disfavored and favored outcomes, and Cohen’s h.
Do the Uniform Guidelines on Employee Selection Procedures address practical significance?
Yes. The Uniform Guidelines on Employee Selection Procedures (UGESP; 41 C.F.R. Part 60-3) contemplate practical significance in Section 4D. "Smaller differences in selection rates [i.e., not meeting the four-fifths rule], may nevertheless constitute adverse impact, where they are significant in both statistical and practical terms."
In addition, the Adoption of Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures (UGESP Q&A; 44 FR 11996) restates and expands on Section 4D with specific examples.
- Q&A 20 provides two hypothetical examples to illustrate how a final determination of adverse impact may be made contrary to the four-fifths rule based on practical significance considerations.
- In the first example, arrest records disqualified 10% of Hispanic applicants and 4% of non-Hispanic White applicants. A disfavored group disqualification rate 2.5 times that of the favored group was considered "large enough to be practically significant," even though the impact ratio was 0.94.
- In the second example, four hires from an applicant pool of 30 was considered "too small to warrant a determination of adverse impact," even though the impact ratio was 0.67.
- Q&A 21 provides what has been referred to as the "flip-flop" or "N of 1" rule, stating, "Generally, it is inappropriate to require validity evidence or to take enforcement action where the number of persons and the difference in selection rates are so small the selection of one different person for one job would shift the result from adverse impact against one group to a situation in which that group has a higher selection rate than the other group." However, note that even if a disparity were determined not to be practically significant by this rule, adverse impact might still be inferred by OFCCP if it "continued over a period of time, so as to constitute a pattern."
Does OFCCP consider practical significance during a compliance review?
Yes, as part of a holistic evaluation of the review, OFCCP considers practical significance along with statistical significance and all other evidence gathered in the course of the investigation. Depending on the employment issue under review, OFCCP utilizes a variety of practical significance measures discussed herein and in the referenced materials. Furthermore, OFCCP reviews will typically employ a combination of those tests and principles to ensure that the agency is efficiently deploying its resources.
1For an overview of the most common measures of practical significance, see the following references and court rulings: Kaye, D. and Freedman, D. (2011), Reference Guide on Statistics. Washington, D.C.: National Academy of Sciences Press, available at https://www.fjc.gov/content/reference-guide-statistics-2 (last accessed July 22, 2019); Oswald, F.L., Dunleavy, E.M., & Shaw, A. (2017), "Measuring practical significance in adverse impact analysis" in Morris, S.B. & Dunleavy, E.M. (Eds.) Adverse impact analysis: Understanding data, statistics, and risk. New York: Routledge; Gastwirth, J. (2017), "Some recurrent problems in interpreting statistical evidence in equal employment cases" in Law Probability and Risk. Vol. 16, Issue 4, 181-201; Waisome v. Port Auth. of N.Y. & N.J., 948 F.2d 1370, 1376 (2d Cir. 1991); Apsley v. Boeing Co., 691 F.3d 1184, 1200-01 (10th Cir. 2012).