Choosing between Demographic/Statistical Parity and Equal Opportunity comes down to one question: are you trying to prevent unequal allocation of opportunities across groups, or to ensure that qualified individuals are treated equally regardless of group membership? These two group fairness metrics measure fundamentally different things, and picking the wrong one can mask the exact harm you set out to detect. This comparison draws on the Fairlearn project's fairness documentation and a Carpentries Incubator training curriculum on trustworthy machine learning to lay out what each metric captures, where it applies, and where it falls short.
Introduction: Evaluating Bias in Large Language Models
As large language models increasingly influence decisions—screening resumes, approving loans, ranking content—practitioners need quantitative ways to test whether those systems treat demographic groups differently. Fairness metrics provide that test in the context of classification tasks, where a model assigns positive or negative outcomes to individuals. But the metrics themselves encode assumptions about what fairness means, and those assumptions vary. Fairlearn's documentation, a widely used open-source toolkit for fairness assessment, treats demographic parity and equal opportunity as distinct tools suited to distinct harms. The Carpencies Incubator lesson on model evaluation and fairness makes the underlying point bluntly: there is no single definition of "fair machine learning," and different notions of fairness are appropriate in different contexts. Understanding which notion matches your application is the first step in any credible bias evaluation.





