Calibration by group, error rate parity, sufficiency, and separation
Published:
In the field of algorithmic fairness, it is well known that there are several definitions of fairness that are impossible to reconcile except in (practically irrelevant) corner cases. In this context, I have recently tried to wrap my head around why – intuitively – it is impossible for any classifier to achieve separation and sufficiency at the same time (unless either the classifier is a perfect classifier or there are no base rate differences between groups – we will get to these details in a minute). Since part of my troubles arose from a misunderstanding of what separation and sufficiency actually mean, let us start by revisiting their definitions.