On Defining “Vulnerable Groups” In AI Bias Mitigation Solutions

Abstract: Artificial intelligence (AI) bias can be defined as a systematic error that leads to unfair outcomes. Much effort has been made in the algorithmic fairness literature to develop bias mitigation solutions to adjust model performance for different groups. However, defining groups for which models might underperform has been overlooked. Vulnerable groups have traditionally been defined from US-protected classes such as race, ethnicity, and gender. This assumes that the groups are known. This assumption does not always hold in practice, as studies have shown that AI developers struggle to define groups for fairness interventions. Groups can be known only post-deployment when biases occur. Another major limitation is the portability of the group labels from one application or context to another. For instance, some studies have shown that US-based protected classes might be irrelevant or incomplete in India and Africa. This raises the question: How can we define context-sensitive vulnerability for AI fairness interventions? We found that an ethical assessment framework at Stanford Health Care initially developed to identify value collisions among varying stakeholders helped identify not-apparent groups at risk of AI unfairness. For instance, we identified Black athletic patients to be at risk of an AI tool to diagnose rare heart disease and patients with limited English proficiency for an AI tool to reduce hospital readmissions. This participatory AI process opens an avenue to defining AI application-specific populations at risk of unfairness, which I will present.

Keywords: AI Bias, Protected Classes, Stakeholder Engagement