Tensions between values and incentives as an academic medical center strives to enact ethics commitments in AI oversight

Leslie Riddle – University of California, San Francisco; Chanel Matney; Janet Shim; Sara Ackerman – University of California, San Francisco

Abstract: There is a growing call to incorporate ethics principles like trustworthiness and transparency into health AI governance in order to mitigate the potential harms of using AI tools in clinical settings. However, uncertainty remains about how to engage abstract ethics principles in practice and in collaboration with patients and others affected by health AI. Moreover, little is known about the perspectives and experiences of those involved in health AI oversight processes, impeding efforts to identify challenges and promote value alignment between institutional stakeholders and affected groups.

We interviewed members of a health AI oversight committee at an academic medical center, exploring how they define and operationalize ethics principles and view the potential role of patient/community engagement in AI oversight. Their reflections reveal tensions between their commitment to translating AI ethics principles into practice and institutional structures and incentives. For instance, financial incentives for health systems to acquire tools from external vendors, many of which do not disclose algorithm training practices or datasets, suggest that AI researchers may be held to a higher standard of transparency and trust than external vendors. Members agree that patient engagement is valuable, but raise questions about how to attain adequate representation and ensure technical literacy. Further, efforts at engaging patients throughout the AI life cycle are in tension with health systems’ prioritization of rapid technological innovation. Finally, a lack of multidisciplinary collaboration impedes AI developers’ ability to discern and address inequities arising from implementation, highlighting the role of team science in actualizing AI ethics principles.

Keywords: AI ethics and governance, community engagement, academic medical center