The Problem – phawm

What’s the problem?

AI can benefit people’s work and everyday lives through decision support systems, finding information and AI-generated text or images. Yet, high-profile failures, such as predictive AI models that cast aside job applications by women or generative AI helping lawyers write judicial arguments that ‘hallucinate’ non-existing court cases, are causing serious concerns. A significant barrier to reaping the benefits of predictive and generative AI is their unassessed potential for harms: an AI that is of poor quality in terms of truthfulness, transparency, information security, bias and robustness which may cause unfair outcomes, treatment and wider harms. Hence, AI auditing has become necessary, in line with existing and impending regulation that calls for regular assessments (e.g. US President Biden’s Executive Order and the EU AI Act).

Yet, AI auditing has been haphazard, unsystematic, without common frameworks or standards and left solely in the hands of experts. Our project aims to fix this fundamental challenge through the novel concept of participatory AI auditing, where a diverse set of stakeholders, such as domain experts, regulators, end-users and decision subjects, are empowered to undertake audits of predictive and generative AI, both individually and collectively. To this end, our project will produce workbenches that support these stakeholders in auditing AI, and embed these audits in methodologies that define how, when and who carries out the audits. We will train stakeholders in carrying out audits and work towards a certification framework for AI solutions that use participatory AI auditing.

Our project is situated against a background of research and practice in AI safety testing and AI evaluations and audits. Some tools (e.g. Google’s What-If tool, or IBM’s AI Fairness 360 system) now exist to help predictive AI solutions to be audited by AI experts in terms of accuracy and bias, using fairness measures. Some auditing methodologies for generative AI,
such as red teaming, have been taken up by industry.

However, numerous challenges require participatory auditing by stake holders

Bias vs. Fairness Perception

Bias, as an AI technical concept, does not necessarily lead to unfairness as perceived by stakeholders or in law. Fairness is often only evaluated for protected characteristics, but stakeholders are also sensitive to other, non-protected features and wider issues of equity and justice in society as well as harms going beyond current technical definitions and mitigations. In predictive AI, assessing bias and fairness often relies on technical measures related to ground truth. However, for many domains and applications, ground truth might not be fully aligned with stakeholders’ perspectives of fairness. Further, there are currently no standards of what counts as ‘good enough’ to pass an audit.

Fairness and Performance Trade-offs

There is often a trade-off between measures of performance and fairness, and even within fairness measures. Stakeholders’ notions of what is ‘fair’ or ‘biased’ differ markedly between domains and application contexts. Even within stakeholder groups there is sometimes little consensus of how to define and achieve less biased, fairer outcomes.

Limitations of AI Auditing

We might not have access to the training data nor the model itself so auditing can only be conducted on input and outputs. Even if we have access to the data and model, we might not be able to expose the training data or the model during the audit without revealing sensitive aspects, such as personal data or confidential/privileged information.

Risks of Expert-Only Audits

If we leave audits to AI experts, they might not do this at all or choose measures that are technically easier to define and operationalise instead of fully capturing stakeholders’ perspectives. This could lead to accusations of ‘fairwashing’ when stakeholder involvement is lacking.

Empowering Stakeholders

However, giving stakeholders without AI expertise the power to audit is complex. Responsible AI-related concepts, technologies and approaches need to be communicated in an accessible way. Support needs to be given to stakeholders for auditing, and stakeholders need to be provided with ways to feedback on issues that need to be acted upon after an audit.