AI Oversight lab
The objective of the AI Oversight Lab is to enable public entities to develop and use AI tools, while adhering to ethical principles and respecting human rights, and to facilitate oversight bodies in their auditing role.
The “AI Oversight Lab” aims to support decentralized and executive public authorities to adopt effective and efficient AI tooling in a trustworthy way (e.g. for inspection, enforcement, supervision and monitoring) as well as central governmental bodies involved in oversight of this AI tooling.
In several use cases proposed by these authorities, the AI Oversight Lab will develop and test AI technology based on the principles of Trustworthy-AI guidelines. TNO is developping this pre-competitive technology to serve public organizations in 5 years from now through co-creation. We are creating a community of practice where public and commercial parties can join, where they can benefit from the Trustworthy AI knowledge, expertise, and insights gained in this project.
Problem description and research approach
AI and Machine learning based tools are more and more available and used both by public and private bodies, influencing decisions and processes in many key sectors of the society. It is becoming therefore extremely important and urgent to develop policies and practices to properly assess and evaluate AI systems under the lens of the Trustworthy AI principles, allowing the developers, the public entities, and the citizenship to answer questions about the fairness, the security, the compliance to privacy rules, and the social and ethical impact of a tool.To tackle such a complex and multi-disciplinary problem, the project is organized in three main lab-rooms, working in strict collaboration:
- Lab Room 1 focuses on the regulatory and policy framework to perform effective auditing and oversight on AI use in the public sector;
- Lab Room 2 is dedicated to identify development best practices and tools to comply with the principle of Trustworthy AI
- Lab Room 3 is aimed to research and develop measures and tools to evaluate and audit AI systems.
Next to these three line of research, the project is working to create a community of practice, where different public and and commercial parties can join (e.g. municipalities or oversight authorities). Concrete problems and use cases are proposed, and all the stakeholders in the community can benefit of the knowledge and solutions developed for specific scenarios.
In the first year, the research conducted in the three lab rooms has been developed next to the concrete use case offered by the municipality of Nissewaard; TNO has conducted an independent evaluation of a commercial ML tool used by the municipality to detect possible abuses of social benefits. This evaluation showed that, despite the fact that the developers clearly put effort to satisfy the principles of Trustworthy AI, the ML algorithm used in the tool was still not robust enough to comply with the European guidelines. Based on this, the municipality of Nissewaard has decided to stop using the tool.
Guided by this specific use case, the research of Lab 2 has focused mainly on identifying guidelines to develop robust and reproducible AI, presented in a handbook. Both Lab 1 and Lab 3 have conducted an extensive review of existing literature, frameworks and tools to evaluate and audit AI, identifying the current state of the art, and future options and challenges. Finally, a first prototype for a tool for (internal) evaluation of AI systems is proposed.
Based on the literature and the experience of different stakeholders, we have developed a first prototype of a visual and interactive auditing tool for AI systems. A list of yes/no questions guides the auditors to analyse the AI system and spot possible problems; where available, information about quantitative measures and tools is provided. At the end of the evaluation a report is generated including the answers, notes and comments of the auditor, a visual representation of the auditing flow; possible weaknesses of the system are also highlighted. At this stage of the development only questions about “Diversity, non-discrimination and fairness” are implemented. For more information contact Ioannis Tolios (email@example.com) or Lucia Tealdi (firstname.lastname@example.org).
The Oversight AI Lab project is a NLAIC project; for more information please visit this link.
- The report from the TNO consultation about a ML decision support tool used by the Nissewaard municipality.
- I. Georgieva , C. Lazo, T. Timan, A. F. van Veenstra. "A reflection and a gap analysis based on recent frameworks and practical experience", under review (available upon request - email@example.com).
- Jok Tang, Sr Consultant, TNO, e-mail: firstname.lastname@example.org.
- Joachim de Greeff, AI consultant, TNO, e-mail: email@example.com