Challenge
Performance of AI systems used in high-risk applications has substantial impact on individuals and society. This is the case of autonomous vehicles that must reliably identify individuals and other vehicles in their surroundings. For this reason, being able to assess the trustworthiness of such systems is crucial. Poorly evaluated AI systems can lead to serious consequences. Besides internal evaluations made by the system developers, in some cases, regulation mandates external certification by accredited bodies.
Appropriate assessment of the system's trustworthiness using validated, up-to-date criteria promotes trust, accelerates adoption, and enables use in critical applications. However, assessing AI trustworthiness is not trivial, since trustworthiness is often described in abstract, non-verifiable terms. This creates a challenge for AI builders who want to prove the benefits of their products and brings uncertainty to users on the suitability of AI-based solutions.
Approach
We addressed this gap by developing a framework that connects abstract regulatory requirements and societal expectations on trustworthiness to concrete procedures and methods for evaluating AI systems. In alignment with the EU AI regulation, we define trustworthiness according to specific dimensions and identified ways to define and verify objectives on the reliability, safety, transparency and human oversight required for AI systems.
"We had to overcome several obstacles, like finding concrete standards for very abstract legal requirements and transforming academic findings with limited applicability to practical solutions that work for real-world use cases", reports Dr. Philipp Denzel, one of the senior researchers of the project. "After careful investigations, we had a collection of very specific technical methods that we implemented with a high level of automation".
The framework was implemented in an AI governance platform that acts as a requirements management system that provides an efficient and traceable way for AI developers, adopters and assessors to validate their AI solutions. The platform enables to manage more than 100 trustworthiness objectives and 260 means to evaluate fulfilment of these objectives. This is complemented by an testing platform that allows stress testing AI models using state-of-the-art, validated methods for assessing transparency, safety and robustness.
Outcome
The AI governance and testing platform was successfully applied to assess robustness, safety, and transparency of an AI vision system for autonomous construction vehicles. It provided qualitative and quantitative evaluation of the system trustworthiness by evaluating the AI algorithms' robustness in multiple weather conditions or in case of data quality deterioration.
Apart from direct applications in industry, the project resulted several scientific publications. One of them won the best paper award at the IEEE Swiss Conference on Data Science 2024.
This work was developed by the ZHAW in collaboration with the Certification company CertX, with the support of Innosuisse. The Swiss Centre for Responsible AI continues this work by maintaining and extending the assessment and testing platform to include criteria for data privacy and accountability. They are used by SCRAI to provide third party assessments of AI systems and to support organizations on their journey to Responsible AI.
Do you want to find out what your organization could achieve? Then let's build responsible AI together.
