Safe AI Sensors – Safety assurance for AI-based driving optimization

You want to use AI to improve your existing product or develop new functionality but are unsure how to ensure and argument its safety? Our approach towards safe AI allows you to upgrade your development process consisting of domain analysis, system design and verification & validation to be applicable for AI-based products. Moreover, it allows you to generate evidence-based safety arguments to demonstrate their safety. Here’s how we applied this for an AI-based sensor in a road vehicle for a road condition recognition use case.

Challenge: Ensuring safety in open environment

Imagine if your car could “hear” the road and make driving decisions. At Fraunhofer IKS, we've helped partners make this a safe reality. We used microphones in a vehicle linked to an AI model to assess road conditions, improving driving behavior. Ensuring safety involves addressing challenges related to AI use, operating in an open environment, and ensuring compliance with laws and regulations.

Solution: Following rigorous iterative development process

A key element in the safe development of AI systems is a rigorous development process. At Fraunhofer IKS, we follow a strict three-step process for all AI projects: domain analysis, system design, and verification & validation. These steps are iterative and adapt to new findings. The outcome is an assurance case summarizing all evidence and arguments for the system's safety.

Key Benefits:

• Enabling Safe AI Sensors: Replace a fixed sensor with a flexible AI-based sensor while safely managing the complex interactions between open environment and the AI algorithm.
• Enhanced Efficiency: Free up safety experts by using reusable safety argumentation templates that can be instantiated with concrete evidence of a project to receive an easy-to-understand documentation.
• Test Effort Reduction: Reduce test effort by defining a systematical validation strategy to attain sufficient scenario coverage with minimal test kilometers and balance between simulation and real-world testing.
• Faster Type Approval: Be first on the market with your AI-based solution following our rigorous development process, achieving compliance with current and upcoming laws and standards.

Insight Background

A key element in the safe development of AI systems is a rigorous development process. At Fraunhofer IKS, we follow a strict three-step process for all AI projects: domain analysis, system design, and verification & validation. These steps are iterative and adapt to new findings. The outcome is an assurance case summarizing all evidence and arguments for the system's safety.

During domain analysis, a systematic process builds a comprehensive understanding of the system by identifying its functional insufficiencies and limitations.

During system development, design patterns like redundancy, monitoring, plausibility checks, and out-of-distribution detection can be integrated to meet defined requirements and address functional insufficiency.

In the final step, the Operational Design Domain (ODD) is divided according to factors influencing the performance of AI components in order to enable statistical verification. A validation strategy defines the balance between simulation and real-world testing, including the number of test kilometers and scenario coverage required, thus helping customers assess their testing needs.

After development, the question of justifying the system's safety arises. Fraunhofer IKS offers reusable templates for safety argumentation, allowing projects to create an easy-to-understand documentation using concrete evidence for AI-based products.

In-depth information

• Safety Assurance of Machine Learning for Chassis Control Functions