Validating AI for Image Analysis

This three-day online workshop provides an introduction to the validation of AI methods in image analysis. Reliable validation is essential to decide whether algorithms have a potential for real-world translation, beyond standard leaderboards. It ensures that model performance estimates are meaningful, reproducible, and reflecting the underlying research questions. The course introduces key principles and practical techniques for validating AI models.

Participants will learn how to select and interpret appropriate performance metrics (day 1), quantify model performance uncertainty (day 2), and assess the robustness of rankings in algorithm benchmarking (day 3).

The workshop combines theoretical lectures with guided hands-on exercises using provided datasets and Jupyter notebooks. We will use open tools such as Metrics Reloaded to map certain problems to suitable metrics, and Rankings Reloaded to assess ranking stability, along with new concepts to assess the probability of false outperformance claims in publications. In addition, short coding tasks will target the implementation and critical assessment of validation workflows.

After completing the workshop, participants will have a thorough understanding of core validation concepts and will be able to apply them in their own AI-based image analysis projects. They will be able to justify metric choices, quantify performance uncertainty, and communicate ranking robustness, which are crucial for publication, regulatory dialogues, and real-world deployment.

Instructors

Annika Reinke (Helmholtz Imaging, DKFZ) & Evangelia Christodoulou (DKFZ)

Learning Goals

The course aims to provide participants with both conceptual understanding and practical skills for reliable validation of AI-based image analysis methods. Each day focuses on a different key aspect of validation, combining theoretical input with hands-on coding exercises.

Day 1: Metrics

• understand the role of performance metrics in validating AI-based image analysis methods
• identify suitable, task-specific metrics and explain their assumptions and limitations
• compute and interpret multiple metrics using open source libraries and toolkits

Day 2: Uncertainty

• describe sources of performance uncertainty in AI model validation;
• compute and interpret performance uncertainty measures such as the standard deviation and confidence intervals

Day 3: Rankings

• analyse and interpret ranking stability in benchmarking studies and assess the probability of false outperformance claims in publications;
• apply best practices for reporting.

Prerequisites

To participate in this course, you need to know

• Basic knowledge of image analysis and machine learning
• Basic Python programming skills

Target Group

• Graduate and PhD students working with and/or interested in AI in image analysis
• Researchers and engineers developing or validating AI-based image analysis models
• Scientists interested in reliable validation and reproducible evaluation practices
• Anyone who wants to gain practical experience with metrics, uncertainty estimation and ranking analysis in AI

Course Days & Times

• Wed, June 3, 2026, 9 am – 12 pm
• Thu, June 9, 2026, 9 am – 12 pm
• Thu, June 25, 2026, 9 am – 12 pm

NOTE: Registration will open May 6, 2026, 12 pm.

Attendance & Certificates

The course content is coordinated, so we strongly recommend that you do not miss any part of the course. To receive a certificate we expect full time and active participation.

Registration

This course is open to individuals affiliated with Helmholtz or a HIDA Partner only.

This course is free of charge.

More information & registration