ORena SAVE FOCUS Challenge tests VLM’s surgical video understanding

Tracking every surgical object during an operation, and making sure none is left behind in the abdomen, is crucial for patient safety, yet it places a heavy cognitive load on surgeons. To find out whether vision-language models (VLMs) can lighten that cognitive load, the Wellcome Leap SAVE program and the DKFZ Intelligent Medical Systems (IMSY) lab launched the ORena SAVE FOCUS challenge. Thanks to the sponsors – Amazon Web Services, Helmholtz Imaging and Intuitive – the prize pool is currently $60k!

VLMs are becoming progressively better at keeping track of what’s happening over time in long videos, but technical capability alone isn’t enough to bring this technology into the operating room. During surgery, something that happened minutes or even hours earlier can be critical to a safety decision. What’s missing between technological development and clinical practice is a standardised, comprehensive benchmark that can test whether VLMs can be trusted to support safety decisions.

ORena SAVE FOCUS addresses this gap. The challenge tests whether VLMs can understand the order of events and logical connections between earlier and later moments, even across surgical procedures lasting up to hours.

The reliability of FOCUS is ensured through its unique dataset. To our knowledge, it’s the first challenge to provide full-length laparoscopic videos with instance-consistent labels, meaning that objects are tracked across the entire video. That allows testing whether models can follow an object over the entire procedure, count how many times it appears, and correctly confirm if it has been removed from the abdomen before closure.

To consistently probe current limitations of VLMs in long videos analysis, the challenge introduces three tracks that progressively raise the demands on the models:

• FRAME Track evaluates whether models can answer clinically relevant questions from a single laparoscopic image;
• SEGMENT Track tests understanding on videos of up to 5 minutes;
• PROCEDURE Track challenges memory over full-length procedures.

FOCUS is now in its pre-submission phase, running until September 1. Teams can submit models for all three tracks and see their performance on the public leaderboard. To qualify for final submission, models need to beat the baseline.

To learn more about the challenge and participation rules, visit the ORena website.

ORena SAVE FOCUS is a joint initiative of the Wellcome Leap SAVE program and the ORena initiative of the DKFZ Intelligent Medical Systems (IMSY) lab, co-sponsored by Helmholtz Imaging.