Tim Rädsch
PhD Student
As a PhD student, Tim focuses on making large-scale and high-quality data available in the biomedical domain. While the majority of the general research efforts in the AI area focus on architectures, data topics are often overlooked. The quality of test data in safety-critical applications in the end determines the real-world (e.g. clinical) applicability of an algorithm. High-quality labels are thus absolutely critical for certification in the medical domain and better labels directly translate to patient benefit.
Prior to his PhD studies, Tim worked on ML training data generation in the automotive domain, where he built and led a team of 25 Data Quality Engineers. Furthermore, he is one of the main authors of the ASAM OpenLabel standard, the predominant labeling standard in the European automotive industry.
Publications
4725570
Rädsch
1
https://helmholtz-imaging.de/apa-bold-title.csl
50
date
desc
831
https://helmholtz-imaging.de/wp-content/plugins/zotpress/
%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3Afalse%2C%22meta%22%3A%7B%22request_last%22%3A0%2C%22request_next%22%3A0%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%22YMWVH5AM%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22R%5Cu00e4dsch%20et%20al.%22%2C%22parsedDate%22%3A%222024-07-26%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BR%26%23xE4%3Bdsch%2C%20T.%2C%20Reinke%2C%20A.%2C%20Weru%2C%20V.%2C%20Tizabi%2C%20M.%20D.%2C%20Heller%2C%20N.%2C%20Isensee%2C%20F.%2C%20Kopp-Schneider%2C%20A.%2C%20%26amp%3B%20Maier-Hein%2C%20L.%20%282024%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BQuality%20Assured%3A%20Rethinking%20Annotation%20Strategies%20in%20Imaging%20AI%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2407.17596%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2407.17596%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2407.17596%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Quality%20Assured%3A%20Rethinking%20Annotation%20Strategies%20in%20Imaging%20AI%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vivienn%22%2C%22lastName%22%3A%22Weru%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicholas%22%2C%22lastName%22%3A%22Heller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22This%20paper%20does%20not%20describe%20a%20novel%20method.%20Instead%2C%20it%20studies%20an%20essential%20foundation%20for%20reliable%20benchmarking%20and%20ultimately%20real-world%20application%20of%20AI-based%20image%20analysis%3A%20generating%20high-quality%20reference%20annotations.%20Previous%20research%20has%20focused%20on%20crowdsourcing%20as%20a%20means%20of%20outsourcing%20annotations.%20However%2C%20little%20attention%20has%20so%20far%20been%20given%20to%20annotation%20companies%2C%20specifically%20regarding%20their%20internal%20quality%20assurance%20%28QA%29%20processes.%20Therefore%2C%20our%20aim%20is%20to%20evaluate%20the%20influence%20of%20QA%20employed%20by%20annotation%20companies%20on%20annotation%20quality%20and%20devise%20methodologies%20for%20maximizing%20data%20annotation%20efficacy.%20Based%20on%20a%20total%20of%2057%2C648%20instance%20segmented%20images%20obtained%20from%20a%20total%20of%20924%20annotators%20and%2034%20QA%20workers%20from%20four%20annotation%20companies%20and%20Amazon%20Mechanical%20Turk%20%28MTurk%29%2C%20we%20derived%20the%20following%20insights%3A%20%281%29%20Annotation%20companies%20perform%20better%20both%20in%20terms%20of%20quantity%20and%20quality%20compared%20to%20the%20widely%20used%20platform%20MTurk.%20%282%29%20Annotation%20companies%26%23039%3B%20internal%20QA%20only%20provides%20marginal%20improvements%2C%20if%20any.%20However%2C%20improving%20labeling%20instructions%20instead%20of%20investing%20in%20QA%20can%20substantially%20boost%20annotation%20performance.%20%283%29%20The%20benefit%20of%20internal%20QA%20depends%20on%20specific%20image%20characteristics.%20Our%20work%20could%20enable%20researchers%20to%20derive%20substantially%20more%20value%20from%20a%20fixed%20annotation%20budget%20and%20change%20the%20way%20annotation%20companies%20conduct%20internal%20QA.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2407.17596%22%2C%22date%22%3A%222024-07-26%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2407.17596%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2407.17596%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-02-03T13%3A07%3A10Z%22%7D%7D%2C%7B%22key%22%3A%225ZQ8WVJ8%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Reinke%20et%20al.%22%2C%22parsedDate%22%3A%222024-02-12%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BReinke%2C%20A.%2C%20Tizabi%2C%20M.%20D.%2C%20Baumgartner%2C%20M.%2C%20Eisenmann%2C%20M.%2C%20Heckmann-N%26%23xF6%3Btzel%2C%20D.%2C%20Kavur%2C%20A.%20E.%2C%20R%26%23xE4%3Bdsch%2C%20T.%2C%20Sudre%2C%20C.%20H.%2C%20Acion%2C%20L.%2C%20Antonelli%2C%20M.%2C%20Arbel%2C%20T.%2C%20Bakas%2C%20S.%2C%20Benis%2C%20A.%2C%20Buettner%2C%20F.%2C%20Cardoso%2C%20M.%20J.%2C%20Cheplygina%2C%20V.%2C%20Chen%2C%20J.%2C%20Christodoulou%2C%20E.%2C%20Cimini%2C%20B.%20A.%2C%20%26%23x2026%3B%20Maier-Hein%2C%20L.%20%282024%29.%20%26lt%3Bb%26gt%3BUnderstanding%20metric-related%20pitfalls%20in%20image%20analysis%20validation%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BNature%20Methods%26lt%3B%5C%2Fi%26gt%3B%2C%201%26%23x2013%3B13.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41592-023-02150-0%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41592-023-02150-0%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Understanding%20metric-related%20pitfalls%20in%20image%20analysis%20validation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Baumgartner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Doreen%22%2C%22lastName%22%3A%22Heckmann-N%5Cu00f6tzel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22A.%20Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%20H.%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Acion%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michela%22%2C%22lastName%22%3A%22Antonelli%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tal%22%2C%22lastName%22%3A%22Arbel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arriel%22%2C%22lastName%22%3A%22Benis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Buettner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Jorge%22%2C%22lastName%22%3A%22Cardoso%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jianxu%22%2C%22lastName%22%3A%22Chen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beth%20A.%22%2C%22lastName%22%3A%22Cimini%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Keyvan%22%2C%22lastName%22%3A%22Farahani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Luciana%22%2C%22lastName%22%3A%22Ferrer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrian%22%2C%22lastName%22%3A%22Galdran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22van%20Ginneken%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Glocker%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%20A.%22%2C%22lastName%22%3A%22Hashimoto%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20M.%22%2C%22lastName%22%3A%22Hoffman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Merel%22%2C%22lastName%22%3A%22Huisman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Jannin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Charles%20E.%22%2C%22lastName%22%3A%22Kahn%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dagmar%22%2C%22lastName%22%3A%22Kainmueller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bernhard%22%2C%22lastName%22%3A%22Kainz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexandros%22%2C%22lastName%22%3A%22Karargyris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Kleesiek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Kofler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thijs%22%2C%22lastName%22%3A%22Kooi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%22%2C%22lastName%22%3A%22Kreshuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tahsin%22%2C%22lastName%22%3A%22Kurc%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%20A.%22%2C%22lastName%22%3A%22Landman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Geert%22%2C%22lastName%22%3A%22Litjens%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amin%22%2C%22lastName%22%3A%22Madani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anne%20L.%22%2C%22lastName%22%3A%22Martel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Erik%22%2C%22lastName%22%3A%22Meijering%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karel%20G.%20M.%22%2C%22lastName%22%3A%22Moons%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Henning%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brennan%22%2C%22lastName%22%3A%22Nichyporuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Nickel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Petersen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Susanne%20M.%22%2C%22lastName%22%3A%22Rafelski%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nasir%22%2C%22lastName%22%3A%22Rajpoot%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mauricio%22%2C%22lastName%22%3A%22Reyes%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20A.%22%2C%22lastName%22%3A%22Riegler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicola%22%2C%22lastName%22%3A%22Rieke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julio%22%2C%22lastName%22%3A%22Saez-Rodriguez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Clara%20I.%22%2C%22lastName%22%3A%22S%5Cu00e1nchez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shravya%22%2C%22lastName%22%3A%22Shetty%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ronald%20M.%22%2C%22lastName%22%3A%22Summers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdel%20A.%22%2C%22lastName%22%3A%22Taha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Aleksei%22%2C%22lastName%22%3A%22Tiulpin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sotirios%20A.%22%2C%22lastName%22%3A%22Tsaftaris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Van%20Calster%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ga%5Cu00ebl%22%2C%22lastName%22%3A%22Varoquaux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ziv%20R.%22%2C%22lastName%22%3A%22Yaniv%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%20F.%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Validation%20metrics%20are%20key%20for%20tracking%20scientific%20progress%20and%20bridging%20the%20current%20chasm%20between%20artificial%20intelligence%20research%20and%20its%20translation%20into%20practice.%20However%2C%20increasing%20evidence%20shows%20that%2C%20particularly%20in%20image%20analysis%2C%20metrics%20are%20often%20chosen%20inadequately.%20Although%20taking%20into%20account%20the%20individual%20strengths%2C%20weaknesses%20and%20limitations%20of%20validation%20metrics%20is%20a%20critical%20prerequisite%20to%20making%20educated%20choices%2C%20the%20relevant%20knowledge%20is%20currently%20scattered%20and%20poorly%20accessible%20to%20individual%20researchers.%20Based%20on%20a%20multistage%20Delphi%20process%20conducted%20by%20a%20multidisciplinary%20expert%20consortium%20as%20well%20as%20extensive%20community%20feedback%2C%20the%20present%20work%20provides%20a%20reliable%20and%20comprehensive%20common%20point%20of%20access%20to%20information%20on%20pitfalls%20related%20to%20validation%20metrics%20in%20image%20analysis.%20Although%20focused%20on%20biomedical%20image%20analysis%2C%20the%20addressed%20pitfalls%20generalize%20across%20application%20domains%20and%20are%20categorized%20according%20to%20a%20newly%20created%2C%20domain-agnostic%20taxonomy.%20The%20work%20serves%20to%20enhance%20global%20comprehension%20of%20a%20key%20topic%20in%20image%20analysis%20validation.%22%2C%22date%22%3A%222024-02-12%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1038%5C%2Fs41592-023-02150-0%22%2C%22ISSN%22%3A%221548-7105%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.nature.com%5C%2Farticles%5C%2Fs41592-023-02150-0%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-02-03T08%3A36%3A30Z%22%7D%7D%2C%7B%22key%22%3A%22CAUD4GCC%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Maier-Hein%20et%20al.%22%2C%22parsedDate%22%3A%222024-02%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BMaier-Hein%2C%20L.%2C%20Reinke%2C%20A.%2C%20Godau%2C%20P.%2C%20Tizabi%2C%20M.%20D.%2C%20Buettner%2C%20F.%2C%20Christodoulou%2C%20E.%2C%20Glocker%2C%20B.%2C%20Isensee%2C%20F.%2C%20Kleesiek%2C%20J.%2C%20Kozubek%2C%20M.%2C%20Reyes%2C%20M.%2C%20Riegler%2C%20M.%20A.%2C%20Wiesenfarth%2C%20M.%2C%20Kavur%2C%20A.%20E.%2C%20Sudre%2C%20C.%20H.%2C%20Baumgartner%2C%20M.%2C%20Eisenmann%2C%20M.%2C%20Heckmann-N%26%23xF6%3Btzel%2C%20D.%2C%20R%26%23xE4%3Bdsch%2C%20T.%2C%20%26%23x2026%3B%20J%26%23xE4%3Bger%2C%20P.%20F.%20%282024%29.%20%26lt%3Bb%26gt%3BMetrics%20reloaded%3A%20recommendations%20for%20image%20analysis%20validation%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BNature%20Methods%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B21%26lt%3B%5C%2Fi%26gt%3B%282%29%2C%20195%26%23x2013%3B212.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41592-023-02151-z%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41592-023-02151-z%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Metrics%20reloaded%3A%20recommendations%20for%20image%20analysis%20validation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Buettner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Glocker%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Kleesiek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mauricio%22%2C%22lastName%22%3A%22Reyes%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20A.%22%2C%22lastName%22%3A%22Riegler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuel%22%2C%22lastName%22%3A%22Wiesenfarth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22A.%20Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%20H.%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Baumgartner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Doreen%22%2C%22lastName%22%3A%22Heckmann-N%5Cu00f6tzel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Acion%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michela%22%2C%22lastName%22%3A%22Antonelli%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tal%22%2C%22lastName%22%3A%22Arbel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arriel%22%2C%22lastName%22%3A%22Benis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthew%20B.%22%2C%22lastName%22%3A%22Blaschko%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Jorge%22%2C%22lastName%22%3A%22Cardoso%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beth%20A.%22%2C%22lastName%22%3A%22Cimini%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gary%20S.%22%2C%22lastName%22%3A%22Collins%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Keyvan%22%2C%22lastName%22%3A%22Farahani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Luciana%22%2C%22lastName%22%3A%22Ferrer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrian%22%2C%22lastName%22%3A%22Galdran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22van%20Ginneken%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Robert%22%2C%22lastName%22%3A%22Haase%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%20A.%22%2C%22lastName%22%3A%22Hashimoto%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20M.%22%2C%22lastName%22%3A%22Hoffman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Merel%22%2C%22lastName%22%3A%22Huisman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Jannin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Charles%20E.%22%2C%22lastName%22%3A%22Kahn%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dagmar%22%2C%22lastName%22%3A%22Kainmueller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bernhard%22%2C%22lastName%22%3A%22Kainz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexandros%22%2C%22lastName%22%3A%22Karargyris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alan%22%2C%22lastName%22%3A%22Karthikesalingam%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Kofler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%22%2C%22lastName%22%3A%22Kreshuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tahsin%22%2C%22lastName%22%3A%22Kurc%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%20A.%22%2C%22lastName%22%3A%22Landman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Geert%22%2C%22lastName%22%3A%22Litjens%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amin%22%2C%22lastName%22%3A%22Madani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anne%20L.%22%2C%22lastName%22%3A%22Martel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Mattson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Erik%22%2C%22lastName%22%3A%22Meijering%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karel%20G.%20M.%22%2C%22lastName%22%3A%22Moons%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Henning%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brennan%22%2C%22lastName%22%3A%22Nichyporuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Nickel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Petersen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nasir%22%2C%22lastName%22%3A%22Rajpoot%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicola%22%2C%22lastName%22%3A%22Rieke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julio%22%2C%22lastName%22%3A%22Saez-Rodriguez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Clara%20I.%22%2C%22lastName%22%3A%22S%5Cu00e1nchez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shravya%22%2C%22lastName%22%3A%22Shetty%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maarten%22%2C%22lastName%22%3A%22van%20Smeden%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ronald%20M.%22%2C%22lastName%22%3A%22Summers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdel%20A.%22%2C%22lastName%22%3A%22Taha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Aleksei%22%2C%22lastName%22%3A%22Tiulpin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sotirios%20A.%22%2C%22lastName%22%3A%22Tsaftaris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Van%20Calster%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ga%5Cu00ebl%22%2C%22lastName%22%3A%22Varoquaux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%20F.%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%5D%2C%22abstractNote%22%3A%22Increasing%20evidence%20shows%20that%20flaws%20in%20machine%20learning%20%28ML%29%20algorithm%20validation%20are%20an%20underestimated%20global%20problem.%20In%20biomedical%20image%20analysis%2C%20chosen%20performance%20metrics%20often%20do%20not%20reflect%20the%20domain%20interest%2C%20and%20thus%20fail%20to%20adequately%20measure%20scientific%20progress%20and%20hinder%20translation%20of%20ML%20techniques%20into%20practice.%20To%20overcome%20this%2C%20we%20created%20Metrics%20Reloaded%2C%20a%20comprehensive%20framework%20guiding%20researchers%20in%20the%20problem-aware%20selection%20of%20metrics.%20Developed%20by%20a%20large%20international%20consortium%20in%20a%20multistage%20Delphi%20process%2C%20it%20is%20based%20on%20the%20novel%20concept%20of%20a%20problem%20fingerprint%5Cu2014a%20structured%20representation%20of%20the%20given%20problem%20that%20captures%20all%20aspects%20that%20are%20relevant%20for%20metric%20selection%2C%20from%20the%20domain%20interest%20to%20the%20properties%20of%20the%20target%20structure%28s%29%2C%20dataset%20and%20algorithm%20output.%20On%20the%20basis%20of%20the%20problem%20fingerprint%2C%20users%20are%20guided%20through%20the%20process%20of%20choosing%20and%20applying%20appropriate%20validation%20metrics%20while%20being%20made%20aware%20of%20potential%20pitfalls.%20Metrics%20Reloaded%20targets%20image%20analysis%20problems%20that%20can%20be%20interpreted%20as%20classification%20tasks%20at%20image%2C%20object%20or%20pixel%20level%2C%20namely%20image-level%20classification%2C%20object%20detection%2C%20semantic%20segmentation%20and%20instance%20segmentation%20tasks.%20To%20improve%20the%20user%20experience%2C%20we%20implemented%20the%20framework%20in%20the%20Metrics%20Reloaded%20online%20tool.%20Following%20the%20convergence%20of%20ML%20methodology%20across%20application%20domains%2C%20Metrics%20Reloaded%20fosters%20the%20convergence%20of%20validation%20methodology.%20Its%20applicability%20is%20demonstrated%20for%20various%20biomedical%20use%20cases.%22%2C%22date%22%3A%222024-02%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1038%5C%2Fs41592-023-02151-z%22%2C%22ISSN%22%3A%221548-7105%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.nature.com%5C%2Farticles%5C%2Fs41592-023-02151-z%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-02-03T08%3A34%3A55Z%22%7D%7D%2C%7B%22key%22%3A%22IATVMD87%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Eisenmann%20et%20al.%22%2C%22parsedDate%22%3A%222023-03-30%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BEisenmann%2C%20M.%2C%20Reinke%2C%20A.%2C%20Weru%2C%20V.%2C%20Tizabi%2C%20M.%20D.%2C%20Isensee%2C%20F.%2C%20Adler%2C%20T.%20J.%2C%20Ali%2C%20S.%2C%20Andrearczyk%2C%20V.%2C%20Aubreville%2C%20M.%2C%20Baid%2C%20U.%2C%20Bakas%2C%20S.%2C%20Balu%2C%20N.%2C%20Bano%2C%20S.%2C%20Bernal%2C%20J.%2C%20Bodenstedt%2C%20S.%2C%20Casella%2C%20A.%2C%20Cheplygina%2C%20V.%2C%20Daum%2C%20M.%2C%20de%20Bruijne%2C%20M.%2C%20%26%23x2026%3B%20Maier-Hein%2C%20L.%20%282023%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BWhy%20is%20the%20winner%20the%20best%3F%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2303.17719%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2303.17719%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2303.17719%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Why%20is%20the%20winner%20the%20best%3F%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vivienn%22%2C%22lastName%22%3A%22Weru%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20Dietlinde%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%20J.%22%2C%22lastName%22%3A%22Adler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sharib%22%2C%22lastName%22%3A%22Ali%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vincent%22%2C%22lastName%22%3A%22Andrearczyk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Aubreville%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ujjwal%22%2C%22lastName%22%3A%22Baid%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Niranjan%22%2C%22lastName%22%3A%22Balu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sophia%22%2C%22lastName%22%3A%22Bano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jorge%22%2C%22lastName%22%3A%22Bernal%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sebastian%22%2C%22lastName%22%3A%22Bodenstedt%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alessandro%22%2C%22lastName%22%3A%22Casella%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marie%22%2C%22lastName%22%3A%22Daum%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marleen%22%2C%22lastName%22%3A%22de%20Bruijne%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrien%22%2C%22lastName%22%3A%22Depeursinge%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Reuben%22%2C%22lastName%22%3A%22Dorent%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jan%22%2C%22lastName%22%3A%22Egger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%20G.%22%2C%22lastName%22%3A%22Ellis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sandy%22%2C%22lastName%22%3A%22Engelhardt%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Melanie%22%2C%22lastName%22%3A%22Ganz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Noha%22%2C%22lastName%22%3A%22Ghatwary%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gabriel%22%2C%22lastName%22%3A%22Girard%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anubha%22%2C%22lastName%22%3A%22Gupta%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lasse%22%2C%22lastName%22%3A%22Hansen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kanako%22%2C%22lastName%22%3A%22Harada%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mattias%22%2C%22lastName%22%3A%22Heinrich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicholas%22%2C%22lastName%22%3A%22Heller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alessa%22%2C%22lastName%22%3A%22Hering%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arnaud%22%2C%22lastName%22%3A%22Huaulm%5Cu00e9%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Jannin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ali%20Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Old%5Cu0159ich%22%2C%22lastName%22%3A%22Kodym%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jianning%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hongwei%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jun%22%2C%22lastName%22%3A%22Ma%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carlos%22%2C%22lastName%22%3A%22Mart%5Cu00edn-Isla%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alison%22%2C%22lastName%22%3A%22Noble%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Valentin%22%2C%22lastName%22%3A%22Oreiller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicolas%22%2C%22lastName%22%3A%22Padoy%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sarthak%22%2C%22lastName%22%3A%22Pati%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kelly%22%2C%22lastName%22%3A%22Payette%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jonathan%22%2C%22lastName%22%3A%22Rafael-Pati%5Cu00f1o%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vivek%20Singh%22%2C%22lastName%22%3A%22Bawa%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stefanie%22%2C%22lastName%22%3A%22Speidel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%20H.%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kimberlin%22%2C%22lastName%22%3A%22van%20Wijnen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Wagner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Donglai%22%2C%22lastName%22%3A%22Wei%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amine%22%2C%22lastName%22%3A%22Yamlahi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Moi%20Hoon%22%2C%22lastName%22%3A%22Yap%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chun%22%2C%22lastName%22%3A%22Yuan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maximilian%22%2C%22lastName%22%3A%22Zenk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Aneeq%22%2C%22lastName%22%3A%22Zia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Zimmerer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dogu%20Baran%22%2C%22lastName%22%3A%22Aydogan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Binod%22%2C%22lastName%22%3A%22Bhattarai%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Louise%22%2C%22lastName%22%3A%22Bloch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Raphael%22%2C%22lastName%22%3A%22Br%5Cu00fcngel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jihoon%22%2C%22lastName%22%3A%22Cho%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chanyeol%22%2C%22lastName%22%3A%22Choi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Qi%22%2C%22lastName%22%3A%22Dou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ivan%22%2C%22lastName%22%3A%22Ezhov%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Christoph%20M.%22%2C%22lastName%22%3A%22Friedrich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Clifton%22%2C%22lastName%22%3A%22Fuller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rebati%20Raman%22%2C%22lastName%22%3A%22Gaire%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrian%22%2C%22lastName%22%3A%22Galdran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22%5Cu00c1lvaro%20Garc%5Cu00eda%22%2C%22lastName%22%3A%22Faura%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maria%22%2C%22lastName%22%3A%22Grammatikopoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22SeulGi%22%2C%22lastName%22%3A%22Hong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mostafa%22%2C%22lastName%22%3A%22Jahanifar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ikbeom%22%2C%22lastName%22%3A%22Jang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdolrahim%22%2C%22lastName%22%3A%22Kadkhodamohammadi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Inha%22%2C%22lastName%22%3A%22Kang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Kofler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Satoshi%22%2C%22lastName%22%3A%22Kondo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hugo%22%2C%22lastName%22%3A%22Kuijf%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mingxing%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minh%20Huan%22%2C%22lastName%22%3A%22Luu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Toma%5Cu017e%22%2C%22lastName%22%3A%22Martin%5Cu010di%5Cu010d%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pedro%22%2C%22lastName%22%3A%22Morais%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mohamed%20A.%22%2C%22lastName%22%3A%22Naser%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bruno%22%2C%22lastName%22%3A%22Oliveira%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Owen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Subeen%22%2C%22lastName%22%3A%22Pang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jinah%22%2C%22lastName%22%3A%22Park%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sung-Hong%22%2C%22lastName%22%3A%22Park%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Szymon%22%2C%22lastName%22%3A%22P%5Cu0142otka%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Elodie%22%2C%22lastName%22%3A%22Puybareau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nasir%22%2C%22lastName%22%3A%22Rajpoot%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kanghyun%22%2C%22lastName%22%3A%22Ryu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Numan%22%2C%22lastName%22%3A%22Saeed%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adam%22%2C%22lastName%22%3A%22Shephard%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pengcheng%22%2C%22lastName%22%3A%22Shi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dejan%22%2C%22lastName%22%3A%22%5Cu0160tepec%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ronast%22%2C%22lastName%22%3A%22Subedi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillaume%22%2C%22lastName%22%3A%22Tochon%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Helena%20R.%22%2C%22lastName%22%3A%22Torres%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Helene%22%2C%22lastName%22%3A%22Urien%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jo%5Cu00e3o%20L.%22%2C%22lastName%22%3A%22Vila%5Cu00e7a%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kareem%20Abdul%22%2C%22lastName%22%3A%22Wahid%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Haojie%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jiacheng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Liansheng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Xiyue%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Benedikt%22%2C%22lastName%22%3A%22Wiestler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marek%22%2C%22lastName%22%3A%22Wodzinski%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fangfang%22%2C%22lastName%22%3A%22Xia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Juanying%22%2C%22lastName%22%3A%22Xie%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhiwei%22%2C%22lastName%22%3A%22Xiong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sen%22%2C%22lastName%22%3A%22Yang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yanwu%22%2C%22lastName%22%3A%22Yang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zixuan%22%2C%22lastName%22%3A%22Zhao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%20F.%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22International%20benchmarking%20competitions%20have%20become%20fundamental%20for%20the%20comparative%20performance%20assessment%20of%20image%20analysis%20methods.%20However%2C%20little%20attention%20has%20been%20given%20to%20investigating%20what%20can%20be%20learnt%20from%20these%20competitions.%20Do%20they%20really%20generate%20scientific%20progress%3F%20What%20are%20common%20and%20successful%20participation%20strategies%3F%20What%20makes%20a%20solution%20superior%20to%20a%20competing%20method%3F%20To%20address%20this%20gap%20in%20the%20literature%2C%20we%20performed%20a%20multi-center%20study%20with%20all%2080%20competitions%20that%20were%20conducted%20in%20the%20scope%20of%20IEEE%20ISBI%202021%20and%20MICCAI%202021.%20Statistical%20analyses%20performed%20based%20on%20comprehensive%20descriptions%20of%20the%20submitted%20algorithms%20linked%20to%20their%20rank%20as%20well%20as%20the%20underlying%20participation%20strategies%20revealed%20common%20characteristics%20of%20winning%20solutions.%20These%20typically%20include%20the%20use%20of%20multi-task%20learning%20%2863%25%29%20and%5C%2For%20multi-stage%20pipelines%20%2861%25%29%2C%20and%20a%20focus%20on%20augmentation%20%28100%25%29%2C%20image%20preprocessing%20%2897%25%29%2C%20data%20curation%20%2879%25%29%2C%20and%20postprocessing%20%2866%25%29.%20The%20%26quot%3Btypical%26quot%3B%20lead%20of%20a%20winning%20team%20is%20a%20computer%20scientist%20with%20a%20doctoral%20degree%2C%20five%20years%20of%20experience%20in%20biomedical%20image%20analysis%2C%20and%20four%20years%20of%20experience%20in%20deep%20learning.%20Two%20core%20general%20development%20strategies%20stood%20out%20for%20highly-ranked%20teams%3A%20the%20reflection%20of%20the%20metrics%20in%20the%20method%20design%20and%20the%20focus%20on%20analyzing%20and%20handling%20failure%20cases.%20According%20to%20the%20organizers%2C%2043%25%20of%20the%20winning%20algorithms%20exceeded%20the%20state%20of%20the%20art%20but%20only%2011%25%20completely%20solved%20the%20respective%20domain%20problem.%20The%20insights%20of%20our%20study%20could%20help%20researchers%20%281%29%20improve%20algorithm%20development%20strategies%20when%20approaching%20new%20problems%2C%20and%20%282%29%20focus%20on%20open%20research%20questions%20revealed%20by%20this%20work.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2303.17719%22%2C%22date%22%3A%222023-03-30%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2303.17719%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2303.17719%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222023-04-03T11%3A36%3A59Z%22%7D%7D%2C%7B%22key%22%3A%22Z2JQTA45%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22lastModifiedByUser%22%3A%7B%22id%22%3A9737705%2C%22username%22%3A%22Niebelsa%22%2C%22name%22%3A%22%22%2C%22links%22%3A%7B%22alternate%22%3A%7B%22href%22%3A%22https%3A%5C%2F%5C%2Fwww.zotero.org%5C%2Fniebelsa%22%2C%22type%22%3A%22text%5C%2Fhtml%22%7D%7D%7D%2C%22creatorSummary%22%3A%22R%5Cu00e4dsch%20et%20al.%22%2C%22parsedDate%22%3A%222023-03-02%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BR%26%23xE4%3Bdsch%2C%20T.%2C%20Reinke%2C%20A.%2C%20Weru%2C%20V.%2C%20Tizabi%2C%20M.%20D.%2C%20Schreck%2C%20N.%2C%20Kavur%2C%20A.%20E.%2C%20Pekdemir%2C%20B.%2C%20Ro%26%23xDF%3B%2C%20T.%2C%20Kopp-Schneider%2C%20A.%2C%20%26amp%3B%20Maier-Hein%2C%20L.%20%282023%29.%20%26lt%3Bb%26gt%3BLabelling%20instructions%20matter%20in%20biomedical%20image%20analysis%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BNature%20Machine%20Intelligence%26lt%3B%5C%2Fi%26gt%3B%2C%201%26%23x2013%3B11.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs42256-023-00625-5%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs42256-023-00625-5%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Labelling%20instructions%20matter%20in%20biomedical%20image%20analysis%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vivienn%22%2C%22lastName%22%3A%22Weru%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicholas%22%2C%22lastName%22%3A%22Schreck%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22A.%20Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22B%5Cu00fcnyamin%22%2C%22lastName%22%3A%22Pekdemir%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tobias%22%2C%22lastName%22%3A%22Ro%5Cu00df%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Biomedical%20image%20analysis%20algorithm%20validation%20depends%20on%20high-quality%20annotation%20of%20reference%20datasets%2C%20for%20which%20labelling%20instructions%20are%20key.%20Despite%20their%20importance%2C%20their%20optimization%20remains%20largely%20unexplored.%20Here%20we%20present%20a%20systematic%20study%20of%20labelling%20instructions%20and%20their%20impact%20on%20annotation%20quality%20in%20the%20field.%20Through%20comprehensive%20examination%20of%20professional%20practice%20and%20international%20competitions%20registered%20at%20the%20Medical%20Image%20Computing%20and%20Computer%20Assisted%20Intervention%20Society%2C%20the%20largest%20international%20society%20in%20the%20biomedical%20imaging%20field%2C%20we%20uncovered%20a%20discrepancy%20between%20annotators%5Cu2019%20needs%20for%20labelling%20instructions%20and%20their%20current%20quality%20and%20availability.%20On%20the%20basis%20of%20an%20analysis%20of%2014%2C040%20images%20annotated%20by%20156%20annotators%20from%20four%20professional%20annotation%20companies%20and%20708%20Amazon%20Mechanical%20Turk%20crowdworkers%20using%20instructions%20with%20different%20information%20density%20levels%2C%20we%20further%20found%20that%20including%20exemplary%20images%20substantially%20boosts%20annotation%20performance%20compared%20with%20text-only%20descriptions%2C%20while%20solely%20extending%20text%20descriptions%20does%20not.%20Finally%2C%20professional%20annotators%20constantly%20outperform%20Amazon%20Mechanical%20Turk%20crowdworkers.%20Our%20study%20raises%20awareness%20for%20the%20need%20of%20quality%20standards%20in%20biomedical%20image%20analysis%20labelling%20instructions.%22%2C%22date%22%3A%222023-03-02%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1038%5C%2Fs42256-023-00625-5%22%2C%22ISSN%22%3A%222522-5839%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.nature.com%5C%2Farticles%5C%2Fs42256-023-00625-5%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-03-04T12%3A48%3A11Z%22%7D%7D%2C%7B%22key%22%3A%223HFXLPJW%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Maier-Hein%20et%20al.%22%2C%22parsedDate%22%3A%222022-09-15%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BMaier-Hein%2C%20L.%2C%20Reinke%2C%20A.%2C%20Godau%2C%20P.%2C%20Tizabi%2C%20M.%20D.%2C%20Christodoulou%2C%20E.%2C%20Glocker%2C%20B.%2C%20Isensee%2C%20F.%2C%20Kleesiek%2C%20J.%2C%20Kozubek%2C%20M.%2C%20Reyes%2C%20M.%2C%20Riegler%2C%20M.%20A.%2C%20Wiesenfarth%2C%20M.%2C%20Baumgartner%2C%20M.%2C%20Eisenmann%2C%20M.%2C%20Heckmann-N%26%23xF6%3Btzel%2C%20D.%2C%20Kavur%2C%20A.%20E.%2C%20R%26%23xE4%3Bdsch%2C%20T.%2C%20Acion%2C%20L.%2C%20Antonelli%2C%20M.%2C%20%26%23x2026%3B%20J%26%23xE4%3Bger%2C%20P.%20F.%20%282022%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BMetrics%20reloaded%3A%20Pitfalls%20and%20recommendations%20for%20image%20analysis%20validation%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2206.01653%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2206.01653%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2206.01653%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Metrics%20reloaded%3A%20Pitfalls%20and%20recommendations%20for%20image%20analysis%20validation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Glocker%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Kleesiek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mauricio%22%2C%22lastName%22%3A%22Reyes%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20A.%22%2C%22lastName%22%3A%22Riegler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuel%22%2C%22lastName%22%3A%22Wiesenfarth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Baumgartner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Doreen%22%2C%22lastName%22%3A%22Heckmann-N%5Cu00f6tzel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22A.%20Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Acion%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michela%22%2C%22lastName%22%3A%22Antonelli%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tal%22%2C%22lastName%22%3A%22Arbel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Bankhead%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arriel%22%2C%22lastName%22%3A%22Benis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Jorge%22%2C%22lastName%22%3A%22Cardoso%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beth%22%2C%22lastName%22%3A%22Cimini%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gary%20S.%22%2C%22lastName%22%3A%22Collins%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Keyvan%22%2C%22lastName%22%3A%22Farahani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Luciana%22%2C%22lastName%22%3A%22Ferrer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrian%22%2C%22lastName%22%3A%22Galdran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22van%20Ginneken%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Robert%22%2C%22lastName%22%3A%22Haase%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%20A.%22%2C%22lastName%22%3A%22Hashimoto%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20M.%22%2C%22lastName%22%3A%22Hoffman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Merel%22%2C%22lastName%22%3A%22Huisman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Jannin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Charles%20E.%22%2C%22lastName%22%3A%22Kahn%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dagmar%22%2C%22lastName%22%3A%22Kainmueller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bernhard%22%2C%22lastName%22%3A%22Kainz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexandros%22%2C%22lastName%22%3A%22Karargyris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alan%22%2C%22lastName%22%3A%22Karthikesalingam%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hannes%22%2C%22lastName%22%3A%22Kenngott%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Kofler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%22%2C%22lastName%22%3A%22Kreshuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tahsin%22%2C%22lastName%22%3A%22Kurc%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%20A.%22%2C%22lastName%22%3A%22Landman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Geert%22%2C%22lastName%22%3A%22Litjens%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amin%22%2C%22lastName%22%3A%22Madani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anne%20L.%22%2C%22lastName%22%3A%22Martel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Mattson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Erik%22%2C%22lastName%22%3A%22Meijering%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Moher%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karel%20G.%20M.%22%2C%22lastName%22%3A%22Moons%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Henning%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brennan%22%2C%22lastName%22%3A%22Nichyporuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Nickel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Petersen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nasir%22%2C%22lastName%22%3A%22Rajpoot%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicola%22%2C%22lastName%22%3A%22Rieke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julio%22%2C%22lastName%22%3A%22Saez-Rodriguez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Clarisa%20S%5Cu00e1nchez%22%2C%22lastName%22%3A%22Guti%5Cu00e9rrez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shravya%22%2C%22lastName%22%3A%22Shetty%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maarten%22%2C%22lastName%22%3A%22van%20Smeden%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%20H.%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ronald%20M.%22%2C%22lastName%22%3A%22Summers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdel%20A.%22%2C%22lastName%22%3A%22Taha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sotirios%20A.%22%2C%22lastName%22%3A%22Tsaftaris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Van%20Calster%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ga%5Cu00ebl%22%2C%22lastName%22%3A%22Varoquaux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%20F.%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%5D%2C%22abstractNote%22%3A%22Increasing%20evidence%20shows%20that%20flaws%20in%20machine%20learning%20%28ML%29%20algorithm%20validation%20are%20an%20underestimated%20global%20problem.%20Particularly%20in%20automatic%20biomedical%20image%20analysis%2C%20chosen%20performance%20metrics%20often%20do%20not%20reflect%20the%20domain%20interest%2C%20thus%20failing%20to%20adequately%20measure%20scientific%20progress%20and%20hindering%20translation%20of%20ML%20techniques%20into%20practice.%20To%20overcome%20this%2C%20a%20large%20international%20expert%20consortium%20created%20Metrics%20Reloaded%2C%20a%20comprehensive%20framework%20guiding%20researchers%20towards%20choosing%20metrics%20in%20a%20problem-aware%20manner.%20Following%20the%20convergence%20of%20ML%20methodology%20across%20application%20domains%2C%20Metrics%20Reloaded%20fosters%20the%20convergence%20of%20validation%20methodology.%20The%20framework%20was%20developed%20in%20a%20multi-stage%20Delphi%20process%20and%20is%20based%20on%20the%20novel%20concept%20of%20a%20problem%20fingerprint%20-%20a%20structured%20representation%20of%20the%20given%20problem%20that%20captures%20all%20aspects%20that%20are%20relevant%20for%20metric%20selection%20from%20the%20domain%20interest%20to%20the%20properties%20of%20the%20target%20structure%28s%29%2C%20data%20set%20and%20algorithm%20output.%20Metrics%20Reloaded%20targets%20image%20analysis%20problems%20that%20can%20be%20interpreted%20as%20a%20classification%20task%20at%20image%2C%20object%20or%20pixel%20level%2C%20namely%20image-level%20classification%2C%20object%20detection%2C%20semantic%20segmentation%2C%20and%20instance%20segmentation%20tasks.%20Users%20are%20guided%20through%20the%20process%20of%20selecting%20and%20applying%20appropriate%20validation%20metrics%20while%20being%20made%20aware%20of%20potential%20pitfalls.%20To%20improve%20the%20user%20experience%2C%20we%20implemented%20the%20framework%20in%20the%20Metrics%20Reloaded%20online%20tool%2C%20which%20also%20provides%20a%20common%20point%20of%20access%20to%20explore%20weaknesses%20and%20strengths%20of%20the%20most%20common%20validation%20metrics.%20An%20instantiation%20of%20the%20framework%20for%20various%20biological%20and%20medical%20image%20analysis%20use%20cases%20demonstrates%20its%20broad%20applicability%20across%20domains.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2206.01653%22%2C%22date%22%3A%222022-09-15%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2206.01653%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2206.01653%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222022-11-21T15%3A54%3A13Z%22%7D%7D%2C%7B%22key%22%3A%22KJ7UJXQG%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Reinke%20et%20al.%22%2C%22parsedDate%22%3A%222022-07-07%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BReinke%2C%20A.%2C%20Tizabi%2C%20M.%20D.%2C%20Sudre%2C%20C.%20H.%2C%20Eisenmann%2C%20M.%2C%20R%26%23xE4%3Bdsch%2C%20T.%2C%20Baumgartner%2C%20M.%2C%20Acion%2C%20L.%2C%20Antonelli%2C%20M.%2C%20Arbel%2C%20T.%2C%20Bakas%2C%20S.%2C%20Bankhead%2C%20P.%2C%20Benis%2C%20A.%2C%20Cardoso%2C%20M.%20J.%2C%20Cheplygina%2C%20V.%2C%20Christodoulou%2C%20E.%2C%20Cimini%2C%20B.%2C%20Collins%2C%20G.%20S.%2C%20Farahani%2C%20K.%2C%20van%20Ginneken%2C%20B.%2C%20%26%23x2026%3B%20Maier-Hein%2C%20L.%20%282022%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BCommon%20Limitations%20of%20Image%20Processing%20Metrics%3A%20A%20Picture%20Story%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2104.05642%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2104.05642%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2104.05642%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Common%20Limitations%20of%20Image%20Processing%20Metrics%3A%20A%20Picture%20Story%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%20H.%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Baumgartner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Acion%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michela%22%2C%22lastName%22%3A%22Antonelli%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tal%22%2C%22lastName%22%3A%22Arbel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Bankhead%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arriel%22%2C%22lastName%22%3A%22Benis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Jorge%22%2C%22lastName%22%3A%22Cardoso%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beth%22%2C%22lastName%22%3A%22Cimini%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gary%20S.%22%2C%22lastName%22%3A%22Collins%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Keyvan%22%2C%22lastName%22%3A%22Farahani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22van%20Ginneken%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Glocker%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fred%22%2C%22lastName%22%3A%22Hamprecht%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%20A.%22%2C%22lastName%22%3A%22Hashimoto%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Doreen%22%2C%22lastName%22%3A%22Heckmann-N%5Cu00f6tzel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20M.%22%2C%22lastName%22%3A%22Hoffman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Merel%22%2C%22lastName%22%3A%22Huisman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Jannin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Charles%20E.%22%2C%22lastName%22%3A%22Kahn%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexandros%22%2C%22lastName%22%3A%22Karargyris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alan%22%2C%22lastName%22%3A%22Karthikesalingam%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bernhard%22%2C%22lastName%22%3A%22Kainz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hannes%22%2C%22lastName%22%3A%22Kenngott%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Kleesiek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thijs%22%2C%22lastName%22%3A%22Kooi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%22%2C%22lastName%22%3A%22Kreshuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tahsin%22%2C%22lastName%22%3A%22Kurc%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%20A.%22%2C%22lastName%22%3A%22Landman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Geert%22%2C%22lastName%22%3A%22Litjens%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amin%22%2C%22lastName%22%3A%22Madani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anne%20L.%22%2C%22lastName%22%3A%22Martel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Mattson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Erik%22%2C%22lastName%22%3A%22Meijering%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Moher%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karel%20G.%20M.%22%2C%22lastName%22%3A%22Moons%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Henning%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brennan%22%2C%22lastName%22%3A%22Nichyporuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Nickel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Alican%22%2C%22lastName%22%3A%22Noyan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Petersen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gorkem%22%2C%22lastName%22%3A%22Polat%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nasir%22%2C%22lastName%22%3A%22Rajpoot%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mauricio%22%2C%22lastName%22%3A%22Reyes%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicola%22%2C%22lastName%22%3A%22Rieke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Riegler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hassan%22%2C%22lastName%22%3A%22Rivaz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julio%22%2C%22lastName%22%3A%22Saez-Rodriguez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Clarisa%20Sanchez%22%2C%22lastName%22%3A%22Gutierrez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julien%22%2C%22lastName%22%3A%22Schroeter%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anindo%22%2C%22lastName%22%3A%22Saha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shravya%22%2C%22lastName%22%3A%22Shetty%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maarten%22%2C%22lastName%22%3A%22van%20Smeden%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22Stieltjes%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ronald%20M.%22%2C%22lastName%22%3A%22Summers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdel%20A.%22%2C%22lastName%22%3A%22Taha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sotirios%20A.%22%2C%22lastName%22%3A%22Tsaftaris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Van%20Calster%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ga%5Cu00ebl%22%2C%22lastName%22%3A%22Varoquaux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuel%22%2C%22lastName%22%3A%22Wiesenfarth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ziv%20R.%22%2C%22lastName%22%3A%22Yaniv%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22While%20the%20importance%20of%20automatic%20image%20analysis%20is%20continuously%20increasing%2C%20recent%20meta-research%20revealed%20major%20flaws%20with%20respect%20to%20algorithm%20validation.%20Performance%20metrics%20are%20particularly%20key%20for%20meaningful%2C%20objective%2C%20and%20transparent%20performance%20assessment%20and%20validation%20of%20the%20used%20automatic%20algorithms%2C%20but%20relatively%20little%20attention%20has%20been%20given%20to%20the%20practical%20pitfalls%20when%20using%20specific%20metrics%20for%20a%20given%20image%20analysis%20task.%20These%20are%20typically%20related%20to%20%281%29%20the%20disregard%20of%20inherent%20metric%20properties%2C%20such%20as%20the%20behaviour%20in%20the%20presence%20of%20class%20imbalance%20or%20small%20target%20structures%2C%20%282%29%20the%20disregard%20of%20inherent%20data%20set%20properties%2C%20such%20as%20the%20non-independence%20of%20the%20test%20cases%2C%20and%20%283%29%20the%20disregard%20of%20the%20actual%20biomedical%20domain%20interest%20that%20the%20metrics%20should%20reflect.%20This%20living%20dynamically%20document%20has%20the%20purpose%20to%20illustrate%20important%20limitations%20of%20performance%20metrics%20commonly%20applied%20in%20the%20field%20of%20image%20analysis.%20In%20this%20context%2C%20it%20focuses%20on%20biomedical%20image%20analysis%20problems%20that%20can%20be%20phrased%20as%20image-level%20classification%2C%20semantic%20segmentation%2C%20instance%20segmentation%2C%20or%20object%20detection%20task.%20The%20current%20version%20is%20based%20on%20a%20Delphi%20process%20on%20metrics%20conducted%20by%20an%20international%20consortium%20of%20image%20analysis%20experts%20from%20more%20than%2060%20institutions%20worldwide.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2104.05642%22%2C%22date%22%3A%222022-07-07%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2104.05642%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2104.05642%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222023-02-03T10%3A12%3A41Z%22%7D%7D%5D%7D
Rädsch, T., Reinke, A., Weru, V., Tizabi, M. D., Heller, N., Isensee, F., Kopp-Schneider, A., & Maier-Hein, L. (2024). Quality Assured: Rethinking Annotation Strategies in Imaging AI (arXiv:2407.17596). arXiv. https://doi.org/10.48550/arXiv.2407.17596
Reinke, A., Tizabi, M. D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Sudre, C. H., Acion, L., Antonelli, M., Arbel, T., Bakas, S., Benis, A., Buettner, F., Cardoso, M. J., Cheplygina, V., Chen, J., Christodoulou, E., Cimini, B. A., … Maier-Hein, L. (2024). Understanding metric-related pitfalls in image analysis validation. Nature Methods, 1–13. https://doi.org/10.1038/s41592-023-02150-0
Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M. D., Buettner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., Reyes, M., Riegler, M. A., Wiesenfarth, M., Kavur, A. E., Sudre, C. H., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Rädsch, T., … Jäger, P. F. (2024). Metrics reloaded: recommendations for image analysis validation. Nature Methods, 21(2), 195–212. https://doi.org/10.1038/s41592-023-02151-z
Eisenmann, M., Reinke, A., Weru, V., Tizabi, M. D., Isensee, F., Adler, T. J., Ali, S., Andrearczyk, V., Aubreville, M., Baid, U., Bakas, S., Balu, N., Bano, S., Bernal, J., Bodenstedt, S., Casella, A., Cheplygina, V., Daum, M., de Bruijne, M., … Maier-Hein, L. (2023). Why is the winner the best? (arXiv:2303.17719). arXiv. https://doi.org/10.48550/arXiv.2303.17719
Rädsch, T., Reinke, A., Weru, V., Tizabi, M. D., Schreck, N., Kavur, A. E., Pekdemir, B., Roß, T., Kopp-Schneider, A., & Maier-Hein, L. (2023). Labelling instructions matter in biomedical image analysis. Nature Machine Intelligence, 1–11. https://doi.org/10.1038/s42256-023-00625-5
Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M. D., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., Reyes, M., Riegler, M. A., Wiesenfarth, M., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Acion, L., Antonelli, M., … Jäger, P. F. (2022). Metrics reloaded: Pitfalls and recommendations for image analysis validation (arXiv:2206.01653). arXiv. https://doi.org/10.48550/arXiv.2206.01653
Reinke, A., Tizabi, M. D., Sudre, C. H., Eisenmann, M., Rädsch, T., Baumgartner, M., Acion, L., Antonelli, M., Arbel, T., Bakas, S., Bankhead, P., Benis, A., Cardoso, M. J., Cheplygina, V., Christodoulou, E., Cimini, B., Collins, G. S., Farahani, K., van Ginneken, B., … Maier-Hein, L. (2022). Common Limitations of Image Processing Metrics: A Picture Story (arXiv:2104.05642). arXiv. https://doi.org/10.48550/arXiv.2104.05642
