Dr. Annika Reinke
DKFZ research representative
Dr. Annika Reinke earned her PhD degree in 2023, focusing on eliminating flaws in biomedical image analysis validation. She continues her work as a postdoctoral researcher and deputy head by addressing underrepresented societally relevant topics, particularly scientific benchmarking and validation. Leading the “Validation of Intelligent Systems” group, Annika aims to enhance the quality of validation pipelines for biomedical AI algorithms. She holds influential positions in various international groups, including the secretary of the MICCAI Special Interest Group on biomedical challenges and the chair of the MONAI Working Group on Evaluation and Benchmarking.
Publications
4725570
Reinke
1
https://helmholtz-imaging.de/apa-bold-title.csl
50
date
desc
792
https://helmholtz-imaging.de/wp-content/plugins/zotpress/
%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3Afalse%2C%22meta%22%3A%7B%22request_last%22%3A0%2C%22request_next%22%3A0%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%228QPTGZ82%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Godau%20et%20al.%22%2C%22parsedDate%22%3A%222025-05-01%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BGodau%2C%20P.%2C%20Kalinowski%2C%20P.%2C%20Christodoulou%2C%20E.%2C%20Reinke%2C%20A.%2C%20Tizabi%2C%20M.%2C%20Ferrer%2C%20L.%2C%20J%26%23xE4%3Bger%2C%20P.%2C%20%26amp%3B%20Maier-Hein%2C%20L.%20%282025%29.%20%26lt%3Bb%26gt%3BNavigating%20prevalence%20shifts%20in%20image%20analysis%20algorithm%20deployment%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BMedical%20Image%20Analysis%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B102%26lt%3B%5C%2Fi%26gt%3B%2C%20103504.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2025.103504%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2025.103504%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Navigating%20prevalence%20shifts%20in%20image%20analysis%20algorithm%20deployment%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Piotr%22%2C%22lastName%22%3A%22Kalinowski%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Luciana%22%2C%22lastName%22%3A%22Ferrer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Domain%20gaps%20are%20significant%20obstacles%20to%20the%20clinical%20implementation%20of%20machine%20learning%20%28ML%29%20solutions%20for%20medical%20image%20analysis.%20Although%20current%20research%20emphasizes%20new%20training%20methods%20and%20network%20architectures%2C%20the%20specific%20impact%20of%20prevalence%20shifts%20on%20algorithms%20in%20real-world%20applications%20is%20often%20overlooked.%20Differences%20in%20class%20frequencies%20between%20development%20and%20deployment%20data%20are%20crucial%2C%20particularly%20for%20the%20widespread%20adoption%20of%20artificial%20intelligence%20%28AI%29%2C%20as%20disease%20prevalence%20can%20vary%20greatly%20across%20different%20times%20and%20locations.%20Our%20contribution%20is%20threefold.%20Based%20on%20a%20diverse%20set%20of%2030%20medical%20classification%20tasks%20%281%29%20we%20demonstrate%20that%20lack%20of%20prevalence%20shift%20handling%20can%20have%20severe%20consequences%20on%20the%20quality%20of%20calibration%2C%20decision%20threshold%2C%20and%20performance%20assessment.%20Furthermore%2C%20%282%29%20we%20show%20that%20prevalences%20can%20be%20accurately%20and%20reliably%20estimated%20in%20a%20data-driven%20manner.%20Finally%2C%20%283%29%20we%20propose%20a%20new%20workflow%20for%20prevalence-aware%20image%20classification%20that%20uses%20estimated%20deployment%20prevalences%20to%20adjust%20a%20trained%20classifier%20to%20a%20new%20environment%2C%20without%20requiring%20additional%20annotated%20deployment%20data.%20Comprehensive%20experiments%20indicate%20that%20our%20proposed%20approach%20could%20contribute%20to%20generating%20better%20classifier%20decisions%20and%20more%20reliable%20performance%20estimates%20compared%20to%20current%20practice.%22%2C%22date%22%3A%222025-05-01%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1016%5C%2Fj.media.2025.103504%22%2C%22ISSN%22%3A%221361-8415%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.sciencedirect.com%5C%2Fscience%5C%2Farticle%5C%2Fpii%5C%2FS1361841525000520%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-03-04T13%3A35%3A19Z%22%7D%7D%2C%7B%22key%22%3A%22GPCM9X32%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22lastModifiedByUser%22%3A%7B%22id%22%3A9737705%2C%22username%22%3A%22Niebelsa%22%2C%22name%22%3A%22%22%2C%22links%22%3A%7B%22alternate%22%3A%7B%22href%22%3A%22https%3A%5C%2F%5C%2Fwww.zotero.org%5C%2Fniebelsa%22%2C%22type%22%3A%22text%5C%2Fhtml%22%7D%7D%7D%2C%22creatorSummary%22%3A%22Adler%20et%20al.%22%2C%22parsedDate%22%3A%222025%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BAdler%2C%20T.%20J.%2C%20N%26%23xF6%3Blke%2C%20J.-H.%2C%20Reinke%2C%20A.%2C%20Tizabi%2C%20M.%20D.%2C%20Gruber%2C%20S.%2C%20Trofimova%2C%20D.%2C%20Ardizzone%2C%20L.%2C%20Jaeger%2C%20P.%20F.%2C%20Buettner%2C%20F.%2C%20K%26%23xF6%3Bthe%2C%20U.%2C%20%26amp%3B%20Maier-Hein%2C%20L.%20%282025%29.%20%26lt%3Bb%26gt%3BApplication-driven%20validation%20of%20posteriors%20in%20inverse%20problems%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BMedical%20Image%20Analysis%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B101%26lt%3B%5C%2Fi%26gt%3B%2C%20103474.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2025.103474%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2025.103474%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Application-driven%20validation%20of%20posteriors%20in%20inverse%20problems%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%20J.%22%2C%22lastName%22%3A%22Adler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jan-Hinrich%22%2C%22lastName%22%3A%22N%5Cu00f6lke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20Dietlinde%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sebastian%22%2C%22lastName%22%3A%22Gruber%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dasha%22%2C%22lastName%22%3A%22Trofimova%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lynton%22%2C%22lastName%22%3A%22Ardizzone%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%20F.%22%2C%22lastName%22%3A%22Jaeger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Buettner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ullrich%22%2C%22lastName%22%3A%22K%5Cu00f6the%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%2204%5C%2F2025%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1016%5C%2Fj.media.2025.103474%22%2C%22ISSN%22%3A%2213618415%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Flinkinghub.elsevier.com%5C%2Fretrieve%5C%2Fpii%5C%2FS1361841525000222%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-05-02T07%3A43%3A28Z%22%7D%7D%2C%7B%22key%22%3A%22ETUKTI3R%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Cimini%20et%20al.%22%2C%22parsedDate%22%3A%222024-10-30%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BCimini%2C%20B.%20A.%2C%20Bankhead%2C%20P.%2C%20D%26%23x2019%3BAntuono%2C%20R.%2C%20Fazeli%2C%20E.%2C%20Fernandez-Rodriguez%2C%20J.%2C%20Fuster-Barcel%26%23xF3%3B%2C%20C.%2C%20Haase%2C%20R.%2C%20Jambor%2C%20H.%20K.%2C%20Jones%2C%20M.%20L.%2C%20Jug%2C%20F.%2C%20Klemm%2C%20A.%20H.%2C%20Kreshuk%2C%20A.%2C%20Marcotti%2C%20S.%2C%20Martins%2C%20G.%20G.%2C%20McArdle%2C%20S.%2C%20Miura%2C%20K.%2C%20Mu%26%23xF1%3Boz-Barrutia%2C%20A.%2C%20Murphy%2C%20L.%20C.%2C%20Nelson%2C%20M.%20S.%2C%20%26%23x2026%3B%20Eliceiri%2C%20K.%20W.%20%282024%29.%20%26lt%3Bb%26gt%3BThe%20crucial%20role%20of%20bioimage%20analysts%20in%20scientific%20research%20and%20publication%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BJournal%20of%20Cell%20Science%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B137%26lt%3B%5C%2Fi%26gt%3B%2820%29%2C%20jcs262322.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1242%5C%2Fjcs.262322%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1242%5C%2Fjcs.262322%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22The%20crucial%20role%20of%20bioimage%20analysts%20in%20scientific%20research%20and%20publication%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beth%20A.%22%2C%22lastName%22%3A%22Cimini%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Bankhead%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rocco%22%2C%22lastName%22%3A%22D%27Antuono%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Elnaz%22%2C%22lastName%22%3A%22Fazeli%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julia%22%2C%22lastName%22%3A%22Fernandez-Rodriguez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Caterina%22%2C%22lastName%22%3A%22Fuster-Barcel%5Cu00f3%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Robert%22%2C%22lastName%22%3A%22Haase%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Helena%20Klara%22%2C%22lastName%22%3A%22Jambor%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%20L.%22%2C%22lastName%22%3A%22Jones%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Jug%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%20H.%22%2C%22lastName%22%3A%22Klemm%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%22%2C%22lastName%22%3A%22Kreshuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stefania%22%2C%22lastName%22%3A%22Marcotti%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gabriel%20G.%22%2C%22lastName%22%3A%22Martins%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sara%22%2C%22lastName%22%3A%22McArdle%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kota%22%2C%22lastName%22%3A%22Miura%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arrate%22%2C%22lastName%22%3A%22Mu%5Cu00f1oz-Barrutia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%20C.%22%2C%22lastName%22%3A%22Murphy%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20S.%22%2C%22lastName%22%3A%22Nelson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Simon%20F.%22%2C%22lastName%22%3A%22N%5Cu00f8rrelykke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Perrine%22%2C%22lastName%22%3A%22Paul-Gilloteaux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thomas%22%2C%22lastName%22%3A%22Pengo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Joanna%20W.%22%2C%22lastName%22%3A%22Pylv%5Cu00e4n%5Cu00e4inen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lior%22%2C%22lastName%22%3A%22Pytowski%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arianna%22%2C%22lastName%22%3A%22Ravera%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yousr%22%2C%22lastName%22%3A%22Rekik%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Caterina%22%2C%22lastName%22%3A%22Strambio-De-Castillia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%22%2C%22lastName%22%3A%22Th%5Cu00e9di%5Cu00e9%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Virginie%22%2C%22lastName%22%3A%22Uhlmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Oliver%22%2C%22lastName%22%3A%22Umney%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Wiggins%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kevin%20W.%22%2C%22lastName%22%3A%22Eliceiri%22%7D%5D%2C%22abstractNote%22%3A%22Bioimage%20analysis%20%28BIA%29%2C%20a%20crucial%20discipline%20in%20biological%20research%2C%20overcomes%20the%20limitations%20of%20subjective%20analysis%20in%20microscopy%20through%20the%20creation%20and%20application%20of%20quantitative%20and%20reproducible%20methods.%20The%20establishment%20of%20dedicated%20BIA%20support%20within%20academic%20institutions%20is%20vital%20to%20improving%20research%20quality%20and%20efficiency%20and%20can%20significantly%20advance%20scientific%20discovery.%20However%2C%20a%20lack%20of%20training%20resources%2C%20limited%20career%20paths%20and%20insufficient%20recognition%20of%20the%20contributions%20made%20by%20bioimage%20analysts%20prevent%20the%20full%20realization%20of%20this%20potential.%20This%20Perspective%20%5Cu2013%20the%20result%20of%20the%20recent%20The%20Company%20of%20Biologists%20Workshop%20%5Cu2018Effectively%20Communicating%20Bioimage%20Analysis%5Cu2019%2C%20which%20aimed%20to%20summarize%20the%20global%20BIA%20landscape%2C%20categorize%20obstacles%20and%20offer%20possible%20solutions%20%5Cu2013%20proposes%20strategies%20to%20bring%20about%20a%20cultural%20shift%20towards%20recognizing%20the%20value%20of%20BIA%20by%20standardizing%20tools%2C%20improving%20training%20and%20encouraging%20formal%20credit%20for%20contributions.%20We%20also%20advocate%20for%20increased%20funding%2C%20standardized%20practices%20and%20enhanced%20collaboration%2C%20and%20we%20conclude%20with%20a%20call%20to%20action%20for%20all%20stakeholders%20to%20join%20efforts%20in%20advancing%20BIA.%22%2C%22date%22%3A%222024-10-30%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1242%5C%2Fjcs.262322%22%2C%22ISSN%22%3A%220021-9533%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1242%5C%2Fjcs.262322%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-02-03T13%3A06%3A02Z%22%7D%7D%2C%7B%22key%22%3A%22Y6HQFQFI%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22lastModifiedByUser%22%3A%7B%22id%22%3A9737705%2C%22username%22%3A%22Niebelsa%22%2C%22name%22%3A%22%22%2C%22links%22%3A%7B%22alternate%22%3A%7B%22href%22%3A%22https%3A%5C%2F%5C%2Fwww.zotero.org%5C%2Fniebelsa%22%2C%22type%22%3A%22text%5C%2Fhtml%22%7D%7D%7D%2C%22creatorSummary%22%3A%22Christodoulou%20et%20al.%22%2C%22parsedDate%22%3A%222024-09-27%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BChristodoulou%2C%20E.%2C%20Reinke%2C%20A.%2C%20Houhou%2C%20R.%2C%20Kalinowski%2C%20P.%2C%20Erkan%2C%20S.%2C%20Sudre%2C%20C.%20H.%2C%20Burgos%2C%20N.%2C%20Boutaj%2C%20S.%2C%20Loizillon%2C%20S.%2C%20Solal%2C%20M.%2C%20Rieke%2C%20N.%2C%20Cheplygina%2C%20V.%2C%20Antonelli%2C%20M.%2C%20Mayer%2C%20L.%20D.%2C%20Tizabi%2C%20M.%20D.%2C%20Cardoso%2C%20M.%20J.%2C%20Simpson%2C%20A.%2C%20J%26%23xE4%3Bger%2C%20P.%20F.%2C%20Kopp-Schneider%2C%20A.%2C%20%26%23x2026%3B%20Maier-Hein%2C%20L.%20%282024%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BConfidence%20intervals%20uncovered%3A%20Are%20we%20ready%20for%20real-world%20medical%20imaging%20AI%3F%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2409.17763%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2409.17763%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2409.17763%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Confidence%20intervals%20uncovered%3A%20Are%20we%20ready%20for%20real-world%20medical%20imaging%20AI%3F%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rola%22%2C%22lastName%22%3A%22Houhou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Piotr%22%2C%22lastName%22%3A%22Kalinowski%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Selen%22%2C%22lastName%22%3A%22Erkan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%20H.%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ninon%22%2C%22lastName%22%3A%22Burgos%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sofi%5Cu00e8ne%22%2C%22lastName%22%3A%22Boutaj%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sophie%22%2C%22lastName%22%3A%22Loizillon%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ma%5Cu00eblys%22%2C%22lastName%22%3A%22Solal%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicola%22%2C%22lastName%22%3A%22Rieke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michela%22%2C%22lastName%22%3A%22Antonelli%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Leon%20D.%22%2C%22lastName%22%3A%22Mayer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Jorge%22%2C%22lastName%22%3A%22Cardoso%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amber%22%2C%22lastName%22%3A%22Simpson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%20F.%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ga%5Cu00ebl%22%2C%22lastName%22%3A%22Varoquaux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Olivier%22%2C%22lastName%22%3A%22Colliot%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Medical%20imaging%20is%20spearheading%20the%20AI%20transformation%20of%20healthcare.%20Performance%20reporting%20is%20key%20to%20determine%20which%20methods%20should%20be%20translated%20into%20clinical%20practice.%20Frequently%2C%20broad%20conclusions%20are%20simply%20derived%20from%20mean%20performance%20values.%20In%20this%20paper%2C%20we%20argue%20that%20this%20common%20practice%20is%20often%20a%20misleading%20simplification%20as%20it%20ignores%20performance%20variability.%20Our%20contribution%20is%20threefold.%20%281%29%20Analyzing%20all%20MICCAI%20segmentation%20papers%20%28n%20%3D%20221%29%20published%20in%202023%2C%20we%20first%20observe%20that%20more%20than%2050%25%20of%20papers%20do%20not%20assess%20performance%20variability%20at%20all.%20Moreover%2C%20only%20one%20%280.5%25%29%20paper%20reported%20confidence%20intervals%20%28CIs%29%20for%20model%20performance.%20%282%29%20To%20address%20the%20reporting%20bottleneck%2C%20we%20show%20that%20the%20unreported%20standard%20deviation%20%28SD%29%20in%20segmentation%20papers%20can%20be%20approximated%20by%20a%20second-order%20polynomial%20function%20of%20the%20mean%20Dice%20similarity%20coefficient%20%28DSC%29.%20Based%20on%20external%20validation%20data%20from%2056%20previous%20MICCAI%20challenges%2C%20we%20demonstrate%20that%20this%20approximation%20can%20accurately%20reconstruct%20the%20CI%20of%20a%20method%20using%20information%20provided%20in%20publications.%20%283%29%20Finally%2C%20we%20reconstructed%2095%25%20CIs%20around%20the%20mean%20DSC%20of%20MICCAI%202023%20segmentation%20papers.%20The%20median%20CI%20width%20was%200.03%20which%20is%20three%20times%20larger%20than%20the%20median%20performance%20gap%20between%20the%20first%20and%20second%20ranked%20method.%20For%20more%20than%2060%25%20of%20papers%2C%20the%20mean%20performance%20of%20the%20second-ranked%20method%20was%20within%20the%20CI%20of%20the%20first-ranked%20method.%20We%20conclude%20that%20current%20publications%20typically%20do%20not%20provide%20sufficient%20evidence%20to%20support%20which%20models%20could%20potentially%20be%20translated%20into%20clinical%20practice.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2409.17763%22%2C%22date%22%3A%222024-09-27%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2409.17763%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2409.17763%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-02-03T13%3A03%3A32Z%22%7D%7D%2C%7B%22key%22%3A%22YMWVH5AM%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22R%5Cu00e4dsch%20et%20al.%22%2C%22parsedDate%22%3A%222024-07-26%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BR%26%23xE4%3Bdsch%2C%20T.%2C%20Reinke%2C%20A.%2C%20Weru%2C%20V.%2C%20Tizabi%2C%20M.%20D.%2C%20Heller%2C%20N.%2C%20Isensee%2C%20F.%2C%20Kopp-Schneider%2C%20A.%2C%20%26amp%3B%20Maier-Hein%2C%20L.%20%282024%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BQuality%20Assured%3A%20Rethinking%20Annotation%20Strategies%20in%20Imaging%20AI%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2407.17596%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2407.17596%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2407.17596%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Quality%20Assured%3A%20Rethinking%20Annotation%20Strategies%20in%20Imaging%20AI%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vivienn%22%2C%22lastName%22%3A%22Weru%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicholas%22%2C%22lastName%22%3A%22Heller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22This%20paper%20does%20not%20describe%20a%20novel%20method.%20Instead%2C%20it%20studies%20an%20essential%20foundation%20for%20reliable%20benchmarking%20and%20ultimately%20real-world%20application%20of%20AI-based%20image%20analysis%3A%20generating%20high-quality%20reference%20annotations.%20Previous%20research%20has%20focused%20on%20crowdsourcing%20as%20a%20means%20of%20outsourcing%20annotations.%20However%2C%20little%20attention%20has%20so%20far%20been%20given%20to%20annotation%20companies%2C%20specifically%20regarding%20their%20internal%20quality%20assurance%20%28QA%29%20processes.%20Therefore%2C%20our%20aim%20is%20to%20evaluate%20the%20influence%20of%20QA%20employed%20by%20annotation%20companies%20on%20annotation%20quality%20and%20devise%20methodologies%20for%20maximizing%20data%20annotation%20efficacy.%20Based%20on%20a%20total%20of%2057%2C648%20instance%20segmented%20images%20obtained%20from%20a%20total%20of%20924%20annotators%20and%2034%20QA%20workers%20from%20four%20annotation%20companies%20and%20Amazon%20Mechanical%20Turk%20%28MTurk%29%2C%20we%20derived%20the%20following%20insights%3A%20%281%29%20Annotation%20companies%20perform%20better%20both%20in%20terms%20of%20quantity%20and%20quality%20compared%20to%20the%20widely%20used%20platform%20MTurk.%20%282%29%20Annotation%20companies%26%23039%3B%20internal%20QA%20only%20provides%20marginal%20improvements%2C%20if%20any.%20However%2C%20improving%20labeling%20instructions%20instead%20of%20investing%20in%20QA%20can%20substantially%20boost%20annotation%20performance.%20%283%29%20The%20benefit%20of%20internal%20QA%20depends%20on%20specific%20image%20characteristics.%20Our%20work%20could%20enable%20researchers%20to%20derive%20substantially%20more%20value%20from%20a%20fixed%20annotation%20budget%20and%20change%20the%20way%20annotation%20companies%20conduct%20internal%20QA.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2407.17596%22%2C%22date%22%3A%222024-07-26%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2407.17596%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2407.17596%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-02-03T13%3A07%3A10Z%22%7D%7D%2C%7B%22key%22%3A%22Y86EVCJF%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Floca%20et%20al.%22%2C%22parsedDate%22%3A%222024-06-03%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BFloca%2C%20R.%2C%20Bohn%2C%20J.%2C%20Haux%2C%20C.%2C%20Wiestler%2C%20B.%2C%20Z%26%23xF6%3Bllner%2C%20F.%20G.%2C%20Reinke%2C%20A.%2C%20Wei%26%23xDF%3B%2C%20J.%2C%20Nolden%2C%20M.%2C%20Albert%2C%20S.%2C%20Persigehl%2C%20T.%2C%20Norajitra%2C%20T.%2C%20Bae%26%23xDF%3Bler%2C%20B.%2C%20Dewey%2C%20M.%2C%20Braren%2C%20R.%2C%20B%26%23xFC%3Bchert%2C%20M.%2C%20Fallenberg%2C%20E.%20M.%2C%20Galldiks%2C%20N.%2C%20Gerken%2C%20A.%2C%20G%26%23xF6%3Btz%2C%20M.%2C%20%26%23x2026%3B%20Bamberg%2C%20F.%20%282024%29.%20%26lt%3Bb%26gt%3BRadiomics%20workflow%20definition%20%26amp%3B%20challenges%20-%20German%20priority%20program%202177%20consensus%20statement%20on%20clinically%20applied%20radiomics%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BInsights%20into%20Imaging%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B15%26lt%3B%5C%2Fi%26gt%3B%281%29%2C%20124.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1186%5C%2Fs13244-024-01704-w%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1186%5C%2Fs13244-024-01704-w%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Radiomics%20workflow%20definition%20%26%20challenges%20-%20German%20priority%20program%202177%20consensus%20statement%20on%20clinically%20applied%20radiomics%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ralf%22%2C%22lastName%22%3A%22Floca%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jonas%22%2C%22lastName%22%3A%22Bohn%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Christian%22%2C%22lastName%22%3A%22Haux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Benedikt%22%2C%22lastName%22%3A%22Wiestler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Frank%20G.%22%2C%22lastName%22%3A%22Z%5Cu00f6llner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jakob%22%2C%22lastName%22%3A%22Wei%5Cu00df%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marco%22%2C%22lastName%22%3A%22Nolden%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Steffen%22%2C%22lastName%22%3A%22Albert%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thorsten%22%2C%22lastName%22%3A%22Persigehl%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tobias%22%2C%22lastName%22%3A%22Norajitra%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bettina%22%2C%22lastName%22%3A%22Bae%5Cu00dfler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Dewey%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rickmer%22%2C%22lastName%22%3A%22Braren%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22B%5Cu00fcchert%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Eva%20Maria%22%2C%22lastName%22%3A%22Fallenberg%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Norbert%22%2C%22lastName%22%3A%22Galldiks%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Gerken%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22G%5Cu00f6tz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Horst%20K.%22%2C%22lastName%22%3A%22Hahn%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Johannes%22%2C%22lastName%22%3A%22Haubold%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tobias%22%2C%22lastName%22%3A%22Haueise%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nils%22%2C%22lastName%22%3A%22Gro%5Cu00dfe%20Hokamp%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Ingrisch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Andra-Iza%22%2C%22lastName%22%3A%22Iuga%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marco%22%2C%22lastName%22%3A%22Janoschke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Jung%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%20Sophie%22%2C%22lastName%22%3A%22Kiefer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Philipp%22%2C%22lastName%22%3A%22Lohmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22J%5Cu00fcrgen%22%2C%22lastName%22%3A%22Machann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jan%20Hendrik%22%2C%22lastName%22%3A%22Moltz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Johanna%22%2C%22lastName%22%3A%22Nattenm%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tobias%22%2C%22lastName%22%3A%22Nonnenmacher%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Benedict%22%2C%22lastName%22%3A%22Oerther%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ahmed%20E.%22%2C%22lastName%22%3A%22Othman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Peisen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fritz%22%2C%22lastName%22%3A%22Schick%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lale%22%2C%22lastName%22%3A%22Umutlu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Barbara%20D.%22%2C%22lastName%22%3A%22Wichtmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wenzhao%22%2C%22lastName%22%3A%22Zhao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Svenja%22%2C%22lastName%22%3A%22Caspers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Heinz-Peter%22%2C%22lastName%22%3A%22Schlemmer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Christopher%20L.%22%2C%22lastName%22%3A%22Schlett%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Bamberg%22%7D%5D%2C%22abstractNote%22%3A%22Achieving%20a%20consensus%20on%20a%20definition%20for%20different%20aspects%20of%20radiomics%20workflows%20to%20support%20their%20translation%20into%20clinical%20usage.%20Furthermore%2C%20to%20assess%20the%20perspective%20of%20experts%20on%20important%20challenges%20for%20a%20successful%20clinical%20workflow%20implementation.%22%2C%22date%22%3A%222024-06-03%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1186%5C%2Fs13244-024-01704-w%22%2C%22ISSN%22%3A%221869-4101%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1186%5C%2Fs13244-024-01704-w%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-02-03T13%3A04%3A44Z%22%7D%7D%2C%7B%22key%22%3A%22YEZZA9X5%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Mais%20et%20al.%22%2C%22parsedDate%22%3A%222024-06%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BMais%2C%20L.%2C%20Hirsch%2C%20P.%2C%20Managan%2C%20C.%2C%20Kandarpa%2C%20R.%2C%20Rumberger%2C%20J.%20L.%2C%20Reinke%2C%20A.%2C%20Maier-Hein%2C%20L.%2C%20Ihrke%2C%20G.%2C%20%26amp%3B%20Kainmueller%2C%20D.%20%282024%29.%20%26lt%3Bb%26gt%3BFISBe%3A%20A%20Real-World%20Benchmark%20Dataset%20for%20Instance%20Segmentation%20of%20Long-Range%20thin%20Filamentous%20Structures%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3B2024%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20%28CVPR%29%26lt%3B%5C%2Fi%26gt%3B%2C%2022249%26%23x2013%3B22259.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FCVPR52733.2024.02100%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FCVPR52733.2024.02100%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22FISBe%3A%20A%20Real-World%20Benchmark%20Dataset%20for%20Instance%20Segmentation%20of%20Long-Range%20thin%20Filamentous%20Structures%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lisa%22%2C%22lastName%22%3A%22Mais%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Hirsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Claire%22%2C%22lastName%22%3A%22Managan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ramya%22%2C%22lastName%22%3A%22Kandarpa%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Josef%20Lorenz%22%2C%22lastName%22%3A%22Rumberger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gudrun%22%2C%22lastName%22%3A%22Ihrke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dagmar%22%2C%22lastName%22%3A%22Kainmueller%22%7D%5D%2C%22abstractNote%22%3A%22Instance%20segmentation%20of%20neurons%20in%20volumetric%20light%20microscopy%20images%20of%20nervous%20systems%20enables%20ground-breaking%20research%20in%20neuroscience%20by%20facilitating%20joint%20functional%20and%20morphological%20analyses%20of%20neural%20circuits%20at%20cel-lular%20resolution.%20Yet%20said%20multi-neuron%20light%20microscopy%20data%20exhibits%20extremely%20challenging%20properties%20for%20the%20task%20of%20instance%20segmentation%3A%20Individual%20neurons%20have%20long-ranging%2C%20thin%20filamentous%20and%20widely%20branching%20morpholo-gies%2C%20multiple%20neurons%20are%20tightly%20inter-weaved%2C%20and%20par-tial%20volume%20effects%2C%20uneven%20illumination%20and%20noise%20inherent%20to%20light%20microscopy%20severely%20impede%20local%20disentan-gling%20as%20well%20as%20long-range%20tracing%20of%20individual%20neurons.%20These%20properties%20reflect%20a%20current%20key%20challenge%20in%20machine%20learning%20research%2C%20namely%20to%20effectively%20capture%20long-range%20dependencies%20in%20the%20data.%20While%20respective%20method-ological%20research%20is%20buzzing%2C%20to%20date%20methods%20are%20typically%20benchmarked%20on%20synthetic%20datasets.%20To%20address%20this%20gap%2C%20we%20release%20the%20FlyLight%20Instance%20Segmentation%20Benchmark%20%28FISBe%29%20dataset%2C%20the%20first%20publicly%20available%20multi-neuron%20light%20microscopy%20dataset%20with%20pixel-wise%20annotations.%20In%20addition%2C%20we%20define%20a%20set%20of%20instance%20segmentation%20metrics%20for%20benchmarking%20that%20we%20designed%20to%20be%20meaningful%20with%20regard%20to%20downstream%20analyses.%20Lastly%2C%20we%20provide%20three%20baselines%20to%20kick%20off%20a%20competition%20that%20we%20envision%20to%20both%20advance%20the%20field%20of%20machine%20learning%20regarding%20methodology%20for%20capturing%20long-range%20data%20dependencies%2C%20and%20facilitate%20scientific%20discovery%20in%20basic%20neuroscience.%20Project%20page%3A%20https%3A%5C%2F%5C%2Fkainmueller-lab.github.io%5C%2Fjisbe.%22%2C%22date%22%3A%222024-06%22%2C%22proceedingsTitle%22%3A%222024%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20%28CVPR%29%22%2C%22conferenceName%22%3A%222024%20IEEE%5C%2FCVF%20Conference%20on%20Computer%20Vision%20and%20Pattern%20Recognition%20%28CVPR%29%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FCVPR52733.2024.02100%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F10656899%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-02-03T12%3A13%3A28Z%22%7D%7D%2C%7B%22key%22%3A%225ZQ8WVJ8%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Reinke%20et%20al.%22%2C%22parsedDate%22%3A%222024-02-12%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BReinke%2C%20A.%2C%20Tizabi%2C%20M.%20D.%2C%20Baumgartner%2C%20M.%2C%20Eisenmann%2C%20M.%2C%20Heckmann-N%26%23xF6%3Btzel%2C%20D.%2C%20Kavur%2C%20A.%20E.%2C%20R%26%23xE4%3Bdsch%2C%20T.%2C%20Sudre%2C%20C.%20H.%2C%20Acion%2C%20L.%2C%20Antonelli%2C%20M.%2C%20Arbel%2C%20T.%2C%20Bakas%2C%20S.%2C%20Benis%2C%20A.%2C%20Buettner%2C%20F.%2C%20Cardoso%2C%20M.%20J.%2C%20Cheplygina%2C%20V.%2C%20Chen%2C%20J.%2C%20Christodoulou%2C%20E.%2C%20Cimini%2C%20B.%20A.%2C%20%26%23x2026%3B%20Maier-Hein%2C%20L.%20%282024%29.%20%26lt%3Bb%26gt%3BUnderstanding%20metric-related%20pitfalls%20in%20image%20analysis%20validation%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BNature%20Methods%26lt%3B%5C%2Fi%26gt%3B%2C%201%26%23x2013%3B13.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41592-023-02150-0%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41592-023-02150-0%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Understanding%20metric-related%20pitfalls%20in%20image%20analysis%20validation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Baumgartner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Doreen%22%2C%22lastName%22%3A%22Heckmann-N%5Cu00f6tzel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22A.%20Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%20H.%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Acion%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michela%22%2C%22lastName%22%3A%22Antonelli%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tal%22%2C%22lastName%22%3A%22Arbel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arriel%22%2C%22lastName%22%3A%22Benis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Buettner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Jorge%22%2C%22lastName%22%3A%22Cardoso%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jianxu%22%2C%22lastName%22%3A%22Chen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beth%20A.%22%2C%22lastName%22%3A%22Cimini%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Keyvan%22%2C%22lastName%22%3A%22Farahani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Luciana%22%2C%22lastName%22%3A%22Ferrer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrian%22%2C%22lastName%22%3A%22Galdran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22van%20Ginneken%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Glocker%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%20A.%22%2C%22lastName%22%3A%22Hashimoto%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20M.%22%2C%22lastName%22%3A%22Hoffman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Merel%22%2C%22lastName%22%3A%22Huisman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Jannin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Charles%20E.%22%2C%22lastName%22%3A%22Kahn%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dagmar%22%2C%22lastName%22%3A%22Kainmueller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bernhard%22%2C%22lastName%22%3A%22Kainz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexandros%22%2C%22lastName%22%3A%22Karargyris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Kleesiek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Kofler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thijs%22%2C%22lastName%22%3A%22Kooi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%22%2C%22lastName%22%3A%22Kreshuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tahsin%22%2C%22lastName%22%3A%22Kurc%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%20A.%22%2C%22lastName%22%3A%22Landman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Geert%22%2C%22lastName%22%3A%22Litjens%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amin%22%2C%22lastName%22%3A%22Madani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anne%20L.%22%2C%22lastName%22%3A%22Martel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Erik%22%2C%22lastName%22%3A%22Meijering%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karel%20G.%20M.%22%2C%22lastName%22%3A%22Moons%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Henning%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brennan%22%2C%22lastName%22%3A%22Nichyporuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Nickel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Petersen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Susanne%20M.%22%2C%22lastName%22%3A%22Rafelski%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nasir%22%2C%22lastName%22%3A%22Rajpoot%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mauricio%22%2C%22lastName%22%3A%22Reyes%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20A.%22%2C%22lastName%22%3A%22Riegler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicola%22%2C%22lastName%22%3A%22Rieke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julio%22%2C%22lastName%22%3A%22Saez-Rodriguez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Clara%20I.%22%2C%22lastName%22%3A%22S%5Cu00e1nchez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shravya%22%2C%22lastName%22%3A%22Shetty%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ronald%20M.%22%2C%22lastName%22%3A%22Summers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdel%20A.%22%2C%22lastName%22%3A%22Taha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Aleksei%22%2C%22lastName%22%3A%22Tiulpin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sotirios%20A.%22%2C%22lastName%22%3A%22Tsaftaris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Van%20Calster%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ga%5Cu00ebl%22%2C%22lastName%22%3A%22Varoquaux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ziv%20R.%22%2C%22lastName%22%3A%22Yaniv%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%20F.%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Validation%20metrics%20are%20key%20for%20tracking%20scientific%20progress%20and%20bridging%20the%20current%20chasm%20between%20artificial%20intelligence%20research%20and%20its%20translation%20into%20practice.%20However%2C%20increasing%20evidence%20shows%20that%2C%20particularly%20in%20image%20analysis%2C%20metrics%20are%20often%20chosen%20inadequately.%20Although%20taking%20into%20account%20the%20individual%20strengths%2C%20weaknesses%20and%20limitations%20of%20validation%20metrics%20is%20a%20critical%20prerequisite%20to%20making%20educated%20choices%2C%20the%20relevant%20knowledge%20is%20currently%20scattered%20and%20poorly%20accessible%20to%20individual%20researchers.%20Based%20on%20a%20multistage%20Delphi%20process%20conducted%20by%20a%20multidisciplinary%20expert%20consortium%20as%20well%20as%20extensive%20community%20feedback%2C%20the%20present%20work%20provides%20a%20reliable%20and%20comprehensive%20common%20point%20of%20access%20to%20information%20on%20pitfalls%20related%20to%20validation%20metrics%20in%20image%20analysis.%20Although%20focused%20on%20biomedical%20image%20analysis%2C%20the%20addressed%20pitfalls%20generalize%20across%20application%20domains%20and%20are%20categorized%20according%20to%20a%20newly%20created%2C%20domain-agnostic%20taxonomy.%20The%20work%20serves%20to%20enhance%20global%20comprehension%20of%20a%20key%20topic%20in%20image%20analysis%20validation.%22%2C%22date%22%3A%222024-02-12%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1038%5C%2Fs41592-023-02150-0%22%2C%22ISSN%22%3A%221548-7105%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.nature.com%5C%2Farticles%5C%2Fs41592-023-02150-0%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-02-03T08%3A36%3A30Z%22%7D%7D%2C%7B%22key%22%3A%22CAUD4GCC%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Maier-Hein%20et%20al.%22%2C%22parsedDate%22%3A%222024-02%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BMaier-Hein%2C%20L.%2C%20Reinke%2C%20A.%2C%20Godau%2C%20P.%2C%20Tizabi%2C%20M.%20D.%2C%20Buettner%2C%20F.%2C%20Christodoulou%2C%20E.%2C%20Glocker%2C%20B.%2C%20Isensee%2C%20F.%2C%20Kleesiek%2C%20J.%2C%20Kozubek%2C%20M.%2C%20Reyes%2C%20M.%2C%20Riegler%2C%20M.%20A.%2C%20Wiesenfarth%2C%20M.%2C%20Kavur%2C%20A.%20E.%2C%20Sudre%2C%20C.%20H.%2C%20Baumgartner%2C%20M.%2C%20Eisenmann%2C%20M.%2C%20Heckmann-N%26%23xF6%3Btzel%2C%20D.%2C%20R%26%23xE4%3Bdsch%2C%20T.%2C%20%26%23x2026%3B%20J%26%23xE4%3Bger%2C%20P.%20F.%20%282024%29.%20%26lt%3Bb%26gt%3BMetrics%20reloaded%3A%20recommendations%20for%20image%20analysis%20validation%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BNature%20Methods%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B21%26lt%3B%5C%2Fi%26gt%3B%282%29%2C%20195%26%23x2013%3B212.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41592-023-02151-z%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41592-023-02151-z%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Metrics%20reloaded%3A%20recommendations%20for%20image%20analysis%20validation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Buettner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Glocker%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Kleesiek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mauricio%22%2C%22lastName%22%3A%22Reyes%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20A.%22%2C%22lastName%22%3A%22Riegler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuel%22%2C%22lastName%22%3A%22Wiesenfarth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22A.%20Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%20H.%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Baumgartner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Doreen%22%2C%22lastName%22%3A%22Heckmann-N%5Cu00f6tzel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Acion%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michela%22%2C%22lastName%22%3A%22Antonelli%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tal%22%2C%22lastName%22%3A%22Arbel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arriel%22%2C%22lastName%22%3A%22Benis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthew%20B.%22%2C%22lastName%22%3A%22Blaschko%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Jorge%22%2C%22lastName%22%3A%22Cardoso%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beth%20A.%22%2C%22lastName%22%3A%22Cimini%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gary%20S.%22%2C%22lastName%22%3A%22Collins%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Keyvan%22%2C%22lastName%22%3A%22Farahani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Luciana%22%2C%22lastName%22%3A%22Ferrer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrian%22%2C%22lastName%22%3A%22Galdran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22van%20Ginneken%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Robert%22%2C%22lastName%22%3A%22Haase%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%20A.%22%2C%22lastName%22%3A%22Hashimoto%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20M.%22%2C%22lastName%22%3A%22Hoffman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Merel%22%2C%22lastName%22%3A%22Huisman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Jannin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Charles%20E.%22%2C%22lastName%22%3A%22Kahn%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dagmar%22%2C%22lastName%22%3A%22Kainmueller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bernhard%22%2C%22lastName%22%3A%22Kainz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexandros%22%2C%22lastName%22%3A%22Karargyris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alan%22%2C%22lastName%22%3A%22Karthikesalingam%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Kofler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%22%2C%22lastName%22%3A%22Kreshuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tahsin%22%2C%22lastName%22%3A%22Kurc%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%20A.%22%2C%22lastName%22%3A%22Landman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Geert%22%2C%22lastName%22%3A%22Litjens%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amin%22%2C%22lastName%22%3A%22Madani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anne%20L.%22%2C%22lastName%22%3A%22Martel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Mattson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Erik%22%2C%22lastName%22%3A%22Meijering%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karel%20G.%20M.%22%2C%22lastName%22%3A%22Moons%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Henning%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brennan%22%2C%22lastName%22%3A%22Nichyporuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Nickel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Petersen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nasir%22%2C%22lastName%22%3A%22Rajpoot%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicola%22%2C%22lastName%22%3A%22Rieke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julio%22%2C%22lastName%22%3A%22Saez-Rodriguez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Clara%20I.%22%2C%22lastName%22%3A%22S%5Cu00e1nchez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shravya%22%2C%22lastName%22%3A%22Shetty%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maarten%22%2C%22lastName%22%3A%22van%20Smeden%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ronald%20M.%22%2C%22lastName%22%3A%22Summers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdel%20A.%22%2C%22lastName%22%3A%22Taha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Aleksei%22%2C%22lastName%22%3A%22Tiulpin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sotirios%20A.%22%2C%22lastName%22%3A%22Tsaftaris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Van%20Calster%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ga%5Cu00ebl%22%2C%22lastName%22%3A%22Varoquaux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%20F.%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%5D%2C%22abstractNote%22%3A%22Increasing%20evidence%20shows%20that%20flaws%20in%20machine%20learning%20%28ML%29%20algorithm%20validation%20are%20an%20underestimated%20global%20problem.%20In%20biomedical%20image%20analysis%2C%20chosen%20performance%20metrics%20often%20do%20not%20reflect%20the%20domain%20interest%2C%20and%20thus%20fail%20to%20adequately%20measure%20scientific%20progress%20and%20hinder%20translation%20of%20ML%20techniques%20into%20practice.%20To%20overcome%20this%2C%20we%20created%20Metrics%20Reloaded%2C%20a%20comprehensive%20framework%20guiding%20researchers%20in%20the%20problem-aware%20selection%20of%20metrics.%20Developed%20by%20a%20large%20international%20consortium%20in%20a%20multistage%20Delphi%20process%2C%20it%20is%20based%20on%20the%20novel%20concept%20of%20a%20problem%20fingerprint%5Cu2014a%20structured%20representation%20of%20the%20given%20problem%20that%20captures%20all%20aspects%20that%20are%20relevant%20for%20metric%20selection%2C%20from%20the%20domain%20interest%20to%20the%20properties%20of%20the%20target%20structure%28s%29%2C%20dataset%20and%20algorithm%20output.%20On%20the%20basis%20of%20the%20problem%20fingerprint%2C%20users%20are%20guided%20through%20the%20process%20of%20choosing%20and%20applying%20appropriate%20validation%20metrics%20while%20being%20made%20aware%20of%20potential%20pitfalls.%20Metrics%20Reloaded%20targets%20image%20analysis%20problems%20that%20can%20be%20interpreted%20as%20classification%20tasks%20at%20image%2C%20object%20or%20pixel%20level%2C%20namely%20image-level%20classification%2C%20object%20detection%2C%20semantic%20segmentation%20and%20instance%20segmentation%20tasks.%20To%20improve%20the%20user%20experience%2C%20we%20implemented%20the%20framework%20in%20the%20Metrics%20Reloaded%20online%20tool.%20Following%20the%20convergence%20of%20ML%20methodology%20across%20application%20domains%2C%20Metrics%20Reloaded%20fosters%20the%20convergence%20of%20validation%20methodology.%20Its%20applicability%20is%20demonstrated%20for%20various%20biomedical%20use%20cases.%22%2C%22date%22%3A%222024-02%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1038%5C%2Fs41592-023-02151-z%22%2C%22ISSN%22%3A%221548-7105%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.nature.com%5C%2Farticles%5C%2Fs41592-023-02151-z%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-02-03T08%3A34%3A55Z%22%7D%7D%2C%7B%22key%22%3A%22HB3HLARY%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Kofler%20et%20al.%22%2C%22parsedDate%22%3A%222023-12-05%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BKofler%2C%20F.%2C%20M%26%23xF6%3Bller%2C%20H.%2C%20Buchner%2C%20J.%20A.%2C%20de%20la%20Rosa%2C%20E.%2C%20Ezhov%2C%20I.%2C%20Rosier%2C%20M.%2C%20Mekki%2C%20I.%2C%20Shit%2C%20S.%2C%20Negwer%2C%20M.%2C%20Al-Maskari%2C%20R.%2C%20Ert%26%23xFC%3Brk%2C%20A.%2C%20Vinayahalingam%2C%20S.%2C%20Isensee%2C%20F.%2C%20Pati%2C%20S.%2C%20Rueckert%2C%20D.%2C%20Kirschke%2C%20J.%20S.%2C%20Ehrlich%2C%20S.%20K.%2C%20Reinke%2C%20A.%2C%20Menze%2C%20B.%2C%20%26%23x2026%3B%20Piraud%2C%20M.%20%282023%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BPanoptica%20--%20instance-wise%20evaluation%20of%203D%20semantic%20and%20instance%20segmentation%20maps%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2312.02608%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2312.02608%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2312.02608%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Panoptica%20--%20instance-wise%20evaluation%20of%203D%20semantic%20and%20instance%20segmentation%20maps%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Kofler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hendrik%22%2C%22lastName%22%3A%22M%5Cu00f6ller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Josef%20A.%22%2C%22lastName%22%3A%22Buchner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ezequiel%22%2C%22lastName%22%3A%22de%20la%20Rosa%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ivan%22%2C%22lastName%22%3A%22Ezhov%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marcel%22%2C%22lastName%22%3A%22Rosier%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Isra%22%2C%22lastName%22%3A%22Mekki%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Suprosanna%22%2C%22lastName%22%3A%22Shit%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Moritz%22%2C%22lastName%22%3A%22Negwer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rami%22%2C%22lastName%22%3A%22Al-Maskari%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ali%22%2C%22lastName%22%3A%22Ert%5Cu00fcrk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shankeeth%22%2C%22lastName%22%3A%22Vinayahalingam%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sarthak%22%2C%22lastName%22%3A%22Pati%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%22%2C%22lastName%22%3A%22Rueckert%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jan%20S.%22%2C%22lastName%22%3A%22Kirschke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stefan%20K.%22%2C%22lastName%22%3A%22Ehrlich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Benedikt%22%2C%22lastName%22%3A%22Wiestler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marie%22%2C%22lastName%22%3A%22Piraud%22%7D%5D%2C%22abstractNote%22%3A%22This%20paper%20introduces%20panoptica%2C%20a%20versatile%20and%20performance-optimized%20package%20designed%20for%20computing%20instance-wise%20segmentation%20quality%20metrics%20from%202D%20and%203D%20segmentation%20maps.%20panoptica%20addresses%20the%20limitations%20of%20existing%20metrics%20and%20provides%20a%20modular%20framework%20that%20complements%20the%20original%20intersection%20over%20union-based%20panoptic%20quality%20with%20other%20metrics%2C%20such%20as%20the%20distance%20metric%20Average%20Symmetric%20Surface%20Distance.%20The%20package%20is%20open-source%2C%20implemented%20in%20Python%2C%20and%20accompanied%20by%20comprehensive%20documentation%20and%20tutorials.%20panoptica%20employs%20a%20three-step%20metrics%20computation%20process%20to%20cover%20diverse%20use%20cases.%20The%20efficacy%20of%20panoptica%20is%20demonstrated%20on%20various%20real-world%20biomedical%20datasets%2C%20where%20an%20instance-wise%20evaluation%20is%20instrumental%20for%20an%20accurate%20representation%20of%20the%20underlying%20clinical%20task.%20Overall%2C%20we%20envision%20panoptica%20as%20a%20valuable%20tool%20facilitating%20in-depth%20evaluation%20of%20segmentation%20methods.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2312.02608%22%2C%22date%22%3A%222023-12-05%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2312.02608%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2312.02608%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-03-04T12%3A33%3A54Z%22%7D%7D%2C%7B%22key%22%3A%224JYLYRHF%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Brandenburg%20et%20al.%22%2C%22parsedDate%22%3A%222023-11%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBrandenburg%2C%20J.%20M.%2C%20Jenke%2C%20A.%20C.%2C%20Stern%2C%20A.%2C%20Daum%2C%20M.%20T.%20J.%2C%20Schulze%2C%20A.%2C%20Younis%2C%20R.%2C%20Petrynowski%2C%20P.%2C%20Davitashvili%2C%20T.%2C%20Vanat%2C%20V.%2C%20Bhasker%2C%20N.%2C%20Schneider%2C%20S.%2C%20M%26%23xFC%3Bndermann%2C%20L.%2C%20Reinke%2C%20A.%2C%20Kolbinger%2C%20F.%20R.%2C%20J%26%23xF6%3Brns%2C%20V.%2C%20Fritz-Kebede%2C%20F.%2C%20Dugas%2C%20M.%2C%20Maier-Hein%2C%20L.%2C%20Klotz%2C%20R.%2C%20%26%23x2026%3B%20Wagner%2C%20M.%20%282023%29.%20%26lt%3Bb%26gt%3BActive%20learning%20for%20extracting%20surgomic%20features%20in%20robot-assisted%20minimally%20invasive%20esophagectomy%3A%20a%20prospective%20annotation%20study%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BSurgical%20Endoscopy%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B37%26lt%3B%5C%2Fi%26gt%3B%2811%29%2C%208577%26%23x2013%3B8593.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs00464-023-10447-6%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs00464-023-10447-6%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Active%20learning%20for%20extracting%20surgomic%20features%20in%20robot-assisted%20minimally%20invasive%20esophagectomy%3A%20a%20prospective%20annotation%20study%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Johanna%20M.%22%2C%22lastName%22%3A%22Brandenburg%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexander%20C.%22%2C%22lastName%22%3A%22Jenke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Antonia%22%2C%22lastName%22%3A%22Stern%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marie%20T.%20J.%22%2C%22lastName%22%3A%22Daum%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Andr%5Cu00e9%22%2C%22lastName%22%3A%22Schulze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rayan%22%2C%22lastName%22%3A%22Younis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Philipp%22%2C%22lastName%22%3A%22Petrynowski%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tornike%22%2C%22lastName%22%3A%22Davitashvili%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vincent%22%2C%22lastName%22%3A%22Vanat%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nithya%22%2C%22lastName%22%3A%22Bhasker%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sophia%22%2C%22lastName%22%3A%22Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lars%22%2C%22lastName%22%3A%22M%5Cu00fcndermann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fiona%20R.%22%2C%22lastName%22%3A%22Kolbinger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vanessa%22%2C%22lastName%22%3A%22J%5Cu00f6rns%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fleur%22%2C%22lastName%22%3A%22Fritz-Kebede%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Dugas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rosa%22%2C%22lastName%22%3A%22Klotz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marius%22%2C%22lastName%22%3A%22Distler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22J%5Cu00fcrgen%22%2C%22lastName%22%3A%22Weitz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beat%20P.%22%2C%22lastName%22%3A%22M%5Cu00fcller-Stich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stefanie%22%2C%22lastName%22%3A%22Speidel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sebastian%22%2C%22lastName%22%3A%22Bodenstedt%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Wagner%22%7D%5D%2C%22abstractNote%22%3A%22BACKGROUND%3A%20With%20Surgomics%2C%20we%20aim%20for%20personalized%20prediction%20of%20the%20patient%26%23039%3Bs%20surgical%20outcome%20using%20machine-learning%20%28ML%29%20on%20multimodal%20intraoperative%20data%20to%20extract%20surgomic%20features%20as%20surgical%20process%20characteristics.%20As%20high-quality%20annotations%20by%20medical%20experts%20are%20crucial%2C%20but%20still%20a%20bottleneck%2C%20we%20prospectively%20investigate%20active%20learning%20%28AL%29%20to%20reduce%20annotation%20effort%20and%20present%20automatic%20recognition%20of%20surgomic%20features.%5CnMETHODS%3A%20To%20establish%20a%20process%20for%20development%20of%20surgomic%20features%2C%20ten%20video-based%20features%20related%20to%20bleeding%2C%20as%20highly%20relevant%20intraoperative%20complication%2C%20were%20chosen.%20They%20comprise%20the%20amount%20of%20blood%20and%20smoke%20in%20the%20surgical%20field%2C%20six%20instruments%2C%20and%20two%20anatomic%20structures.%20Annotation%20of%20selected%20frames%20from%20robot-assisted%20minimally%20invasive%20esophagectomies%20was%20performed%20by%20at%20least%20three%20independent%20medical%20experts.%20To%20test%20whether%20AL%20reduces%20annotation%20effort%2C%20we%20performed%20a%20prospective%20annotation%20study%20comparing%20AL%20with%20equidistant%20sampling%20%28EQS%29%20for%20frame%20selection.%20Multiple%20Bayesian%20ResNet18%20architectures%20were%20trained%20on%20a%20multicentric%20dataset%2C%20consisting%20of%2022%20videos%20from%20two%20centers.%5CnRESULTS%3A%20In%20total%2C%2014%2C004%20frames%20were%20tag%20annotated.%20A%20mean%20F1-score%20of%200.75%5Cu2009%5Cu00b1%5Cu20090.16%20was%20achieved%20for%20all%20features.%20The%20highest%20F1-score%20was%20achieved%20for%20the%20instruments%20%28mean%200.80%5Cu2009%5Cu00b1%5Cu20090.17%29.%20This%20result%20is%20also%20reflected%20in%20the%20inter-rater-agreement%20%281-rater-kappa%5Cu2009%26gt%3B%5Cu20090.82%29.%20Compared%20to%20EQS%2C%20AL%20showed%20better%20recognition%20results%20for%20the%20instruments%20with%20a%20significant%20difference%20in%20the%20McNemar%20test%20comparing%20correctness%20of%20predictions.%20Moreover%2C%20in%20contrast%20to%20EQS%2C%20AL%20selected%20more%20frames%20of%20the%20four%20less%20common%20instruments%20%281512%20vs.%20607%20frames%29%20and%20achieved%20higher%20F1-scores%20for%20common%20instruments%20while%20requiring%20less%20training%20frames.%5CnCONCLUSION%3A%20We%20presented%20ten%20surgomic%20features%20relevant%20for%20bleeding%20events%20in%20esophageal%20surgery%20automatically%20extracted%20from%20surgical%20video%20using%20ML.%20AL%20showed%20the%20potential%20to%20reduce%20annotation%20effort%20while%20keeping%20ML%20performance%20high%20for%20selected%20features.%20The%20source%20code%20and%20the%20trained%20models%20are%20published%5Cu00a0open%20source.%22%2C%22date%22%3A%222023-11%22%2C%22language%22%3A%22eng%22%2C%22DOI%22%3A%2210.1007%5C%2Fs00464-023-10447-6%22%2C%22ISSN%22%3A%221432-2218%22%2C%22url%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-03-04T13%3A52%3A11Z%22%7D%7D%2C%7B%22key%22%3A%22YCR73ZC9%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Tran%20et%20al.%22%2C%22parsedDate%22%3A%222023-07-01%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BTran%2C%20T.%20N.%2C%20Adler%2C%20T.%20J.%2C%20Yamlahi%2C%20A.%2C%20Christodoulou%2C%20E.%2C%20Godau%2C%20P.%2C%20Reinke%2C%20A.%2C%20Tizabi%2C%20M.%20D.%2C%20Sauer%2C%20P.%2C%20Persicke%2C%20T.%2C%20Albert%2C%20J.%20G.%2C%20%26amp%3B%20Maier-Hein%2C%20L.%20%282023%29.%20%26lt%3Bb%26gt%3BSources%20of%20performance%20variability%20in%20deep%20learning-based%20polyp%20detection%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BInternational%20Journal%20of%20Computer%20Assisted%20Radiology%20and%20Surgery%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B18%26lt%3B%5C%2Fi%26gt%3B%287%29%2C%201311%26%23x2013%3B1322.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs11548-023-02936-9%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs11548-023-02936-9%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Sources%20of%20performance%20variability%20in%20deep%20learning-based%20polyp%20detection%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22T.%20N.%22%2C%22lastName%22%3A%22Tran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22T.%20J.%22%2C%22lastName%22%3A%22Adler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22A.%22%2C%22lastName%22%3A%22Yamlahi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22E.%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22P.%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22A.%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22P.%22%2C%22lastName%22%3A%22Sauer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22T.%22%2C%22lastName%22%3A%22Persicke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22J.%20G.%22%2C%22lastName%22%3A%22Albert%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22L.%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Validation%20metrics%20are%20a%20key%20prerequisite%20for%20the%20reliable%20tracking%20of%20scientific%20progress%20and%20for%20deciding%20on%20the%20potential%20clinical%20translation%20of%20methods.%20While%20recent%20initiatives%20aim%20to%20develop%20comprehensive%20theoretical%20frameworks%20for%20understanding%20metric-related%20pitfalls%20in%20image%20analysis%20problems%2C%20there%20is%20a%20lack%20of%20experimental%20evidence%20on%20the%20concrete%20effects%20of%20common%20and%20rare%20pitfalls%20on%20specific%20applications.%20We%20address%20this%20gap%20in%20the%20literature%20in%20the%20context%20of%20colon%20cancer%20screening.%22%2C%22date%22%3A%222023-07-01%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1007%5C%2Fs11548-023-02936-9%22%2C%22ISSN%22%3A%221861-6429%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs11548-023-02936-9%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-03-04T14%3A47%3A17Z%22%7D%7D%2C%7B%22key%22%3A%22JS4P6WFE%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Wagner%20et%20al.%22%2C%22parsedDate%22%3A%222023-05-01%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BWagner%2C%20M.%2C%20M%26%23xFC%3Bller-Stich%2C%20B.-P.%2C%20Kisilenko%2C%20A.%2C%20Tran%2C%20D.%2C%20Heger%2C%20P.%2C%20M%26%23xFC%3Bndermann%2C%20L.%2C%20Lubotsky%2C%20D.%20M.%2C%20M%26%23xFC%3Bller%2C%20B.%2C%20Davitashvili%2C%20T.%2C%20Capek%2C%20M.%2C%20Reinke%2C%20A.%2C%20Reid%2C%20C.%2C%20Yu%2C%20T.%2C%20Vardazaryan%2C%20A.%2C%20Nwoye%2C%20C.%20I.%2C%20Padoy%2C%20N.%2C%20Liu%2C%20X.%2C%20Lee%2C%20E.-J.%2C%20Disch%2C%20C.%2C%20%26%23x2026%3B%20Bodenstedt%2C%20S.%20%282023%29.%20%26lt%3Bb%26gt%3BComparative%20validation%20of%20machine%20learning%20algorithms%20for%20surgical%20workflow%20and%20skill%20analysis%20with%20the%20HeiChole%20benchmark%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BMedical%20Image%20Analysis%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B86%26lt%3B%5C%2Fi%26gt%3B%2C%20102770.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2023.102770%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2023.102770%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Comparative%20validation%20of%20machine%20learning%20algorithms%20for%20surgical%20workflow%20and%20skill%20analysis%20with%20the%20HeiChole%20benchmark%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Wagner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beat-Peter%22%2C%22lastName%22%3A%22M%5Cu00fcller-Stich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%22%2C%22lastName%22%3A%22Kisilenko%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Duc%22%2C%22lastName%22%3A%22Tran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Heger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lars%22%2C%22lastName%22%3A%22M%5Cu00fcndermann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%20M%22%2C%22lastName%22%3A%22Lubotsky%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Benjamin%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tornike%22%2C%22lastName%22%3A%22Davitashvili%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuela%22%2C%22lastName%22%3A%22Capek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carissa%22%2C%22lastName%22%3A%22Reid%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tong%22%2C%22lastName%22%3A%22Yu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Armine%22%2C%22lastName%22%3A%22Vardazaryan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chinedu%20Innocent%22%2C%22lastName%22%3A%22Nwoye%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicolas%22%2C%22lastName%22%3A%22Padoy%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Xinyang%22%2C%22lastName%22%3A%22Liu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Eung-Joo%22%2C%22lastName%22%3A%22Lee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Constantin%22%2C%22lastName%22%3A%22Disch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hans%22%2C%22lastName%22%3A%22Meine%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tong%22%2C%22lastName%22%3A%22Xia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fucang%22%2C%22lastName%22%3A%22Jia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Satoshi%22%2C%22lastName%22%3A%22Kondo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wolfgang%22%2C%22lastName%22%3A%22Reiter%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yueming%22%2C%22lastName%22%3A%22Jin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yonghao%22%2C%22lastName%22%3A%22Long%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Meirui%22%2C%22lastName%22%3A%22Jiang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Qi%22%2C%22lastName%22%3A%22Dou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pheng%20Ann%22%2C%22lastName%22%3A%22Heng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Isabell%22%2C%22lastName%22%3A%22Twick%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kadir%22%2C%22lastName%22%3A%22Kirtac%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Enes%22%2C%22lastName%22%3A%22Hosgor%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jon%20Lindstr%5Cu00f6m%22%2C%22lastName%22%3A%22Bolmgren%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Stenzel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bj%5Cu00f6rn%22%2C%22lastName%22%3A%22von%20Siemens%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Long%22%2C%22lastName%22%3A%22Zhao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhenxiao%22%2C%22lastName%22%3A%22Ge%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Haiming%22%2C%22lastName%22%3A%22Sun%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Di%22%2C%22lastName%22%3A%22Xie%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mengqi%22%2C%22lastName%22%3A%22Guo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daochang%22%2C%22lastName%22%3A%22Liu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hannes%20G.%22%2C%22lastName%22%3A%22Kenngott%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Nickel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Moritz%20von%22%2C%22lastName%22%3A%22Frankenberg%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Franziska%22%2C%22lastName%22%3A%22Mathis-Ullrich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stefanie%22%2C%22lastName%22%3A%22Speidel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sebastian%22%2C%22lastName%22%3A%22Bodenstedt%22%7D%5D%2C%22abstractNote%22%3A%22Purpose%5CnSurgical%20workflow%20and%20skill%20analysis%20are%20key%20technologies%20for%20the%20next%20generation%20of%20cognitive%20surgical%20assistance%20systems.%20These%20systems%20could%20increase%20the%20safety%20of%20the%20operation%20through%20context-sensitive%20warnings%20and%20semi-autonomous%20robotic%20assistance%20or%20improve%20training%20of%20surgeons%20via%20data-driven%20feedback.%20In%20surgical%20workflow%20analysis%20up%20to%2091%25%20average%20precision%20has%20been%20reported%20for%20phase%20recognition%20on%20an%20open%20data%20single-center%20video%20dataset.%20In%20this%20work%20we%20investigated%20the%20generalizability%20of%20phase%20recognition%20algorithms%20in%20a%20multicenter%20setting%20including%20more%20difficult%20recognition%20tasks%20such%20as%20surgical%20action%20and%20surgical%20skill.%5CnMethods%5CnTo%20achieve%20this%20goal%2C%20a%20dataset%20with%2033%20laparoscopic%20cholecystectomy%20videos%20from%20three%20surgical%20centers%20with%20a%20total%20operation%20time%20of%2022%5Cu00a0h%20was%20created.%20Labels%20included%20framewise%20annotation%20of%20seven%20surgical%20phases%20with%20250%20phase%20transitions%2C%205514%20occurences%20of%20four%20surgical%20actions%2C%206980%20occurences%20of%2021%20surgical%20instruments%20from%20seven%20instrument%20categories%20and%20495%20skill%20classifications%20in%20five%20skill%20dimensions.%20The%20dataset%20was%20used%20in%20the%202019%20international%20Endoscopic%20Vision%20challenge%2C%20sub-challenge%20for%20surgical%20workflow%20and%20skill%20analysis.%20Here%2C%2012%20research%20teams%20trained%20and%20submitted%20their%20machine%20learning%20algorithms%20for%20recognition%20of%20phase%2C%20action%2C%20instrument%20and%5C%2For%20skill%20assessment.%5CnResults%5CnF1-scores%20were%20achieved%20for%20phase%20recognition%20between%2023.9%25%20and%2067.7%25%20%28n%5Cu00a0%3D%5Cu00a09%20teams%29%2C%20for%20instrument%20presence%20detection%20between%2038.5%25%20and%2063.8%25%20%28n%5Cu00a0%3D%5Cu00a08%20teams%29%2C%20but%20for%20action%20recognition%20only%20between%2021.8%25%20and%2023.3%25%20%28n%5Cu00a0%3D%5Cu00a05%20teams%29.%20The%20average%20absolute%20error%20for%20skill%20assessment%20was%200.78%20%28n%5Cu00a0%3D%5Cu00a01%20team%29.%5CnConclusion%5CnSurgical%20workflow%20and%20skill%20analysis%20are%20promising%20technologies%20to%20support%20the%20surgical%20team%2C%20but%20there%20is%20still%20room%20for%20improvement%2C%20as%20shown%20by%20our%20comparison%20of%20machine%20learning%20algorithms.%20This%20novel%20HeiChole%20benchmark%20can%20be%20used%20for%20comparable%20evaluation%20and%20validation%20of%20future%20work.%20In%20future%20studies%2C%20it%20is%20of%20utmost%20importance%20to%20create%20more%20open%2C%20high-quality%20datasets%20in%20order%20to%20allow%20the%20development%20of%20artificial%20intelligence%20and%20cognitive%20robotics%20in%20surgery.%22%2C%22date%22%3A%222023-05-01%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1016%5C%2Fj.media.2023.102770%22%2C%22ISSN%22%3A%221361-8415%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.sciencedirect.com%5C%2Fscience%5C%2Farticle%5C%2Fpii%5C%2FS1361841523000312%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-03-04T14%3A52%3A02Z%22%7D%7D%2C%7B%22key%22%3A%22IATVMD87%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Eisenmann%20et%20al.%22%2C%22parsedDate%22%3A%222023-03-30%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BEisenmann%2C%20M.%2C%20Reinke%2C%20A.%2C%20Weru%2C%20V.%2C%20Tizabi%2C%20M.%20D.%2C%20Isensee%2C%20F.%2C%20Adler%2C%20T.%20J.%2C%20Ali%2C%20S.%2C%20Andrearczyk%2C%20V.%2C%20Aubreville%2C%20M.%2C%20Baid%2C%20U.%2C%20Bakas%2C%20S.%2C%20Balu%2C%20N.%2C%20Bano%2C%20S.%2C%20Bernal%2C%20J.%2C%20Bodenstedt%2C%20S.%2C%20Casella%2C%20A.%2C%20Cheplygina%2C%20V.%2C%20Daum%2C%20M.%2C%20de%20Bruijne%2C%20M.%2C%20%26%23x2026%3B%20Maier-Hein%2C%20L.%20%282023%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BWhy%20is%20the%20winner%20the%20best%3F%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2303.17719%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2303.17719%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2303.17719%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Why%20is%20the%20winner%20the%20best%3F%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vivienn%22%2C%22lastName%22%3A%22Weru%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20Dietlinde%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%20J.%22%2C%22lastName%22%3A%22Adler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sharib%22%2C%22lastName%22%3A%22Ali%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vincent%22%2C%22lastName%22%3A%22Andrearczyk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Aubreville%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ujjwal%22%2C%22lastName%22%3A%22Baid%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Niranjan%22%2C%22lastName%22%3A%22Balu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sophia%22%2C%22lastName%22%3A%22Bano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jorge%22%2C%22lastName%22%3A%22Bernal%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sebastian%22%2C%22lastName%22%3A%22Bodenstedt%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alessandro%22%2C%22lastName%22%3A%22Casella%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marie%22%2C%22lastName%22%3A%22Daum%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marleen%22%2C%22lastName%22%3A%22de%20Bruijne%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrien%22%2C%22lastName%22%3A%22Depeursinge%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Reuben%22%2C%22lastName%22%3A%22Dorent%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jan%22%2C%22lastName%22%3A%22Egger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%20G.%22%2C%22lastName%22%3A%22Ellis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sandy%22%2C%22lastName%22%3A%22Engelhardt%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Melanie%22%2C%22lastName%22%3A%22Ganz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Noha%22%2C%22lastName%22%3A%22Ghatwary%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gabriel%22%2C%22lastName%22%3A%22Girard%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anubha%22%2C%22lastName%22%3A%22Gupta%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lasse%22%2C%22lastName%22%3A%22Hansen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kanako%22%2C%22lastName%22%3A%22Harada%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mattias%22%2C%22lastName%22%3A%22Heinrich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicholas%22%2C%22lastName%22%3A%22Heller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alessa%22%2C%22lastName%22%3A%22Hering%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arnaud%22%2C%22lastName%22%3A%22Huaulm%5Cu00e9%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Jannin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ali%20Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Old%5Cu0159ich%22%2C%22lastName%22%3A%22Kodym%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jianning%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hongwei%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jun%22%2C%22lastName%22%3A%22Ma%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carlos%22%2C%22lastName%22%3A%22Mart%5Cu00edn-Isla%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alison%22%2C%22lastName%22%3A%22Noble%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Valentin%22%2C%22lastName%22%3A%22Oreiller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicolas%22%2C%22lastName%22%3A%22Padoy%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sarthak%22%2C%22lastName%22%3A%22Pati%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kelly%22%2C%22lastName%22%3A%22Payette%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jonathan%22%2C%22lastName%22%3A%22Rafael-Pati%5Cu00f1o%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vivek%20Singh%22%2C%22lastName%22%3A%22Bawa%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stefanie%22%2C%22lastName%22%3A%22Speidel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%20H.%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kimberlin%22%2C%22lastName%22%3A%22van%20Wijnen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Wagner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Donglai%22%2C%22lastName%22%3A%22Wei%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amine%22%2C%22lastName%22%3A%22Yamlahi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Moi%20Hoon%22%2C%22lastName%22%3A%22Yap%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chun%22%2C%22lastName%22%3A%22Yuan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maximilian%22%2C%22lastName%22%3A%22Zenk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Aneeq%22%2C%22lastName%22%3A%22Zia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Zimmerer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dogu%20Baran%22%2C%22lastName%22%3A%22Aydogan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Binod%22%2C%22lastName%22%3A%22Bhattarai%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Louise%22%2C%22lastName%22%3A%22Bloch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Raphael%22%2C%22lastName%22%3A%22Br%5Cu00fcngel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jihoon%22%2C%22lastName%22%3A%22Cho%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chanyeol%22%2C%22lastName%22%3A%22Choi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Qi%22%2C%22lastName%22%3A%22Dou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ivan%22%2C%22lastName%22%3A%22Ezhov%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Christoph%20M.%22%2C%22lastName%22%3A%22Friedrich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Clifton%22%2C%22lastName%22%3A%22Fuller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rebati%20Raman%22%2C%22lastName%22%3A%22Gaire%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrian%22%2C%22lastName%22%3A%22Galdran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22%5Cu00c1lvaro%20Garc%5Cu00eda%22%2C%22lastName%22%3A%22Faura%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maria%22%2C%22lastName%22%3A%22Grammatikopoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22SeulGi%22%2C%22lastName%22%3A%22Hong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mostafa%22%2C%22lastName%22%3A%22Jahanifar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ikbeom%22%2C%22lastName%22%3A%22Jang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdolrahim%22%2C%22lastName%22%3A%22Kadkhodamohammadi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Inha%22%2C%22lastName%22%3A%22Kang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Kofler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Satoshi%22%2C%22lastName%22%3A%22Kondo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hugo%22%2C%22lastName%22%3A%22Kuijf%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mingxing%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minh%20Huan%22%2C%22lastName%22%3A%22Luu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Toma%5Cu017e%22%2C%22lastName%22%3A%22Martin%5Cu010di%5Cu010d%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pedro%22%2C%22lastName%22%3A%22Morais%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mohamed%20A.%22%2C%22lastName%22%3A%22Naser%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bruno%22%2C%22lastName%22%3A%22Oliveira%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Owen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Subeen%22%2C%22lastName%22%3A%22Pang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jinah%22%2C%22lastName%22%3A%22Park%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sung-Hong%22%2C%22lastName%22%3A%22Park%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Szymon%22%2C%22lastName%22%3A%22P%5Cu0142otka%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Elodie%22%2C%22lastName%22%3A%22Puybareau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nasir%22%2C%22lastName%22%3A%22Rajpoot%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kanghyun%22%2C%22lastName%22%3A%22Ryu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Numan%22%2C%22lastName%22%3A%22Saeed%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adam%22%2C%22lastName%22%3A%22Shephard%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pengcheng%22%2C%22lastName%22%3A%22Shi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dejan%22%2C%22lastName%22%3A%22%5Cu0160tepec%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ronast%22%2C%22lastName%22%3A%22Subedi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guillaume%22%2C%22lastName%22%3A%22Tochon%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Helena%20R.%22%2C%22lastName%22%3A%22Torres%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Helene%22%2C%22lastName%22%3A%22Urien%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jo%5Cu00e3o%20L.%22%2C%22lastName%22%3A%22Vila%5Cu00e7a%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kareem%20Abdul%22%2C%22lastName%22%3A%22Wahid%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Haojie%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jiacheng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Liansheng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Xiyue%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Benedikt%22%2C%22lastName%22%3A%22Wiestler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marek%22%2C%22lastName%22%3A%22Wodzinski%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fangfang%22%2C%22lastName%22%3A%22Xia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Juanying%22%2C%22lastName%22%3A%22Xie%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhiwei%22%2C%22lastName%22%3A%22Xiong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sen%22%2C%22lastName%22%3A%22Yang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yanwu%22%2C%22lastName%22%3A%22Yang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zixuan%22%2C%22lastName%22%3A%22Zhao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%20F.%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22International%20benchmarking%20competitions%20have%20become%20fundamental%20for%20the%20comparative%20performance%20assessment%20of%20image%20analysis%20methods.%20However%2C%20little%20attention%20has%20been%20given%20to%20investigating%20what%20can%20be%20learnt%20from%20these%20competitions.%20Do%20they%20really%20generate%20scientific%20progress%3F%20What%20are%20common%20and%20successful%20participation%20strategies%3F%20What%20makes%20a%20solution%20superior%20to%20a%20competing%20method%3F%20To%20address%20this%20gap%20in%20the%20literature%2C%20we%20performed%20a%20multi-center%20study%20with%20all%2080%20competitions%20that%20were%20conducted%20in%20the%20scope%20of%20IEEE%20ISBI%202021%20and%20MICCAI%202021.%20Statistical%20analyses%20performed%20based%20on%20comprehensive%20descriptions%20of%20the%20submitted%20algorithms%20linked%20to%20their%20rank%20as%20well%20as%20the%20underlying%20participation%20strategies%20revealed%20common%20characteristics%20of%20winning%20solutions.%20These%20typically%20include%20the%20use%20of%20multi-task%20learning%20%2863%25%29%20and%5C%2For%20multi-stage%20pipelines%20%2861%25%29%2C%20and%20a%20focus%20on%20augmentation%20%28100%25%29%2C%20image%20preprocessing%20%2897%25%29%2C%20data%20curation%20%2879%25%29%2C%20and%20postprocessing%20%2866%25%29.%20The%20%26quot%3Btypical%26quot%3B%20lead%20of%20a%20winning%20team%20is%20a%20computer%20scientist%20with%20a%20doctoral%20degree%2C%20five%20years%20of%20experience%20in%20biomedical%20image%20analysis%2C%20and%20four%20years%20of%20experience%20in%20deep%20learning.%20Two%20core%20general%20development%20strategies%20stood%20out%20for%20highly-ranked%20teams%3A%20the%20reflection%20of%20the%20metrics%20in%20the%20method%20design%20and%20the%20focus%20on%20analyzing%20and%20handling%20failure%20cases.%20According%20to%20the%20organizers%2C%2043%25%20of%20the%20winning%20algorithms%20exceeded%20the%20state%20of%20the%20art%20but%20only%2011%25%20completely%20solved%20the%20respective%20domain%20problem.%20The%20insights%20of%20our%20study%20could%20help%20researchers%20%281%29%20improve%20algorithm%20development%20strategies%20when%20approaching%20new%20problems%2C%20and%20%282%29%20focus%20on%20open%20research%20questions%20revealed%20by%20this%20work.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2303.17719%22%2C%22date%22%3A%222023-03-30%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2303.17719%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2303.17719%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222023-04-03T11%3A36%3A59Z%22%7D%7D%2C%7B%22key%22%3A%22Z2JQTA45%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22lastModifiedByUser%22%3A%7B%22id%22%3A9737705%2C%22username%22%3A%22Niebelsa%22%2C%22name%22%3A%22%22%2C%22links%22%3A%7B%22alternate%22%3A%7B%22href%22%3A%22https%3A%5C%2F%5C%2Fwww.zotero.org%5C%2Fniebelsa%22%2C%22type%22%3A%22text%5C%2Fhtml%22%7D%7D%7D%2C%22creatorSummary%22%3A%22R%5Cu00e4dsch%20et%20al.%22%2C%22parsedDate%22%3A%222023-03-02%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BR%26%23xE4%3Bdsch%2C%20T.%2C%20Reinke%2C%20A.%2C%20Weru%2C%20V.%2C%20Tizabi%2C%20M.%20D.%2C%20Schreck%2C%20N.%2C%20Kavur%2C%20A.%20E.%2C%20Pekdemir%2C%20B.%2C%20Ro%26%23xDF%3B%2C%20T.%2C%20Kopp-Schneider%2C%20A.%2C%20%26amp%3B%20Maier-Hein%2C%20L.%20%282023%29.%20%26lt%3Bb%26gt%3BLabelling%20instructions%20matter%20in%20biomedical%20image%20analysis%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BNature%20Machine%20Intelligence%26lt%3B%5C%2Fi%26gt%3B%2C%201%26%23x2013%3B11.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs42256-023-00625-5%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs42256-023-00625-5%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Labelling%20instructions%20matter%20in%20biomedical%20image%20analysis%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vivienn%22%2C%22lastName%22%3A%22Weru%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicholas%22%2C%22lastName%22%3A%22Schreck%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22A.%20Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22B%5Cu00fcnyamin%22%2C%22lastName%22%3A%22Pekdemir%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tobias%22%2C%22lastName%22%3A%22Ro%5Cu00df%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Biomedical%20image%20analysis%20algorithm%20validation%20depends%20on%20high-quality%20annotation%20of%20reference%20datasets%2C%20for%20which%20labelling%20instructions%20are%20key.%20Despite%20their%20importance%2C%20their%20optimization%20remains%20largely%20unexplored.%20Here%20we%20present%20a%20systematic%20study%20of%20labelling%20instructions%20and%20their%20impact%20on%20annotation%20quality%20in%20the%20field.%20Through%20comprehensive%20examination%20of%20professional%20practice%20and%20international%20competitions%20registered%20at%20the%20Medical%20Image%20Computing%20and%20Computer%20Assisted%20Intervention%20Society%2C%20the%20largest%20international%20society%20in%20the%20biomedical%20imaging%20field%2C%20we%20uncovered%20a%20discrepancy%20between%20annotators%5Cu2019%20needs%20for%20labelling%20instructions%20and%20their%20current%20quality%20and%20availability.%20On%20the%20basis%20of%20an%20analysis%20of%2014%2C040%20images%20annotated%20by%20156%20annotators%20from%20four%20professional%20annotation%20companies%20and%20708%20Amazon%20Mechanical%20Turk%20crowdworkers%20using%20instructions%20with%20different%20information%20density%20levels%2C%20we%20further%20found%20that%20including%20exemplary%20images%20substantially%20boosts%20annotation%20performance%20compared%20with%20text-only%20descriptions%2C%20while%20solely%20extending%20text%20descriptions%20does%20not.%20Finally%2C%20professional%20annotators%20constantly%20outperform%20Amazon%20Mechanical%20Turk%20crowdworkers.%20Our%20study%20raises%20awareness%20for%20the%20need%20of%20quality%20standards%20in%20biomedical%20image%20analysis%20labelling%20instructions.%22%2C%22date%22%3A%222023-03-02%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1038%5C%2Fs42256-023-00625-5%22%2C%22ISSN%22%3A%222522-5839%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.nature.com%5C%2Farticles%5C%2Fs42256-023-00625-5%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-03-04T12%3A48%3A11Z%22%7D%7D%2C%7B%22key%22%3A%22Y7MNSG2Q%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Ro%5Cu00df%20et%20al.%22%2C%22parsedDate%22%3A%222023-03-01%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BRo%26%23xDF%3B%2C%20T.%2C%20Bruno%2C%20P.%2C%20Reinke%2C%20A.%2C%20Wiesenfarth%2C%20M.%2C%20Koeppel%2C%20L.%2C%20Full%2C%20P.%20M.%2C%20Pekdemir%2C%20B.%2C%20Godau%2C%20P.%2C%20Trofimova%2C%20D.%2C%20Isensee%2C%20F.%2C%20Adler%2C%20T.%20J.%2C%20Tran%2C%20T.%20N.%2C%20Moccia%2C%20S.%2C%20Calimeri%2C%20F.%2C%20M%26%23xFC%3Bller-Stich%2C%20B.%20P.%2C%20Kopp-Schneider%2C%20A.%2C%20%26amp%3B%20Maier-Hein%2C%20L.%20%282023%29.%20%26lt%3Bb%26gt%3BBeyond%20rankings%3A%20Learning%20%28more%29%20from%20algorithm%20validation%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BMedical%20Image%20Analysis%26lt%3B%5C%2Fi%26gt%3B%2C%20102765.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2023.102765%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2023.102765%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Beyond%20rankings%3A%20Learning%20%28more%29%20from%20algorithm%20validation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tobias%22%2C%22lastName%22%3A%22Ro%5Cu00df%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierangela%22%2C%22lastName%22%3A%22Bruno%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuel%22%2C%22lastName%22%3A%22Wiesenfarth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lisa%22%2C%22lastName%22%3A%22Koeppel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%20M.%22%2C%22lastName%22%3A%22Full%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22B%5Cu00fcnyamin%22%2C%22lastName%22%3A%22Pekdemir%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Darya%22%2C%22lastName%22%3A%22Trofimova%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%20J.%22%2C%22lastName%22%3A%22Adler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thuy%20N.%22%2C%22lastName%22%3A%22Tran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sara%22%2C%22lastName%22%3A%22Moccia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Francesco%22%2C%22lastName%22%3A%22Calimeri%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beat%20P.%22%2C%22lastName%22%3A%22M%5Cu00fcller-Stich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Challenges%20have%20become%20the%20state-of-the-art%20approach%20to%20benchmark%20image%20analysis%20algorithms%20in%20a%20comparative%20manner.%20While%20the%20validation%20on%20identical%20data%20sets%20was%20a%20great%20step%20forward%2C%20results%20analysis%20is%20often%20restricted%20to%20pure%20ranking%20tables%2C%20leaving%20relevant%20questions%20unanswered.%20Specifically%2C%20little%20effort%20has%20been%20put%20into%20the%20systematic%20investigation%20on%20what%20characterizes%20images%20in%20which%20state-of-the-art%20algorithms%20fail.%20To%20address%20this%20gap%20in%20the%20literature%2C%20we%20%281%29%20present%20a%20statistical%20framework%20for%20learning%20from%20challenges%20and%20%282%29%20instantiate%20it%20for%20the%20specific%20task%20of%20instrument%20instance%20segmentation%20in%20laparoscopic%20videos.%20Our%20framework%20relies%20on%20the%20semantic%20meta%20data%20annotation%20of%20images%2C%20which%20serves%20as%20foundation%20for%20a%20General%20Linear%20Mixed%20Models%20%28GLMM%29%20analysis.%20Based%20on%2051%2C542%20meta%20data%20annotations%20performed%20on%202%2C728%20images%2C%20we%20applied%20our%20approach%20to%20the%20results%20of%20the%20Robust%20Medical%20Instrument%20Segmentation%20Challenge%20%28ROBUST-MIS%29%20challenge%202019%20and%20revealed%20underexposure%2C%20motion%20and%20occlusion%20of%20instruments%20as%20well%20as%20the%20presence%20of%20smoke%20or%20other%20objects%20in%20the%20background%20as%20major%20sources%20of%20algorithm%20failure.%20Our%20subsequent%20method%20development%2C%20tailored%20to%20the%20specific%20remaining%20issues%2C%20yielded%20a%20deep%20learning%20model%20with%20state-of-the-art%20overall%20performance%20and%20specific%20strengths%20in%20the%20processing%20of%20images%20in%20which%20previous%20methods%20tended%20to%20fail.%20Due%20to%20the%20objectivity%20and%20generic%20applicability%20of%20our%20approach%2C%20it%20could%20become%20a%20valuable%20tool%20for%20validation%20in%20the%20field%20of%20medical%20image%20analysis%20and%20beyond.%22%2C%22date%22%3A%222023-03-01%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1016%5C%2Fj.media.2023.102765%22%2C%22ISSN%22%3A%221361-8415%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.sciencedirect.com%5C%2Fscience%5C%2Farticle%5C%2Fpii%5C%2FS1361841523000269%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222023-03-02T15%3A52%3A01Z%22%7D%7D%2C%7B%22key%22%3A%22VK4W7V63%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22lastModifiedByUser%22%3A%7B%22id%22%3A9737705%2C%22username%22%3A%22Niebelsa%22%2C%22name%22%3A%22%22%2C%22links%22%3A%7B%22alternate%22%3A%7B%22href%22%3A%22https%3A%5C%2F%5C%2Fwww.zotero.org%5C%2Fniebelsa%22%2C%22type%22%3A%22text%5C%2Fhtml%22%7D%7D%7D%2C%22creatorSummary%22%3A%22Godau%20et%20al.%22%2C%22parsedDate%22%3A%222023%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BGodau%2C%20P.%2C%20Kalinowski%2C%20P.%2C%20Christodoulou%2C%20E.%2C%20Reinke%2C%20A.%2C%20Tizabi%2C%20M.%2C%20Ferrer%2C%20L.%2C%20J%26%23xE4%3Bger%2C%20P.%2C%20%26amp%3B%20Maier-Hein%2C%20L.%20%282023%29.%20%26lt%3Bb%26gt%3BDeployment%20of%20Image%20Analysis%20Algorithms%20under%20Prevalence%20Shifts%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FARXIV.2303.12540%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FARXIV.2303.12540%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Deployment%20of%20Image%20Analysis%20Algorithms%20under%20Prevalence%20Shifts%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Piotr%22%2C%22lastName%22%3A%22Kalinowski%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Luciana%22%2C%22lastName%22%3A%22Ferrer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Domain%20gaps%20are%20among%20the%20most%20relevant%20roadblocks%20in%20the%20clinical%20translation%20of%20machine%20learning%20%28ML%29-based%20solutions%20for%20medical%20image%20analysis.%20While%20current%20research%20focuses%20on%20new%20training%20paradigms%20and%20network%20architectures%2C%20little%20attention%20is%20given%20to%20the%20specific%20effect%20of%20prevalence%20shifts%20on%20an%20algorithm%20deployed%20in%20practice.%20Such%20discrepancies%20between%20class%20frequencies%20in%20the%20data%20used%20for%20a%20method%26%23039%3Bs%20development%5C%2Fvalidation%20and%20that%20in%20its%20deployment%20environment%28s%29%20are%20of%20great%20importance%2C%20for%20example%20in%20the%20context%20of%20artificial%20intelligence%20%28AI%29%20democratization%2C%20as%20disease%20prevalences%20may%20vary%20widely%20across%20time%20and%20location.%20Our%20contribution%20is%20twofold.%20First%2C%20we%20empirically%20demonstrate%20the%20potentially%20severe%20consequences%20of%20missing%20prevalence%20handling%20by%20analyzing%20%28i%29%20the%20extent%20of%20miscalibration%2C%20%28ii%29%20the%20deviation%20of%20the%20decision%20threshold%20from%20the%20optimum%2C%20and%20%28iii%29%20the%20ability%20of%20validation%20metrics%20to%20reflect%20neural%20network%20performance%20on%20the%20deployment%20population%20as%20a%20function%20of%20the%20discrepancy%20between%20development%20and%20deployment%20prevalence.%20Second%2C%20we%20propose%20a%20workflow%20for%20prevalence-aware%20image%20classification%20that%20uses%20estimated%20deployment%20prevalences%20to%20adjust%20a%20trained%20classifier%20to%20a%20new%20environment%2C%20without%20requiring%20additional%20annotated%20deployment%20data.%20Comprehensive%20experiments%20based%20on%20a%20diverse%20set%20of%2030%20medical%20classification%20tasks%20showcase%20the%20benefit%20of%20the%20proposed%20workflow%20in%20generating%20better%20classifier%20decisions%20and%20more%20reliable%20performance%20estimates%20compared%20to%20current%20practice.%22%2C%22date%22%3A%222023%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.48550%5C%2FARXIV.2303.12540%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2303.12540%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-03-04T13%3A57%3A25Z%22%7D%7D%2C%7B%22key%22%3A%22EJK7AA8A%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22lastModifiedByUser%22%3A%7B%22id%22%3A9737705%2C%22username%22%3A%22Niebelsa%22%2C%22name%22%3A%22%22%2C%22links%22%3A%7B%22alternate%22%3A%7B%22href%22%3A%22https%3A%5C%2F%5C%2Fwww.zotero.org%5C%2Fniebelsa%22%2C%22type%22%3A%22text%5C%2Fhtml%22%7D%7D%7D%2C%22creatorSummary%22%3A%22Eisenmann%20et%20al.%22%2C%22parsedDate%22%3A%222022-12-16%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BEisenmann%2C%20M.%2C%20Reinke%2C%20A.%2C%20Weru%2C%20V.%2C%20Tizabi%2C%20M.%20D.%2C%20Isensee%2C%20F.%2C%20Adler%2C%20T.%20J.%2C%20Godau%2C%20P.%2C%20Cheplygina%2C%20V.%2C%20Kozubek%2C%20M.%2C%20Ali%2C%20S.%2C%20Gupta%2C%20A.%2C%20Kybic%2C%20J.%2C%20Noble%2C%20A.%2C%20de%20Sol%26%23xF3%3Brzano%2C%20C.%20O.%2C%20Pachade%2C%20S.%2C%20Petitjean%2C%20C.%2C%20Sage%2C%20D.%2C%20Wei%2C%20D.%2C%20Wilden%2C%20E.%2C%20%26%23x2026%3B%20Maier-Hein%2C%20L.%20%282022%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BBiomedical%20image%20analysis%20competitions%3A%20The%20state%20of%20current%20participation%20practice%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2212.08568%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2212.08568%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2212.08568%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Biomedical%20image%20analysis%20competitions%3A%20The%20state%20of%20current%20participation%20practice%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vivienn%22%2C%22lastName%22%3A%22Weru%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20Dietlinde%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%20J.%22%2C%22lastName%22%3A%22Adler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sharib%22%2C%22lastName%22%3A%22Ali%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anubha%22%2C%22lastName%22%3A%22Gupta%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jan%22%2C%22lastName%22%3A%22Kybic%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alison%22%2C%22lastName%22%3A%22Noble%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carlos%20Ortiz%22%2C%22lastName%22%3A%22de%20Sol%5Cu00f3rzano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Samiksha%22%2C%22lastName%22%3A%22Pachade%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Caroline%22%2C%22lastName%22%3A%22Petitjean%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%22%2C%22lastName%22%3A%22Sage%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Donglai%22%2C%22lastName%22%3A%22Wei%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Elizabeth%22%2C%22lastName%22%3A%22Wilden%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Deepak%22%2C%22lastName%22%3A%22Alapatt%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vincent%22%2C%22lastName%22%3A%22Andrearczyk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ujjwal%22%2C%22lastName%22%3A%22Baid%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Niranjan%22%2C%22lastName%22%3A%22Balu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sophia%22%2C%22lastName%22%3A%22Bano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vivek%20Singh%22%2C%22lastName%22%3A%22Bawa%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jorge%22%2C%22lastName%22%3A%22Bernal%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sebastian%22%2C%22lastName%22%3A%22Bodenstedt%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alessandro%22%2C%22lastName%22%3A%22Casella%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jinwook%22%2C%22lastName%22%3A%22Choi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Olivier%22%2C%22lastName%22%3A%22Commowick%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marie%22%2C%22lastName%22%3A%22Daum%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrien%22%2C%22lastName%22%3A%22Depeursinge%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Reuben%22%2C%22lastName%22%3A%22Dorent%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jan%22%2C%22lastName%22%3A%22Egger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hannah%22%2C%22lastName%22%3A%22Eichhorn%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sandy%22%2C%22lastName%22%3A%22Engelhardt%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Melanie%22%2C%22lastName%22%3A%22Ganz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gabriel%22%2C%22lastName%22%3A%22Girard%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lasse%22%2C%22lastName%22%3A%22Hansen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mattias%22%2C%22lastName%22%3A%22Heinrich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicholas%22%2C%22lastName%22%3A%22Heller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alessa%22%2C%22lastName%22%3A%22Hering%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arnaud%22%2C%22lastName%22%3A%22Huaulm%5Cu00e9%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hyunjeong%22%2C%22lastName%22%3A%22Kim%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%22%2C%22lastName%22%3A%22Landman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hongwei%20Bran%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jianning%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jun%22%2C%22lastName%22%3A%22Ma%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anne%22%2C%22lastName%22%3A%22Martel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carlos%22%2C%22lastName%22%3A%22Mart%5Cu00edn-Isla%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chinedu%20Innocent%22%2C%22lastName%22%3A%22Nwoye%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Valentin%22%2C%22lastName%22%3A%22Oreiller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicolas%22%2C%22lastName%22%3A%22Padoy%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sarthak%22%2C%22lastName%22%3A%22Pati%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kelly%22%2C%22lastName%22%3A%22Payette%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kimberlin%22%2C%22lastName%22%3A%22van%20Wijnen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Armine%22%2C%22lastName%22%3A%22Vardazaryan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tom%22%2C%22lastName%22%3A%22Vercauteren%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Wagner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chuanbo%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Moi%20Hoon%22%2C%22lastName%22%3A%22Yap%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zeyun%22%2C%22lastName%22%3A%22Yu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chun%22%2C%22lastName%22%3A%22Yuan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maximilian%22%2C%22lastName%22%3A%22Zenk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Aneeq%22%2C%22lastName%22%3A%22Zia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Zimmerer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rina%22%2C%22lastName%22%3A%22Bao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chanyeol%22%2C%22lastName%22%3A%22Choi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Andrew%22%2C%22lastName%22%3A%22Cohen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Oleh%22%2C%22lastName%22%3A%22Dzyubachyk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrian%22%2C%22lastName%22%3A%22Galdran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tianyuan%22%2C%22lastName%22%3A%22Gan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tianqi%22%2C%22lastName%22%3A%22Guo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pradyumna%22%2C%22lastName%22%3A%22Gupta%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mahmood%22%2C%22lastName%22%3A%22Haithami%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Edward%22%2C%22lastName%22%3A%22Ho%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ikbeom%22%2C%22lastName%22%3A%22Jang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhili%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhengbo%22%2C%22lastName%22%3A%22Luo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Filip%22%2C%22lastName%22%3A%22Lux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sokratis%22%2C%22lastName%22%3A%22Makrogiannis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dominik%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Young-tack%22%2C%22lastName%22%3A%22Oh%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Subeen%22%2C%22lastName%22%3A%22Pang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Constantin%22%2C%22lastName%22%3A%22Pape%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gorkem%22%2C%22lastName%22%3A%22Polat%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Charlotte%20Rosalie%22%2C%22lastName%22%3A%22Reed%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kanghyun%22%2C%22lastName%22%3A%22Ryu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22Scherr%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vajira%22%2C%22lastName%22%3A%22Thambawita%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Haoyu%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Xinliang%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kele%22%2C%22lastName%22%3A%22Xu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hung%22%2C%22lastName%22%3A%22Yeh%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Doyeob%22%2C%22lastName%22%3A%22Yeo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yixuan%22%2C%22lastName%22%3A%22Yuan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yan%22%2C%22lastName%22%3A%22Zeng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Xin%22%2C%22lastName%22%3A%22Zhao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julian%22%2C%22lastName%22%3A%22Abbing%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jannes%22%2C%22lastName%22%3A%22Adam%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nagesh%22%2C%22lastName%22%3A%22Adluru%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Niklas%22%2C%22lastName%22%3A%22Agethen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Salman%22%2C%22lastName%22%3A%22Ahmed%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yasmina%20Al%22%2C%22lastName%22%3A%22Khalil%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mireia%22%2C%22lastName%22%3A%22Aleny%5Cu00e0%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Esa%22%2C%22lastName%22%3A%22Alhoniemi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chengyang%22%2C%22lastName%22%3A%22An%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Talha%22%2C%22lastName%22%3A%22Anwar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tewodros%20Weldebirhan%22%2C%22lastName%22%3A%22Arega%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Netanell%22%2C%22lastName%22%3A%22Avisdris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dogu%20Baran%22%2C%22lastName%22%3A%22Aydogan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yingbin%22%2C%22lastName%22%3A%22Bai%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maria%20Baldeon%22%2C%22lastName%22%3A%22Calisto%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Berke%20Doga%22%2C%22lastName%22%3A%22Basaran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marcel%22%2C%22lastName%22%3A%22Beetz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Cheng%22%2C%22lastName%22%3A%22Bian%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hao%22%2C%22lastName%22%3A%22Bian%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kevin%22%2C%22lastName%22%3A%22Blansit%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Louise%22%2C%22lastName%22%3A%22Bloch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Robert%22%2C%22lastName%22%3A%22Bohnsack%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sara%22%2C%22lastName%22%3A%22Bosticardo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jack%22%2C%22lastName%22%3A%22Breen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mikael%22%2C%22lastName%22%3A%22Brudfors%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Raphael%22%2C%22lastName%22%3A%22Br%5Cu00fcngel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mariano%22%2C%22lastName%22%3A%22Cabezas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alberto%22%2C%22lastName%22%3A%22Cacciola%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhiwei%22%2C%22lastName%22%3A%22Chen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yucong%22%2C%22lastName%22%3A%22Chen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%20Tianming%22%2C%22lastName%22%3A%22Chen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minjeong%22%2C%22lastName%22%3A%22Cho%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Min-Kook%22%2C%22lastName%22%3A%22Choi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chuantao%20Xie%20Chuantao%22%2C%22lastName%22%3A%22Xie%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dana%22%2C%22lastName%22%3A%22Cobzas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julien%22%2C%22lastName%22%3A%22Cohen-Adad%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jorge%20Corral%22%2C%22lastName%22%3A%22Acero%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sujit%20Kumar%22%2C%22lastName%22%3A%22Das%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marcela%22%2C%22lastName%22%3A%22de%20Oliveira%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hanqiu%22%2C%22lastName%22%3A%22Deng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Guiming%22%2C%22lastName%22%3A%22Dong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lars%22%2C%22lastName%22%3A%22Doorenbos%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Cory%22%2C%22lastName%22%3A%22Efird%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Di%22%2C%22lastName%22%3A%22Fan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mehdi%20Fatan%22%2C%22lastName%22%3A%22Serj%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexandre%22%2C%22lastName%22%3A%22Fenneteau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lucas%22%2C%22lastName%22%3A%22Fidon%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patryk%22%2C%22lastName%22%3A%22Filipiak%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ren%5Cu00e9%22%2C%22lastName%22%3A%22Finzel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nuno%20R.%22%2C%22lastName%22%3A%22Freitas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Christoph%20M.%22%2C%22lastName%22%3A%22Friedrich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mitchell%22%2C%22lastName%22%3A%22Fulton%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Finn%22%2C%22lastName%22%3A%22Gaida%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Francesco%22%2C%22lastName%22%3A%22Galati%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Christoforos%22%2C%22lastName%22%3A%22Galazis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chang%20Hee%22%2C%22lastName%22%3A%22Gan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zheyao%22%2C%22lastName%22%3A%22Gao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shengbo%22%2C%22lastName%22%3A%22Gao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matej%22%2C%22lastName%22%3A%22Gazda%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beerend%22%2C%22lastName%22%3A%22Gerats%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Neil%22%2C%22lastName%22%3A%22Getty%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adam%22%2C%22lastName%22%3A%22Gibicar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ryan%22%2C%22lastName%22%3A%22Gifford%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sajan%22%2C%22lastName%22%3A%22Gohil%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maria%22%2C%22lastName%22%3A%22Grammatikopoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%22%2C%22lastName%22%3A%22Grzech%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Orhun%22%2C%22lastName%22%3A%22G%5Cu00fcley%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Timo%22%2C%22lastName%22%3A%22G%5Cu00fcnnemann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chunxu%22%2C%22lastName%22%3A%22Guo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sylvain%22%2C%22lastName%22%3A%22Guy%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Heonjin%22%2C%22lastName%22%3A%22Ha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Luyi%22%2C%22lastName%22%3A%22Han%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Il%20Song%22%2C%22lastName%22%3A%22Han%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ali%22%2C%22lastName%22%3A%22Hatamizadeh%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tian%22%2C%22lastName%22%3A%22He%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jimin%22%2C%22lastName%22%3A%22Heo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sebastian%22%2C%22lastName%22%3A%22Hitziger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22SeulGi%22%2C%22lastName%22%3A%22Hong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22SeungBum%22%2C%22lastName%22%3A%22Hong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rian%22%2C%22lastName%22%3A%22Huang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ziyan%22%2C%22lastName%22%3A%22Huang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Markus%22%2C%22lastName%22%3A%22Huellebrand%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stephan%22%2C%22lastName%22%3A%22Huschauer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mustaffa%22%2C%22lastName%22%3A%22Hussain%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tomoo%22%2C%22lastName%22%3A%22Inubushi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ece%20Isik%22%2C%22lastName%22%3A%22Polat%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mojtaba%22%2C%22lastName%22%3A%22Jafaritadi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22SeongHun%22%2C%22lastName%22%3A%22Jeong%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bailiang%22%2C%22lastName%22%3A%22Jian%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yuanhong%22%2C%22lastName%22%3A%22Jiang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhifan%22%2C%22lastName%22%3A%22Jiang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yueming%22%2C%22lastName%22%3A%22Jin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Smriti%22%2C%22lastName%22%3A%22Joshi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdolrahim%22%2C%22lastName%22%3A%22Kadkhodamohammadi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Reda%20Abdellah%22%2C%22lastName%22%3A%22Kamraoui%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Inha%22%2C%22lastName%22%3A%22Kang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Junghwa%22%2C%22lastName%22%3A%22Kang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Davood%22%2C%22lastName%22%3A%22Karimi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22April%22%2C%22lastName%22%3A%22Khademi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Muhammad%20Irfan%22%2C%22lastName%22%3A%22Khan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Suleiman%20A.%22%2C%22lastName%22%3A%22Khan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rishab%22%2C%22lastName%22%3A%22Khantwal%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kwang-Ju%22%2C%22lastName%22%3A%22Kim%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Timothy%22%2C%22lastName%22%3A%22Kline%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Satoshi%22%2C%22lastName%22%3A%22Kondo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Elina%22%2C%22lastName%22%3A%22Kontio%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrian%22%2C%22lastName%22%3A%22Krenzer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Artem%22%2C%22lastName%22%3A%22Kroviakov%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hugo%22%2C%22lastName%22%3A%22Kuijf%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Satyadwyoom%22%2C%22lastName%22%3A%22Kumar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Francesco%22%2C%22lastName%22%3A%22La%20Rosa%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abhi%22%2C%22lastName%22%3A%22Lad%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Doohee%22%2C%22lastName%22%3A%22Lee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minho%22%2C%22lastName%22%3A%22Lee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chiara%22%2C%22lastName%22%3A%22Lena%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hao%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ling%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Xingyu%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fuyuan%22%2C%22lastName%22%3A%22Liao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22KuanLun%22%2C%22lastName%22%3A%22Liao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arlindo%20Limede%22%2C%22lastName%22%3A%22Oliveira%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chaonan%22%2C%22lastName%22%3A%22Lin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shan%22%2C%22lastName%22%3A%22Lin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Akis%22%2C%22lastName%22%3A%22Linardos%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marius%20George%22%2C%22lastName%22%3A%22Linguraru%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Han%22%2C%22lastName%22%3A%22Liu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tao%22%2C%22lastName%22%3A%22Liu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Di%22%2C%22lastName%22%3A%22Liu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yanling%22%2C%22lastName%22%3A%22Liu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jo%5Cu00e3o%22%2C%22lastName%22%3A%22Louren%5Cu00e7o-Silva%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jingpei%22%2C%22lastName%22%3A%22Lu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jiangshan%22%2C%22lastName%22%3A%22Lu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Imanol%22%2C%22lastName%22%3A%22Luengo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Christina%20B.%22%2C%22lastName%22%3A%22Lund%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Huan%20Minh%22%2C%22lastName%22%3A%22Luu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yi%22%2C%22lastName%22%3A%22Lv%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yi%22%2C%22lastName%22%3A%22Lv%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Uzay%22%2C%22lastName%22%3A%22Macar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Leon%22%2C%22lastName%22%3A%22Maechler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sina%20Mansour%22%2C%22lastName%22%3A%22L.%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kenji%22%2C%22lastName%22%3A%22Marshall%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Moona%22%2C%22lastName%22%3A%22Mazher%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Richard%22%2C%22lastName%22%3A%22McKinley%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alfonso%22%2C%22lastName%22%3A%22Medela%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Meissen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mingyuan%22%2C%22lastName%22%3A%22Meng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dylan%22%2C%22lastName%22%3A%22Miller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Seyed%20Hossein%22%2C%22lastName%22%3A%22Mirjahanmardi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arnab%22%2C%22lastName%22%3A%22Mishra%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Samir%22%2C%22lastName%22%3A%22Mitha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hassan%22%2C%22lastName%22%3A%22Mohy-ud-Din%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tony%20Chi%20Wing%22%2C%22lastName%22%3A%22Mok%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gowtham%20Krishnan%22%2C%22lastName%22%3A%22Murugesan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Enamundram%20Naga%22%2C%22lastName%22%3A%22Karthik%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sahil%22%2C%22lastName%22%3A%22Nalawade%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jakub%22%2C%22lastName%22%3A%22Nalepa%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mohamed%22%2C%22lastName%22%3A%22Naser%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ramin%22%2C%22lastName%22%3A%22Nateghi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hammad%22%2C%22lastName%22%3A%22Naveed%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Quang-Minh%22%2C%22lastName%22%3A%22Nguyen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Cuong%20Nguyen%22%2C%22lastName%22%3A%22Quoc%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brennan%22%2C%22lastName%22%3A%22Nichyporuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bruno%22%2C%22lastName%22%3A%22Oliveira%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Owen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jimut%20Bahan%22%2C%22lastName%22%3A%22Pal%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Junwen%22%2C%22lastName%22%3A%22Pan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wentao%22%2C%22lastName%22%3A%22Pan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Winnie%22%2C%22lastName%22%3A%22Pang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bogyu%22%2C%22lastName%22%3A%22Park%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vivek%22%2C%22lastName%22%3A%22Pawar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kamlesh%22%2C%22lastName%22%3A%22Pawar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Peven%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Philipp%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tomasz%22%2C%22lastName%22%3A%22Pieciak%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Szymon%22%2C%22lastName%22%3A%22Plotka%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marcel%22%2C%22lastName%22%3A%22Plutat%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fattaneh%22%2C%22lastName%22%3A%22Pourakpour%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Domen%22%2C%22lastName%22%3A%22Prelo%5Cu017enik%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kumaradevan%22%2C%22lastName%22%3A%22Punithakumar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdul%22%2C%22lastName%22%3A%22Qayyum%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sandro%22%2C%22lastName%22%3A%22Queir%5Cu00f3s%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arman%22%2C%22lastName%22%3A%22Rahmim%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Salar%22%2C%22lastName%22%3A%22Razavi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jintao%22%2C%22lastName%22%3A%22Ren%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mina%22%2C%22lastName%22%3A%22Rezaei%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jonathan%20Adam%22%2C%22lastName%22%3A%22Rico%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22ZunHyan%22%2C%22lastName%22%3A%22Rieu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Markus%22%2C%22lastName%22%3A%22Rink%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Johannes%22%2C%22lastName%22%3A%22Roth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yusely%22%2C%22lastName%22%3A%22Ruiz-Gonzalez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Numan%22%2C%22lastName%22%3A%22Saeed%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anindo%22%2C%22lastName%22%3A%22Saha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mostafa%22%2C%22lastName%22%3A%22Salem%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ricardo%22%2C%22lastName%22%3A%22Sanchez-Matilla%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kurt%22%2C%22lastName%22%3A%22Schilling%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wei%22%2C%22lastName%22%3A%22Shao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhiqiang%22%2C%22lastName%22%3A%22Shen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ruize%22%2C%22lastName%22%3A%22Shi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pengcheng%22%2C%22lastName%22%3A%22Shi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%22%2C%22lastName%22%3A%22Sobotka%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Th%5Cu00e9odore%22%2C%22lastName%22%3A%22Soulier%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bella%20Specktor%22%2C%22lastName%22%3A%22Fadida%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Danail%22%2C%22lastName%22%3A%22Stoyanov%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Timothy%20Sum%20Hon%22%2C%22lastName%22%3A%22Mun%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Xiaowu%22%2C%22lastName%22%3A%22Sun%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rong%22%2C%22lastName%22%3A%22Tao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Franz%22%2C%22lastName%22%3A%22Thaler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Antoine%22%2C%22lastName%22%3A%22Th%5Cu00e9berge%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Thielke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Helena%22%2C%22lastName%22%3A%22Torres%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kareem%20A.%22%2C%22lastName%22%3A%22Wahid%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jiacheng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22YiFei%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wei%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Xiong%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jianhui%22%2C%22lastName%22%3A%22Wen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ning%22%2C%22lastName%22%3A%22Wen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marek%22%2C%22lastName%22%3A%22Wodzinski%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ye%22%2C%22lastName%22%3A%22Wu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fangfang%22%2C%22lastName%22%3A%22Xia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tianqi%22%2C%22lastName%22%3A%22Xiang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Chen%22%2C%22lastName%22%3A%22Xiaofei%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lizhan%22%2C%22lastName%22%3A%22Xu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tingting%22%2C%22lastName%22%3A%22Xue%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yuxuan%22%2C%22lastName%22%3A%22Yang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lin%22%2C%22lastName%22%3A%22Yang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kai%22%2C%22lastName%22%3A%22Yao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Huifeng%22%2C%22lastName%22%3A%22Yao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amirsaeed%22%2C%22lastName%22%3A%22Yazdani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Yip%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hwanseung%22%2C%22lastName%22%3A%22Yoo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fereshteh%22%2C%22lastName%22%3A%22Yousefirizi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shunkai%22%2C%22lastName%22%3A%22Yu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lei%22%2C%22lastName%22%3A%22Yu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jonathan%22%2C%22lastName%22%3A%22Zamora%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ramy%20Ashraf%22%2C%22lastName%22%3A%22Zeineldin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dewen%22%2C%22lastName%22%3A%22Zeng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jianpeng%22%2C%22lastName%22%3A%22Zhang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bokai%22%2C%22lastName%22%3A%22Zhang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jiapeng%22%2C%22lastName%22%3A%22Zhang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fan%22%2C%22lastName%22%3A%22Zhang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Huahong%22%2C%22lastName%22%3A%22Zhang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhongchen%22%2C%22lastName%22%3A%22Zhao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zixuan%22%2C%22lastName%22%3A%22Zhao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jiachen%22%2C%22lastName%22%3A%22Zhao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Can%22%2C%22lastName%22%3A%22Zhao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Qingshuo%22%2C%22lastName%22%3A%22Zheng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yuheng%22%2C%22lastName%22%3A%22Zhi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ziqi%22%2C%22lastName%22%3A%22Zhou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Baosheng%22%2C%22lastName%22%3A%22Zou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%20F.%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22The%20number%20of%20international%20benchmarking%20competitions%20is%20steadily%20increasing%20in%20various%20fields%20of%20machine%20learning%20%28ML%29%20research%20and%20practice.%20So%20far%2C%20however%2C%20little%20is%20known%20about%20the%20common%20practice%20as%20well%20as%20bottlenecks%20faced%20by%20the%20community%20in%20tackling%20the%20research%20questions%20posed.%20To%20shed%20light%20on%20the%20status%20quo%20of%20algorithm%20development%20in%20the%20specific%20field%20of%20biomedical%20imaging%20analysis%2C%20we%20designed%20an%20international%20survey%20that%20was%20issued%20to%20all%20participants%20of%20challenges%20conducted%20in%20conjunction%20with%20the%20IEEE%20ISBI%202021%20and%20MICCAI%202021%20conferences%20%2880%20competitions%20in%20total%29.%20The%20survey%20covered%20participants%26%23039%3B%20expertise%20and%20working%20environments%2C%20their%20chosen%20strategies%2C%20as%20well%20as%20algorithm%20characteristics.%20A%20median%20of%2072%25%20challenge%20participants%20took%20part%20in%20the%20survey.%20According%20to%20our%20results%2C%20knowledge%20exchange%20was%20the%20primary%20incentive%20%2870%25%29%20for%20participation%2C%20while%20the%20reception%20of%20prize%20money%20played%20only%20a%20minor%20role%20%2816%25%29.%20While%20a%20median%20of%2080%20working%20hours%20was%20spent%20on%20method%20development%2C%20a%20large%20portion%20of%20participants%20stated%20that%20they%20did%20not%20have%20enough%20time%20for%20method%20development%20%2832%25%29.%2025%25%20perceived%20the%20infrastructure%20to%20be%20a%20bottleneck.%20Overall%2C%2094%25%20of%20all%20solutions%20were%20deep%20learning-based.%20Of%20these%2C%2084%25%20were%20based%20on%20standard%20architectures.%2043%25%20of%20the%20respondents%20reported%20that%20the%20data%20samples%20%28e.g.%2C%20images%29%20were%20too%20large%20to%20be%20processed%20at%20once.%20This%20was%20most%20commonly%20addressed%20by%20patch-based%20training%20%2869%25%29%2C%20downsampling%20%2837%25%29%2C%20and%20solving%203D%20analysis%20tasks%20as%20a%20series%20of%202D%20tasks.%20K-fold%20cross-validation%20on%20the%20training%20set%20was%20performed%20by%20only%2037%25%20of%20the%20participants%20and%20only%2050%25%20of%20the%20participants%20performed%20ensembling%20based%20on%20multiple%20identical%20models%20%2861%25%29%20or%20heterogeneous%20models%20%2839%25%29.%2048%25%20of%20the%20respondents%20applied%20postprocessing%20steps.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2212.08568%22%2C%22date%22%3A%222022-12-16%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2212.08568%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2212.08568%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222025-05-02T07%3A46%3A47Z%22%7D%7D%2C%7B%22key%22%3A%22YMCF7L2Q%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Tran%20et%20al.%22%2C%22parsedDate%22%3A%222022-11-17%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BTran%2C%20T.%20N.%2C%20Adler%2C%20T.%2C%20Yamlahi%2C%20A.%2C%20Christodoulou%2C%20E.%2C%20Godau%2C%20P.%2C%20Reinke%2C%20A.%2C%20Tizabi%2C%20M.%20D.%2C%20Sauer%2C%20P.%2C%20Persicke%2C%20T.%2C%20Albert%2C%20J.%20G.%2C%20%26amp%3B%20Maier-Hein%2C%20L.%20%282022%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BSources%20of%20performance%20variability%20in%20deep%20learning-based%20polyp%20detection%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2211.09708%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2211.09708%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2211.09708%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Sources%20of%20performance%20variability%20in%20deep%20learning-based%20polyp%20detection%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thuy%20Nuong%22%2C%22lastName%22%3A%22Tran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22Adler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amine%22%2C%22lastName%22%3A%22Yamlahi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20Dietlinde%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Sauer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tillmann%22%2C%22lastName%22%3A%22Persicke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22J%5Cu00f6rg%20Gerhard%22%2C%22lastName%22%3A%22Albert%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Validation%20metrics%20are%20a%20key%20prerequisite%20for%20the%20reliable%20tracking%20of%20scientific%20progress%20and%20for%20deciding%20on%20the%20potential%20clinical%20translation%20of%20methods.%20While%20recent%20initiatives%20aim%20to%20develop%20comprehensive%20theoretical%20frameworks%20for%20understanding%20metric-related%20pitfalls%20in%20image%20analysis%20problems%2C%20there%20is%20a%20lack%20of%20experimental%20evidence%20on%20the%20concrete%20effects%20of%20common%20and%20rare%20pitfalls%20on%20specific%20applications.%20We%20address%20this%20gap%20in%20the%20literature%20in%20the%20context%20of%20colon%20cancer%20screening.%20Our%20contribution%20is%20twofold.%20Firstly%2C%20we%20present%20the%20winning%20solution%20of%20the%20Endoscopy%20computer%20vision%20challenge%20%28EndoCV%29%20on%20colon%20cancer%20detection%2C%20conducted%20in%20conjunction%20with%20the%20IEEE%20International%20Symposium%20on%20Biomedical%20Imaging%20%28ISBI%29%202022.%20Secondly%2C%20we%20demonstrate%20the%20sensitivity%20of%20commonly%20used%20metrics%20to%20a%20range%20of%20hyperparameters%20as%20well%20as%20the%20consequences%20of%20poor%20metric%20choices.%20Based%20on%20comprehensive%20validation%20studies%20performed%20with%20patient%20data%20from%20six%20clinical%20centers%2C%20we%20found%20all%20commonly%20applied%20object%20detection%20metrics%20to%20be%20subject%20to%20high%20inter-center%20variability.%20Furthermore%2C%20our%20results%20clearly%20demonstrate%20that%20the%20adaptation%20of%20standard%20hyperparameters%20used%20in%20the%20computer%20vision%20community%20does%20not%20generally%20lead%20to%20the%20clinically%20most%20plausible%20results.%20Finally%2C%20we%20present%20localization%20criteria%20that%20correspond%20well%20to%20clinical%20relevance.%20Our%20work%20could%20be%20a%20first%20step%20towards%20reconsidering%20common%20validation%20strategies%20in%20automatic%20colon%20cancer%20screening%20applications.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2211.09708%22%2C%22date%22%3A%222022-11-17%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2211.09708%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2211.09708%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222022-11-21T15%3A57%3A07Z%22%7D%7D%2C%7B%22key%22%3A%22YWJJRSAY%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Zimmerer%20et%20al.%22%2C%22parsedDate%22%3A%222022-10%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BZimmerer%2C%20D.%2C%20Full%2C%20P.%20M.%2C%20Isensee%2C%20F.%2C%20J%26%23xE4%3Bger%2C%20P.%2C%20Adler%2C%20T.%2C%20Petersen%2C%20J.%2C%20K%26%23xF6%3Bhler%2C%20G.%2C%20Ross%2C%20T.%2C%20Reinke%2C%20A.%2C%20Kascenas%2C%20A.%2C%20Jensen%2C%20B.%20S.%2C%20O%26%23x2019%3BNeil%2C%20A.%20Q.%2C%20Tan%2C%20J.%2C%20Hou%2C%20B.%2C%20Batten%2C%20J.%2C%20Qiu%2C%20H.%2C%20Kainz%2C%20B.%2C%20Shvetsova%2C%20N.%2C%20Fedulova%2C%20I.%2C%20%26%23x2026%3B%20Maier-Hein%2C%20K.%20%282022%29.%20%26lt%3Bb%26gt%3BMOOD%202020%3A%20A%20Public%20Benchmark%20for%20Out-of-Distribution%20Detection%20and%20Localization%20on%20Medical%20Images%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BIEEE%20Transactions%20on%20Medical%20Imaging%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B41%26lt%3B%5C%2Fi%26gt%3B%2810%29%2C%202728%26%23x2013%3B2738.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FTMI.2022.3170077%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FTMI.2022.3170077%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22MOOD%202020%3A%20A%20Public%20Benchmark%20for%20Out-of-Distribution%20Detection%20and%20Localization%20on%20Medical%20Images%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Zimmerer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%20M.%22%2C%22lastName%22%3A%22Full%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22Adler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Petersen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gregor%22%2C%22lastName%22%3A%22K%5Cu00f6hler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tobias%22%2C%22lastName%22%3A%22Ross%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Antanas%22%2C%22lastName%22%3A%22Kascenas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bj%5Cu00f8rn%20Sand%22%2C%22lastName%22%3A%22Jensen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alison%20Q.%22%2C%22lastName%22%3A%22O%5Cu2019Neil%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jeremy%22%2C%22lastName%22%3A%22Tan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Benjamin%22%2C%22lastName%22%3A%22Hou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22James%22%2C%22lastName%22%3A%22Batten%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Huaqi%22%2C%22lastName%22%3A%22Qiu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bernhard%22%2C%22lastName%22%3A%22Kainz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nina%22%2C%22lastName%22%3A%22Shvetsova%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Irina%22%2C%22lastName%22%3A%22Fedulova%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dmitry%20V.%22%2C%22lastName%22%3A%22Dylov%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Baolun%22%2C%22lastName%22%3A%22Yu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jianyang%22%2C%22lastName%22%3A%22Zhai%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jingtao%22%2C%22lastName%22%3A%22Hu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Runxuan%22%2C%22lastName%22%3A%22Si%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sihang%22%2C%22lastName%22%3A%22Zhou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Siqi%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Xinyang%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Xuerun%22%2C%22lastName%22%3A%22Chen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yang%22%2C%22lastName%22%3A%22Zhao%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sergio%20Naval%22%2C%22lastName%22%3A%22Marimont%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Giacomo%22%2C%22lastName%22%3A%22Tarroni%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Victor%22%2C%22lastName%22%3A%22Saase%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Detecting%20Out-of-Distribution%20%28OoD%29%20data%20is%20one%20of%20the%20greatest%20challenges%20in%20safe%20and%20robust%20deployment%20of%20machine%20learning%20algorithms%20in%20medicine.%20When%20the%20algorithms%20encounter%20cases%20that%20deviate%20from%20the%20distribution%20of%20the%20training%20data%2C%20they%20often%20produce%20incorrect%20and%20over-confident%20predictions.%20OoD%20detection%20algorithms%20aim%20to%20catch%20erroneous%20predictions%20in%20advance%20by%20analysing%20the%20data%20distribution%20and%20detecting%20potential%20instances%20of%20failure.%20Moreover%2C%20flagging%20OoD%20cases%20may%20support%20human%20readers%20in%20identifying%20incidental%20findings.%20Due%20to%20the%20increased%20interest%20in%20OoD%20algorithms%2C%20benchmarks%20for%20different%20domains%20have%20recently%20been%20established.%20In%20the%20medical%20imaging%20domain%2C%20for%20which%20reliable%20predictions%20are%20often%20essential%2C%20an%20open%20benchmark%20has%20been%20missing.%20We%20introduce%20the%20Medical-Out-Of-Distribution-Analysis-Challenge%20%28MOOD%29%20as%20an%20open%2C%20fair%2C%20and%20unbiased%20benchmark%20for%20OoD%20methods%20in%20the%20medical%20imaging%20domain.%20The%20analysis%20of%20the%20submitted%20algorithms%20shows%20that%20performance%20has%20a%20strong%20positive%20correlation%20with%20the%20perceived%20difficulty%2C%20and%20that%20all%20algorithms%20show%20a%20high%20variance%20for%20different%20anomalies%2C%20making%20it%20yet%20hard%20to%20recommend%20them%20for%20clinical%20practice.%20We%20also%20see%20a%20strong%20correlation%20between%20challenge%20ranking%20and%20performance%20on%20a%20simple%20toy%20test%20set%2C%20indicating%20that%20this%20might%20be%20a%20valuable%20addition%20as%20a%20proxy%20dataset%20during%20anomaly%20detection%20algorithm%20development.%22%2C%22date%22%3A%222022-10%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FTMI.2022.3170077%22%2C%22ISSN%22%3A%221558-254X%22%2C%22url%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222023-02-13T14%3A47%3A14Z%22%7D%7D%2C%7B%22key%22%3A%223HFXLPJW%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Maier-Hein%20et%20al.%22%2C%22parsedDate%22%3A%222022-09-15%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BMaier-Hein%2C%20L.%2C%20Reinke%2C%20A.%2C%20Godau%2C%20P.%2C%20Tizabi%2C%20M.%20D.%2C%20Christodoulou%2C%20E.%2C%20Glocker%2C%20B.%2C%20Isensee%2C%20F.%2C%20Kleesiek%2C%20J.%2C%20Kozubek%2C%20M.%2C%20Reyes%2C%20M.%2C%20Riegler%2C%20M.%20A.%2C%20Wiesenfarth%2C%20M.%2C%20Baumgartner%2C%20M.%2C%20Eisenmann%2C%20M.%2C%20Heckmann-N%26%23xF6%3Btzel%2C%20D.%2C%20Kavur%2C%20A.%20E.%2C%20R%26%23xE4%3Bdsch%2C%20T.%2C%20Acion%2C%20L.%2C%20Antonelli%2C%20M.%2C%20%26%23x2026%3B%20J%26%23xE4%3Bger%2C%20P.%20F.%20%282022%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BMetrics%20reloaded%3A%20Pitfalls%20and%20recommendations%20for%20image%20analysis%20validation%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2206.01653%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2206.01653%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2206.01653%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Metrics%20reloaded%3A%20Pitfalls%20and%20recommendations%20for%20image%20analysis%20validation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Glocker%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Kleesiek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mauricio%22%2C%22lastName%22%3A%22Reyes%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20A.%22%2C%22lastName%22%3A%22Riegler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuel%22%2C%22lastName%22%3A%22Wiesenfarth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Baumgartner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Doreen%22%2C%22lastName%22%3A%22Heckmann-N%5Cu00f6tzel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22A.%20Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Acion%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michela%22%2C%22lastName%22%3A%22Antonelli%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tal%22%2C%22lastName%22%3A%22Arbel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Bankhead%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arriel%22%2C%22lastName%22%3A%22Benis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Jorge%22%2C%22lastName%22%3A%22Cardoso%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beth%22%2C%22lastName%22%3A%22Cimini%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gary%20S.%22%2C%22lastName%22%3A%22Collins%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Keyvan%22%2C%22lastName%22%3A%22Farahani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Luciana%22%2C%22lastName%22%3A%22Ferrer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adrian%22%2C%22lastName%22%3A%22Galdran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22van%20Ginneken%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Robert%22%2C%22lastName%22%3A%22Haase%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%20A.%22%2C%22lastName%22%3A%22Hashimoto%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20M.%22%2C%22lastName%22%3A%22Hoffman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Merel%22%2C%22lastName%22%3A%22Huisman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Jannin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Charles%20E.%22%2C%22lastName%22%3A%22Kahn%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dagmar%22%2C%22lastName%22%3A%22Kainmueller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bernhard%22%2C%22lastName%22%3A%22Kainz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexandros%22%2C%22lastName%22%3A%22Karargyris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alan%22%2C%22lastName%22%3A%22Karthikesalingam%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hannes%22%2C%22lastName%22%3A%22Kenngott%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Florian%22%2C%22lastName%22%3A%22Kofler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%22%2C%22lastName%22%3A%22Kreshuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tahsin%22%2C%22lastName%22%3A%22Kurc%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%20A.%22%2C%22lastName%22%3A%22Landman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Geert%22%2C%22lastName%22%3A%22Litjens%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amin%22%2C%22lastName%22%3A%22Madani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anne%20L.%22%2C%22lastName%22%3A%22Martel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Mattson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Erik%22%2C%22lastName%22%3A%22Meijering%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Moher%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karel%20G.%20M.%22%2C%22lastName%22%3A%22Moons%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Henning%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brennan%22%2C%22lastName%22%3A%22Nichyporuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Nickel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Petersen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nasir%22%2C%22lastName%22%3A%22Rajpoot%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicola%22%2C%22lastName%22%3A%22Rieke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julio%22%2C%22lastName%22%3A%22Saez-Rodriguez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Clarisa%20S%5Cu00e1nchez%22%2C%22lastName%22%3A%22Guti%5Cu00e9rrez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shravya%22%2C%22lastName%22%3A%22Shetty%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maarten%22%2C%22lastName%22%3A%22van%20Smeden%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%20H.%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ronald%20M.%22%2C%22lastName%22%3A%22Summers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdel%20A.%22%2C%22lastName%22%3A%22Taha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sotirios%20A.%22%2C%22lastName%22%3A%22Tsaftaris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Van%20Calster%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ga%5Cu00ebl%22%2C%22lastName%22%3A%22Varoquaux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%20F.%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%5D%2C%22abstractNote%22%3A%22Increasing%20evidence%20shows%20that%20flaws%20in%20machine%20learning%20%28ML%29%20algorithm%20validation%20are%20an%20underestimated%20global%20problem.%20Particularly%20in%20automatic%20biomedical%20image%20analysis%2C%20chosen%20performance%20metrics%20often%20do%20not%20reflect%20the%20domain%20interest%2C%20thus%20failing%20to%20adequately%20measure%20scientific%20progress%20and%20hindering%20translation%20of%20ML%20techniques%20into%20practice.%20To%20overcome%20this%2C%20a%20large%20international%20expert%20consortium%20created%20Metrics%20Reloaded%2C%20a%20comprehensive%20framework%20guiding%20researchers%20towards%20choosing%20metrics%20in%20a%20problem-aware%20manner.%20Following%20the%20convergence%20of%20ML%20methodology%20across%20application%20domains%2C%20Metrics%20Reloaded%20fosters%20the%20convergence%20of%20validation%20methodology.%20The%20framework%20was%20developed%20in%20a%20multi-stage%20Delphi%20process%20and%20is%20based%20on%20the%20novel%20concept%20of%20a%20problem%20fingerprint%20-%20a%20structured%20representation%20of%20the%20given%20problem%20that%20captures%20all%20aspects%20that%20are%20relevant%20for%20metric%20selection%20from%20the%20domain%20interest%20to%20the%20properties%20of%20the%20target%20structure%28s%29%2C%20data%20set%20and%20algorithm%20output.%20Metrics%20Reloaded%20targets%20image%20analysis%20problems%20that%20can%20be%20interpreted%20as%20a%20classification%20task%20at%20image%2C%20object%20or%20pixel%20level%2C%20namely%20image-level%20classification%2C%20object%20detection%2C%20semantic%20segmentation%2C%20and%20instance%20segmentation%20tasks.%20Users%20are%20guided%20through%20the%20process%20of%20selecting%20and%20applying%20appropriate%20validation%20metrics%20while%20being%20made%20aware%20of%20potential%20pitfalls.%20To%20improve%20the%20user%20experience%2C%20we%20implemented%20the%20framework%20in%20the%20Metrics%20Reloaded%20online%20tool%2C%20which%20also%20provides%20a%20common%20point%20of%20access%20to%20explore%20weaknesses%20and%20strengths%20of%20the%20most%20common%20validation%20metrics.%20An%20instantiation%20of%20the%20framework%20for%20various%20biological%20and%20medical%20image%20analysis%20use%20cases%20demonstrates%20its%20broad%20applicability%20across%20domains.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2206.01653%22%2C%22date%22%3A%222022-09-15%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2206.01653%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2206.01653%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222022-11-21T15%3A54%3A13Z%22%7D%7D%2C%7B%22key%22%3A%2298IGNRCT%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Antonelli%20et%20al.%22%2C%22parsedDate%22%3A%222022-07-15%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BAntonelli%2C%20M.%2C%20Reinke%2C%20A.%2C%20Bakas%2C%20S.%2C%20Farahani%2C%20K.%2C%20Kopp-Schneider%2C%20A.%2C%20Landman%2C%20B.%20A.%2C%20Litjens%2C%20G.%2C%20Menze%2C%20B.%2C%20Ronneberger%2C%20O.%2C%20Summers%2C%20R.%20M.%2C%20van%20Ginneken%2C%20B.%2C%20Bilello%2C%20M.%2C%20Bilic%2C%20P.%2C%20Christ%2C%20P.%20F.%2C%20Do%2C%20R.%20K.%20G.%2C%20Gollub%2C%20M.%20J.%2C%20Heckers%2C%20S.%20H.%2C%20Huisman%2C%20H.%2C%20Jarnagin%2C%20W.%20R.%2C%20%26%23x2026%3B%20Cardoso%2C%20M.%20J.%20%282022%29.%20%26lt%3Bb%26gt%3BThe%20Medical%20Segmentation%20Decathlon%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BNature%20Communications%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B13%26lt%3B%5C%2Fi%26gt%3B%281%29%2C%204128.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41467-022-30695-9%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41467-022-30695-9%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22The%20Medical%20Segmentation%20Decathlon%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michela%22%2C%22lastName%22%3A%22Antonelli%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Keyvan%22%2C%22lastName%22%3A%22Farahani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%20A.%22%2C%22lastName%22%3A%22Landman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Geert%22%2C%22lastName%22%3A%22Litjens%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Olaf%22%2C%22lastName%22%3A%22Ronneberger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ronald%20M.%22%2C%22lastName%22%3A%22Summers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22van%20Ginneken%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michel%22%2C%22lastName%22%3A%22Bilello%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Bilic%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%20F.%22%2C%22lastName%22%3A%22Christ%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Richard%20K.%20G.%22%2C%22lastName%22%3A%22Do%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%20J.%22%2C%22lastName%22%3A%22Gollub%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stephan%20H.%22%2C%22lastName%22%3A%22Heckers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Henkjan%22%2C%22lastName%22%3A%22Huisman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22William%20R.%22%2C%22lastName%22%3A%22Jarnagin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maureen%20K.%22%2C%22lastName%22%3A%22McHugo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sandy%22%2C%22lastName%22%3A%22Napel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jennifer%20S.%20Golia%22%2C%22lastName%22%3A%22Pernicka%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kawal%22%2C%22lastName%22%3A%22Rhode%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Catalina%22%2C%22lastName%22%3A%22Tobon-Gomez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Eugene%22%2C%22lastName%22%3A%22Vorontsov%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22James%20A.%22%2C%22lastName%22%3A%22Meakin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sebastien%22%2C%22lastName%22%3A%22Ourselin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuel%22%2C%22lastName%22%3A%22Wiesenfarth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pablo%22%2C%22lastName%22%3A%22Arbel%5Cu00e1ez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Byeonguk%22%2C%22lastName%22%3A%22Bae%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sihong%22%2C%22lastName%22%3A%22Chen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Daza%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jianjiang%22%2C%22lastName%22%3A%22Feng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Baochun%22%2C%22lastName%22%3A%22He%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yuanfeng%22%2C%22lastName%22%3A%22Ji%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fucang%22%2C%22lastName%22%3A%22Jia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ildoo%22%2C%22lastName%22%3A%22Kim%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dorit%22%2C%22lastName%22%3A%22Merhof%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Akshay%22%2C%22lastName%22%3A%22Pai%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beomhee%22%2C%22lastName%22%3A%22Park%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mathias%22%2C%22lastName%22%3A%22Perslev%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ramin%22%2C%22lastName%22%3A%22Rezaiifar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Oliver%22%2C%22lastName%22%3A%22Rippel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ignacio%22%2C%22lastName%22%3A%22Sarasua%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Wei%22%2C%22lastName%22%3A%22Shen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jaemin%22%2C%22lastName%22%3A%22Son%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Christian%22%2C%22lastName%22%3A%22Wachinger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Liansheng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yan%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yingda%22%2C%22lastName%22%3A%22Xia%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daguang%22%2C%22lastName%22%3A%22Xu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhanwei%22%2C%22lastName%22%3A%22Xu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yefeng%22%2C%22lastName%22%3A%22Zheng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amber%20L.%22%2C%22lastName%22%3A%22Simpson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Jorge%22%2C%22lastName%22%3A%22Cardoso%22%7D%5D%2C%22abstractNote%22%3A%22International%20challenges%20have%20become%20the%20de%20facto%20standard%20for%20comparative%20assessment%20of%20image%20analysis%20algorithms.%20Although%20segmentation%20is%20the%20most%20widely%20investigated%20medical%20image%20processing%20task%2C%20the%20various%20challenges%20have%20been%20organized%20to%20focus%20only%20on%20specific%20clinical%20tasks.%20We%20organized%20the%20Medical%20Segmentation%20Decathlon%20%28MSD%29%5Cu2014a%20biomedical%20image%20analysis%20challenge%2C%20in%20which%20algorithms%20compete%20in%20a%20multitude%20of%20both%20tasks%20and%20modalities%20to%20investigate%20the%20hypothesis%20that%20a%20method%20capable%20of%20performing%20well%20on%20multiple%20tasks%20will%20generalize%20well%20to%20a%20previously%20unseen%20task%20and%20potentially%20outperform%20a%20custom-designed%20solution.%20MSD%20results%20confirmed%20this%20hypothesis%2C%20moreover%2C%20MSD%20winner%20continued%20generalizing%20well%20to%20a%20wide%20range%20of%20other%20clinical%20problems%20for%20the%20next%20two%20years.%20Three%20main%20conclusions%20can%20be%20drawn%20from%20this%20study%3A%20%281%29%20state-of-the-art%20image%20segmentation%20algorithms%20generalize%20well%20when%20retrained%20on%20unseen%20tasks%3B%20%282%29%20consistent%20algorithmic%20performance%20across%20multiple%20tasks%20is%20a%20strong%20surrogate%20of%20algorithmic%20generalizability%3B%20%283%29%20the%20training%20of%20accurate%20AI%20segmentation%20models%20is%20now%20commoditized%20to%20scientists%20that%20are%20not%20versed%20in%20AI%20model%20training.%22%2C%22date%22%3A%222022-07-15%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1038%5C%2Fs41467-022-30695-9%22%2C%22ISSN%22%3A%222041-1723%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.nature.com%5C%2Farticles%5C%2Fs41467-022-30695-9%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222022-11-21T15%3A55%3A24Z%22%7D%7D%2C%7B%22key%22%3A%22KJ7UJXQG%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Reinke%20et%20al.%22%2C%22parsedDate%22%3A%222022-07-07%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BReinke%2C%20A.%2C%20Tizabi%2C%20M.%20D.%2C%20Sudre%2C%20C.%20H.%2C%20Eisenmann%2C%20M.%2C%20R%26%23xE4%3Bdsch%2C%20T.%2C%20Baumgartner%2C%20M.%2C%20Acion%2C%20L.%2C%20Antonelli%2C%20M.%2C%20Arbel%2C%20T.%2C%20Bakas%2C%20S.%2C%20Bankhead%2C%20P.%2C%20Benis%2C%20A.%2C%20Cardoso%2C%20M.%20J.%2C%20Cheplygina%2C%20V.%2C%20Christodoulou%2C%20E.%2C%20Cimini%2C%20B.%2C%20Collins%2C%20G.%20S.%2C%20Farahani%2C%20K.%2C%20van%20Ginneken%2C%20B.%2C%20%26%23x2026%3B%20Maier-Hein%2C%20L.%20%282022%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BCommon%20Limitations%20of%20Image%20Processing%20Metrics%3A%20A%20Picture%20Story%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2104.05642%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2104.05642%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2104.05642%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Common%20Limitations%20of%20Image%20Processing%20Metrics%3A%20A%20Picture%20Story%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carole%20H.%22%2C%22lastName%22%3A%22Sudre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%22%2C%22lastName%22%3A%22R%5Cu00e4dsch%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Baumgartner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Acion%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michela%22%2C%22lastName%22%3A%22Antonelli%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tal%22%2C%22lastName%22%3A%22Arbel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Spyridon%22%2C%22lastName%22%3A%22Bakas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Bankhead%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Arriel%22%2C%22lastName%22%3A%22Benis%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Jorge%22%2C%22lastName%22%3A%22Cardoso%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veronika%22%2C%22lastName%22%3A%22Cheplygina%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Evangelia%22%2C%22lastName%22%3A%22Christodoulou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beth%22%2C%22lastName%22%3A%22Cimini%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gary%20S.%22%2C%22lastName%22%3A%22Collins%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Keyvan%22%2C%22lastName%22%3A%22Farahani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22van%20Ginneken%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Glocker%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Godau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fred%22%2C%22lastName%22%3A%22Hamprecht%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%20A.%22%2C%22lastName%22%3A%22Hashimoto%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Doreen%22%2C%22lastName%22%3A%22Heckmann-N%5Cu00f6tzel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20M.%22%2C%22lastName%22%3A%22Hoffman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Merel%22%2C%22lastName%22%3A%22Huisman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Jannin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Charles%20E.%22%2C%22lastName%22%3A%22Kahn%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexandros%22%2C%22lastName%22%3A%22Karargyris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alan%22%2C%22lastName%22%3A%22Karthikesalingam%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bernhard%22%2C%22lastName%22%3A%22Kainz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Emre%22%2C%22lastName%22%3A%22Kavur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hannes%22%2C%22lastName%22%3A%22Kenngott%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Kleesiek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thijs%22%2C%22lastName%22%3A%22Kooi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%22%2C%22lastName%22%3A%22Kreshuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tahsin%22%2C%22lastName%22%3A%22Kurc%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%20A.%22%2C%22lastName%22%3A%22Landman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Geert%22%2C%22lastName%22%3A%22Litjens%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amin%22%2C%22lastName%22%3A%22Madani%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anne%20L.%22%2C%22lastName%22%3A%22Martel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%22%2C%22lastName%22%3A%22Mattson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Erik%22%2C%22lastName%22%3A%22Meijering%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bjoern%22%2C%22lastName%22%3A%22Menze%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%22%2C%22lastName%22%3A%22Moher%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karel%20G.%20M.%22%2C%22lastName%22%3A%22Moons%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Henning%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brennan%22%2C%22lastName%22%3A%22Nichyporuk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Felix%22%2C%22lastName%22%3A%22Nickel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Alican%22%2C%22lastName%22%3A%22Noyan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jens%22%2C%22lastName%22%3A%22Petersen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gorkem%22%2C%22lastName%22%3A%22Polat%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nasir%22%2C%22lastName%22%3A%22Rajpoot%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mauricio%22%2C%22lastName%22%3A%22Reyes%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicola%22%2C%22lastName%22%3A%22Rieke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Riegler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hassan%22%2C%22lastName%22%3A%22Rivaz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julio%22%2C%22lastName%22%3A%22Saez-Rodriguez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Clarisa%20Sanchez%22%2C%22lastName%22%3A%22Gutierrez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julien%22%2C%22lastName%22%3A%22Schroeter%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anindo%22%2C%22lastName%22%3A%22Saha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shravya%22%2C%22lastName%22%3A%22Shetty%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Maarten%22%2C%22lastName%22%3A%22van%20Smeden%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22Stieltjes%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ronald%20M.%22%2C%22lastName%22%3A%22Summers%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abdel%20A.%22%2C%22lastName%22%3A%22Taha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sotirios%20A.%22%2C%22lastName%22%3A%22Tsaftaris%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Van%20Calster%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ga%5Cu00ebl%22%2C%22lastName%22%3A%22Varoquaux%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuel%22%2C%22lastName%22%3A%22Wiesenfarth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ziv%20R.%22%2C%22lastName%22%3A%22Yaniv%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%22%2C%22lastName%22%3A%22J%5Cu00e4ger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22While%20the%20importance%20of%20automatic%20image%20analysis%20is%20continuously%20increasing%2C%20recent%20meta-research%20revealed%20major%20flaws%20with%20respect%20to%20algorithm%20validation.%20Performance%20metrics%20are%20particularly%20key%20for%20meaningful%2C%20objective%2C%20and%20transparent%20performance%20assessment%20and%20validation%20of%20the%20used%20automatic%20algorithms%2C%20but%20relatively%20little%20attention%20has%20been%20given%20to%20the%20practical%20pitfalls%20when%20using%20specific%20metrics%20for%20a%20given%20image%20analysis%20task.%20These%20are%20typically%20related%20to%20%281%29%20the%20disregard%20of%20inherent%20metric%20properties%2C%20such%20as%20the%20behaviour%20in%20the%20presence%20of%20class%20imbalance%20or%20small%20target%20structures%2C%20%282%29%20the%20disregard%20of%20inherent%20data%20set%20properties%2C%20such%20as%20the%20non-independence%20of%20the%20test%20cases%2C%20and%20%283%29%20the%20disregard%20of%20the%20actual%20biomedical%20domain%20interest%20that%20the%20metrics%20should%20reflect.%20This%20living%20dynamically%20document%20has%20the%20purpose%20to%20illustrate%20important%20limitations%20of%20performance%20metrics%20commonly%20applied%20in%20the%20field%20of%20image%20analysis.%20In%20this%20context%2C%20it%20focuses%20on%20biomedical%20image%20analysis%20problems%20that%20can%20be%20phrased%20as%20image-level%20classification%2C%20semantic%20segmentation%2C%20instance%20segmentation%2C%20or%20object%20detection%20task.%20The%20current%20version%20is%20based%20on%20a%20Delphi%20process%20on%20metrics%20conducted%20by%20an%20international%20consortium%20of%20image%20analysis%20experts%20from%20more%20than%2060%20institutions%20worldwide.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2104.05642%22%2C%22date%22%3A%222022-07-07%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2104.05642%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2104.05642%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222023-02-03T10%3A12%3A41Z%22%7D%7D%2C%7B%22key%22%3A%227FR7NCMK%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Schellenberg%20et%20al.%22%2C%22parsedDate%22%3A%222022-06-01%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BSchellenberg%2C%20M.%2C%20Dreher%2C%20K.%20K.%2C%20Holzwarth%2C%20N.%2C%20Isensee%2C%20F.%2C%20Reinke%2C%20A.%2C%20Schreck%2C%20N.%2C%20Seitel%2C%20A.%2C%20Tizabi%2C%20M.%20D.%2C%20Maier-Hein%2C%20L.%2C%20%26amp%3B%20Gr%26%23xF6%3Bhl%2C%20J.%20%282022%29.%20%26lt%3Bb%26gt%3BSemantic%20segmentation%20of%20multispectral%20photoacoustic%20images%20using%20deep%20learning%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BPhotoacoustics%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B26%26lt%3B%5C%2Fi%26gt%3B%2C%20100341.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.pacs.2022.100341%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.pacs.2022.100341%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Semantic%20segmentation%20of%20multispectral%20photoacoustic%20images%20using%20deep%20learning%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Melanie%22%2C%22lastName%22%3A%22Schellenberg%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kris%20K.%22%2C%22lastName%22%3A%22Dreher%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Niklas%22%2C%22lastName%22%3A%22Holzwarth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Nicholas%22%2C%22lastName%22%3A%22Schreck%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alexander%22%2C%22lastName%22%3A%22Seitel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%20D.%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Janek%22%2C%22lastName%22%3A%22Gr%5Cu00f6hl%22%7D%5D%2C%22abstractNote%22%3A%22Photoacoustic%20%28PA%29%20imaging%20has%20the%20potential%20to%20revolutionize%20functional%20medical%20imaging%20in%20healthcare%20due%20to%20the%20valuable%20information%20on%20tissue%20physiology%20contained%20in%20multispectral%20photoacoustic%20measurements.%20Clinical%20translation%20of%20the%20technology%20requires%20conversion%20of%20the%20high-dimensional%20acquired%20data%20into%20clinically%20relevant%20and%20interpretable%20information.%20In%20this%20work%2C%20we%20present%20a%20deep%20learning-based%20approach%20to%20semantic%20segmentation%20of%20multispectral%20photoacoustic%20images%20to%20facilitate%20image%20interpretability.%20Manually%20annotated%20photoacoustic%20and%20ultrasound%20imaging%20data%20are%20used%20as%20reference%20and%20enable%20the%20training%20of%20a%20deep%20learning-based%20segmentation%20algorithm%20in%20a%20supervised%20manner.%20Based%20on%20a%20validation%20study%20with%20experimentally%20acquired%20data%20from%2016%20healthy%20human%20volunteers%2C%20we%20show%20that%20automatic%20tissue%20segmentation%20can%20be%20used%20to%20create%20powerful%20analyses%20and%20visualizations%20of%20multispectral%20photoacoustic%20images.%20Due%20to%20the%20intuitive%20representation%20of%20high-dimensional%20information%2C%20such%20a%20preprocessing%20algorithm%20could%20be%20a%20valuable%20means%20to%20facilitate%20the%20clinical%20translation%20of%20photoacoustic%20imaging.%22%2C%22date%22%3A%222022-06-01%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1016%5C%2Fj.pacs.2022.100341%22%2C%22ISSN%22%3A%222213-5979%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.sciencedirect.com%5C%2Fscience%5C%2Farticle%5C%2Fpii%5C%2FS2213597922000118%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222022-11-24T07%3A57%3A19Z%22%7D%7D%2C%7B%22key%22%3A%22HMKYF7UC%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Bron%20et%20al.%22%2C%22parsedDate%22%3A%222022-06-01%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBron%2C%20E.%20E.%2C%20Klein%2C%20S.%2C%20Reinke%2C%20A.%2C%20Papma%2C%20J.%20M.%2C%20Maier-Hein%2C%20L.%2C%20Alexander%2C%20D.%20C.%2C%20%26amp%3B%20Oxtoby%2C%20N.%20P.%20%282022%29.%20%26lt%3Bb%26gt%3BTen%20years%20of%20image%20analysis%20and%20machine%20learning%20competitions%20in%20dementia%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BNeuroImage%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B253%26lt%3B%5C%2Fi%26gt%3B%2C%20119083.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.neuroimage.2022.119083%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.neuroimage.2022.119083%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Ten%20years%20of%20image%20analysis%20and%20machine%20learning%20competitions%20in%20dementia%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Esther%20E.%22%2C%22lastName%22%3A%22Bron%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stefan%22%2C%22lastName%22%3A%22Klein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Janne%20M.%22%2C%22lastName%22%3A%22Papma%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daniel%20C.%22%2C%22lastName%22%3A%22Alexander%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Neil%20P.%22%2C%22lastName%22%3A%22Oxtoby%22%7D%5D%2C%22abstractNote%22%3A%22Machine%20learning%20methods%20exploiting%20multi-parametric%20biomarkers%2C%20especially%20based%20on%20neuroimaging%2C%20have%20huge%20potential%20to%20improve%20early%20diagnosis%20of%20dementia%20and%20to%20predict%20which%20individuals%20are%20at-risk%20of%20developing%20dementia.%20To%20benchmark%20algorithms%20in%20the%20field%20of%20machine%20learning%20and%20neuroimaging%20in%20dementia%20and%20assess%20their%20potential%20for%20use%20in%20clinical%20practice%20and%20clinical%20trials%2C%20seven%20grand%20challenges%20have%20been%20organized%20in%20the%20last%20decade%3A%20MIRIAD%20%282012%29%2C%20Alzheimer%5Cu2019s%20Disease%20Big%20Data%20DREAM%20%282014%29%2C%20CADDementia%20%282014%29%2C%20Machine%20Learning%20Challenge%20%282014%29%2C%20MCI%20Neuroimaging%20%282017%29%2C%20TADPOLE%20%282017%29%2C%20and%20the%20Predictive%20Analytics%20Competition%20%282019%29.%20Based%20on%20two%20challenge%20evaluation%20frameworks%2C%20we%20analyzed%20how%20these%20grand%20challenges%20are%20complementing%20each%20other%20regarding%20research%20questions%2C%20datasets%2C%20validation%20approaches%2C%20results%20and%20impact.%20The%20seven%20grand%20challenges%20addressed%20questions%20related%20to%20screening%2C%20clinical%20status%20estimation%2C%20prediction%20and%20monitoring%20in%20%28pre-clinical%29%20dementia.%20There%20was%20little%20overlap%20in%20clinical%20questions%2C%20tasks%20and%20performance%20metrics.%20Whereas%20this%20aids%20providing%20insight%20on%20a%20broad%20range%20of%20questions%2C%20it%20also%20limits%20the%20validation%20of%20results%20across%20challenges.%20The%20validation%20process%20itself%20was%20mostly%20comparable%20between%20challenges%2C%20using%20similar%20methods%20for%20ensuring%20objective%20comparison%2C%20uncertainty%20estimation%20and%20statistical%20testing.%20In%20general%2C%20winning%20algorithms%20performed%20rigorous%20data%20pre-processing%20and%20combined%20a%20wide%20range%20of%20input%20features.%20Despite%20high%20state-of-the-art%20performances%2C%20most%20of%20the%20methods%20evaluated%20by%20the%20challenges%20are%20not%20clinically%20used.%20To%20increase%20impact%2C%20future%20challenges%20could%20pay%20more%20attention%20to%20statistical%20analysis%20of%20which%20factors%20%28i.e.%2C%20features%2C%20models%29%20relate%20to%20higher%20performance%2C%20to%20clinical%20questions%20beyond%20Alzheimer%5Cu2019s%20disease%2C%20and%20to%20using%20testing%20data%20beyond%20the%20Alzheimer%5Cu2019s%20Disease%20Neuroimaging%20Initiative.%20Grand%20challenges%20would%20be%20an%20ideal%20venue%20for%20assessing%20the%20generalizability%20of%20algorithm%20performance%20to%20unseen%20data%20of%20other%20cohorts.%20Key%20for%20increasing%20impact%20in%20this%20way%20are%20larger%20testing%20data%20sizes%2C%20which%20could%20be%20reached%20by%20sharing%20algorithms%20rather%20than%20data%20to%20exploit%20data%20that%20cannot%20be%20shared.%20Given%20the%20potential%20and%20lessons%20learned%20in%20the%20past%20ten%20years%2C%20we%20are%20excited%20by%20the%20prospects%20of%20grand%20challenges%20in%20machine%20learning%20and%20neuroimaging%20for%20the%20next%20ten%20years%20and%20beyond.%22%2C%22date%22%3A%222022-06-01%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1016%5C%2Fj.neuroimage.2022.119083%22%2C%22ISSN%22%3A%221053-8119%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.sciencedirect.com%5C%2Fscience%5C%2Farticle%5C%2Fpii%5C%2FS1053811922002129%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222022-11-21T15%3A56%3A20Z%22%7D%7D%2C%7B%22key%22%3A%224KSV7F5L%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Reinke%22%2C%22parsedDate%22%3A%222021-10-06%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BReinke%2C%20A.%20%282021%2C%20October%206%29.%20%26lt%3Bb%26gt%3BA%20discovery%20dive%20into%20the%20world%20of%20evaluation%20%26%23x2014%3B%20Do%26%23x2019%3Bs%20don%26%23x2019%3Bts%20and%20other%20considerations%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BMICCAI%20Educational%20Initiative%26lt%3B%5C%2Fi%26gt%3B.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fmedium.com%5C%2Fmiccai-educational-initiative%5C%2Fa-discovery-dive-into-the-world-of-evaluation-dos-don-ts-and-other-considerations-4189ab46fe06%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fmedium.com%5C%2Fmiccai-educational-initiative%5C%2Fa-discovery-dive-into-the-world-of-evaluation-dos-don-ts-and-other-considerations-4189ab46fe06%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22blogPost%22%2C%22title%22%3A%22A%20discovery%20dive%20into%20the%20world%20of%20evaluation%20%5Cu2014%20Do%5Cu2019s%20don%5Cu2019ts%20and%20other%20considerations%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%5D%2C%22abstractNote%22%3A%22written%20by%20Annika%20Reinke%2C%20Minu%20D.%20Tizabi%20and%20Carole%20H.%20Sudre%22%2C%22blogTitle%22%3A%22MICCAI%20Educational%20Initiative%22%2C%22date%22%3A%222021-10-06T12%3A51%3A49.645Z%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fmedium.com%5C%2Fmiccai-educational-initiative%5C%2Fa-discovery-dive-into-the-world-of-evaluation-dos-don-ts-and-other-considerations-4189ab46fe06%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222022-07-05T10%3A11%3A47Z%22%7D%7D%2C%7B%22key%22%3A%225YVY4PXX%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Ro%5Cu00df%20et%20al.%22%2C%22parsedDate%22%3A%222021-05-01%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BRo%26%23xDF%3B%2C%20T.%2C%20Reinke%2C%20A.%2C%20Full%2C%20P.%20M.%2C%20Wagner%2C%20M.%2C%20Kenngott%2C%20H.%2C%20Apitz%2C%20M.%2C%20Hempe%2C%20H.%2C%20Mindroc-Filimon%2C%20D.%2C%20Scholz%2C%20P.%2C%20Tran%2C%20T.%20N.%2C%20Bruno%2C%20P.%2C%20Arbel%26%23xE1%3Bez%2C%20P.%2C%20Bian%2C%20G.-B.%2C%20Bodenstedt%2C%20S.%2C%20Bolmgren%2C%20J.%20L.%2C%20Bravo-S%26%23xE1%3Bnchez%2C%20L.%2C%20Chen%2C%20H.-B.%2C%20Gonz%26%23xE1%3Blez%2C%20C.%2C%20Guo%2C%20D.%2C%20%26%23x2026%3B%20Maier-Hein%2C%20L.%20%282021%29.%20%26lt%3Bb%26gt%3BComparative%20validation%20of%20multi-instance%20instrument%20segmentation%20in%20endoscopy%3A%20Results%20of%20the%20ROBUST-MIS%202019%20challenge%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BMedical%20Image%20Analysis%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B70%26lt%3B%5C%2Fi%26gt%3B%2C%20101920.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2020.101920%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2020.101920%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Comparative%20validation%20of%20multi-instance%20instrument%20segmentation%20in%20endoscopy%3A%20Results%20of%20the%20ROBUST-MIS%202019%20challenge%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tobias%22%2C%22lastName%22%3A%22Ro%5Cu00df%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%20M.%22%2C%22lastName%22%3A%22Full%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Wagner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hannes%22%2C%22lastName%22%3A%22Kenngott%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Apitz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hellena%22%2C%22lastName%22%3A%22Hempe%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Diana%22%2C%22lastName%22%3A%22Mindroc-Filimon%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Scholz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thuy%20Nuong%22%2C%22lastName%22%3A%22Tran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierangela%22%2C%22lastName%22%3A%22Bruno%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pablo%22%2C%22lastName%22%3A%22Arbel%5Cu00e1ez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gui-Bin%22%2C%22lastName%22%3A%22Bian%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sebastian%22%2C%22lastName%22%3A%22Bodenstedt%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jon%20Lindstr%5Cu00f6m%22%2C%22lastName%22%3A%22Bolmgren%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%22%2C%22lastName%22%3A%22Bravo-S%5Cu00e1nchez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hua-Bin%22%2C%22lastName%22%3A%22Chen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Cristina%22%2C%22lastName%22%3A%22Gonz%5Cu00e1lez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dong%22%2C%22lastName%22%3A%22Guo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22P%5Cu00e5l%22%2C%22lastName%22%3A%22Halvorsen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pheng-Ann%22%2C%22lastName%22%3A%22Heng%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Enes%22%2C%22lastName%22%3A%22Hosgor%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zeng-Guang%22%2C%22lastName%22%3A%22Hou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Isensee%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Debesh%22%2C%22lastName%22%3A%22Jha%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tingting%22%2C%22lastName%22%3A%22Jiang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yueming%22%2C%22lastName%22%3A%22Jin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kadir%22%2C%22lastName%22%3A%22Kirtac%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sabrina%22%2C%22lastName%22%3A%22Kletz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stefan%22%2C%22lastName%22%3A%22Leger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhixuan%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%20H.%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Zhen-Liang%22%2C%22lastName%22%3A%22Ni%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%20A.%22%2C%22lastName%22%3A%22Riegler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Klaus%22%2C%22lastName%22%3A%22Schoeffmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ruohua%22%2C%22lastName%22%3A%22Shi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stefanie%22%2C%22lastName%22%3A%22Speidel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michael%22%2C%22lastName%22%3A%22Stenzel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Isabell%22%2C%22lastName%22%3A%22Twick%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gutai%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jiacheng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Liansheng%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lu%22%2C%22lastName%22%3A%22Wang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yujie%22%2C%22lastName%22%3A%22Zhang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yan-Jie%22%2C%22lastName%22%3A%22Zhou%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lei%22%2C%22lastName%22%3A%22Zhu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuel%22%2C%22lastName%22%3A%22Wiesenfarth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beat%20P.%22%2C%22lastName%22%3A%22M%5Cu00fcller-Stich%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22Intraoperative%20tracking%20of%20laparoscopic%20instruments%20is%20often%20a%20prerequisite%20for%20computer%20and%20robotic-assisted%20interventions.%20While%20numerous%20methods%20for%20detecting%2C%20segmenting%20and%20tracking%20of%20medical%20instruments%20based%20on%20endoscopic%20video%20images%20have%20been%20proposed%20in%20the%20literature%2C%20key%20limitations%20remain%20to%20be%20addressed%3A%20Firstly%2C%20robustness%2C%20that%20is%2C%20the%20reliable%20performance%20of%20state-of-the-art%20methods%20when%20run%20on%20challenging%20images%20%28e.g.%20in%20the%20presence%20of%20blood%2C%20smoke%20or%20motion%20artifacts%29.%20Secondly%2C%20generalization%3B%20algorithms%20trained%20for%20a%20specific%20intervention%20in%20a%20specific%20hospital%20should%20generalize%20to%20other%20interventions%20or%20institutions.%20In%20an%20effort%20to%20promote%20solutions%20for%20these%20limitations%2C%20we%20organized%20the%20Robust%20Medical%20Instrument%20Segmentation%20%28ROBUST-MIS%29%20challenge%20as%20an%20international%20benchmarking%20competition%20with%20a%20specific%20focus%20on%20the%20robustness%20and%20generalization%20capabilities%20of%20algorithms.%20For%20the%20first%20time%20in%20the%20field%20of%20endoscopic%20image%20processing%2C%20our%20challenge%20included%20a%20task%20on%20binary%20segmentation%20and%20also%20addressed%20multi-instance%20detection%20and%20segmentation.%20The%20challenge%20was%20based%20on%20a%20surgical%20data%20set%20comprising%2010%2C040%20annotated%20images%20acquired%20from%20a%20total%20of%2030%20surgical%20procedures%20from%20three%20different%20types%20of%20surgery.%20The%20validation%20of%20the%20competing%20methods%20for%20the%20three%20tasks%20%28binary%20segmentation%2C%20multi-instance%20detection%20and%20multi-instance%20segmentation%29%20was%20performed%20in%20three%20different%20stages%20with%20an%20increasing%20domain%20gap%20between%20the%20training%20and%20the%20test%20data.%20The%20results%20confirm%20the%20initial%20hypothesis%2C%20namely%20that%20algorithm%20performance%20degrades%20with%20an%20increasing%20domain%20gap.%20While%20the%20average%20detection%20and%20segmentation%20quality%20of%20the%20best-performing%20algorithms%20is%20high%2C%20future%20research%20should%20concentrate%20on%20detection%20and%20segmentation%20of%20small%2C%20crossing%2C%20moving%20and%20transparent%20instrument%28s%29%20%28parts%29.%22%2C%22date%22%3A%222021-05-01%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1016%5C%2Fj.media.2020.101920%22%2C%22ISSN%22%3A%221361-8415%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.sciencedirect.com%5C%2Fscience%5C%2Farticle%5C%2Fpii%5C%2FS136184152030284X%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222022-07-14T06%3A33%3A25Z%22%7D%7D%2C%7B%22key%22%3A%22DPDT6I36%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Maier-Hein%20et%20al.%22%2C%22parsedDate%22%3A%222021-02-23%22%2C%22numChildren%22%3A3%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BMaier-Hein%2C%20L.%2C%20Wagner%2C%20M.%2C%20Ross%2C%20T.%2C%20Reinke%2C%20A.%2C%20Bodenstedt%2C%20S.%2C%20Full%2C%20P.%20M.%2C%20Hempe%2C%20H.%2C%20Mindroc-Filimon%2C%20D.%2C%20Scholz%2C%20P.%2C%20Tran%2C%20T.%20N.%2C%20Bruno%2C%20P.%2C%20Kisilenko%2C%20A.%2C%20M%26%23xFC%3Bller%2C%20B.%2C%20Davitashvili%2C%20T.%2C%20Capek%2C%20M.%2C%20Tizabi%2C%20M.%2C%20Eisenmann%2C%20M.%2C%20Adler%2C%20T.%20J.%2C%20Gr%26%23xF6%3Bhl%2C%20J.%2C%20%26%23x2026%3B%20M%26%23xFC%3Bller-Stich%2C%20B.%20P.%20%282021%29.%20%26lt%3Bi%26gt%3B%26lt%3Bb%26gt%3B%26lt%3Bspan%20style%3D%26quot%3Bfont-style%3Anormal%3B%26quot%3B%26gt%3BHeidelberg%20Colorectal%20Data%20Set%20for%20Surgical%20Data%20Science%20in%20the%20Sensor%20Operating%20Room%26lt%3B%5C%2Fspan%26gt%3B%26lt%3B%5C%2Fb%26gt%3B%26lt%3B%5C%2Fi%26gt%3B%20%28arXiv%3A2005.03501%29.%20arXiv.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2005.03501%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.48550%5C%2FarXiv.2005.03501%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Heidelberg%20Colorectal%20Data%20Set%20for%20Surgical%20Data%20Science%20in%20the%20Sensor%20Operating%20Room%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Wagner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tobias%22%2C%22lastName%22%3A%22Ross%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sebastian%22%2C%22lastName%22%3A%22Bodenstedt%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%20M.%22%2C%22lastName%22%3A%22Full%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hellena%22%2C%22lastName%22%3A%22Hempe%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Diana%22%2C%22lastName%22%3A%22Mindroc-Filimon%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Patrick%22%2C%22lastName%22%3A%22Scholz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thuy%20Nuong%22%2C%22lastName%22%3A%22Tran%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierangela%22%2C%22lastName%22%3A%22Bruno%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anna%22%2C%22lastName%22%3A%22Kisilenko%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Benjamin%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tornike%22%2C%22lastName%22%3A%22Davitashvili%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuela%22%2C%22lastName%22%3A%22Capek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Minu%22%2C%22lastName%22%3A%22Tizabi%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tim%20J.%22%2C%22lastName%22%3A%22Adler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Janek%22%2C%22lastName%22%3A%22Gr%5Cu00f6hl%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Melanie%22%2C%22lastName%22%3A%22Schellenberg%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Silvia%22%2C%22lastName%22%3A%22Seidlitz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22T.%20Y.%20Emmy%22%2C%22lastName%22%3A%22Lai%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22B%5Cu00fcnyamin%22%2C%22lastName%22%3A%22Pekdemir%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Veith%22%2C%22lastName%22%3A%22Roethlingshoefer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fabian%22%2C%22lastName%22%3A%22Both%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sebastian%22%2C%22lastName%22%3A%22Bittel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Marc%22%2C%22lastName%22%3A%22Mengler%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lars%22%2C%22lastName%22%3A%22M%5Cu00fcndermann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Martin%22%2C%22lastName%22%3A%22Apitz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stefanie%22%2C%22lastName%22%3A%22Speidel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hannes%20G.%22%2C%22lastName%22%3A%22Kenngott%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Beat%20P.%22%2C%22lastName%22%3A%22M%5Cu00fcller-Stich%22%7D%5D%2C%22abstractNote%22%3A%22Image-based%20tracking%20of%20medical%20instruments%20is%20an%20integral%20part%20of%20surgical%20data%20science%20applications.%20Previous%20research%20has%20addressed%20the%20tasks%20of%20detecting%2C%20segmenting%20and%20tracking%20medical%20instruments%20based%20on%20laparoscopic%20video%20data.%20However%2C%20the%20proposed%20methods%20still%20tend%20to%20fail%20when%20applied%20to%20challenging%20images%20and%20do%20not%20generalize%20well%20to%20data%20they%20have%20not%20been%20trained%20on.%20This%20paper%20introduces%20the%20Heidelberg%20Colorectal%20%28HeiCo%29%20data%20set%20-%20the%20first%20publicly%20available%20data%20set%20enabling%20comprehensive%20benchmarking%20of%20medical%20instrument%20detection%20and%20segmentation%20algorithms%20with%20a%20specific%20emphasis%20on%20method%20robustness%20and%20generalization%20capabilities.%20Our%20data%20set%20comprises%2030%20laparoscopic%20videos%20and%20corresponding%20sensor%20data%20from%20medical%20devices%20in%20the%20operating%20room%20for%20three%20different%20types%20of%20laparoscopic%20surgery.%20Annotations%20include%20surgical%20phase%20labels%20for%20all%20video%20frames%20as%20well%20as%20information%20on%20instrument%20presence%20and%20corresponding%20instance-wise%20segmentation%20masks%20for%20surgical%20instruments%20%28if%20any%29%20in%20more%20than%2010%2C000%20individual%20frames.%20The%20data%20has%20successfully%20been%20used%20to%20organize%20international%20competitions%20within%20the%20Endoscopic%20Vision%20Challenges%202017%20and%202019.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2005.03501%22%2C%22date%22%3A%222021-02-23%22%2C%22DOI%22%3A%2210.48550%5C%2FarXiv.2005.03501%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2005.03501%22%2C%22language%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222023-03-02T13%3A54%3A45Z%22%7D%7D%2C%7B%22key%22%3A%22IKIHGIR4%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Wiesenfarth%20et%20al.%22%2C%22parsedDate%22%3A%222021-01-27%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BWiesenfarth%2C%20M.%2C%20Reinke%2C%20A.%2C%20Landman%2C%20B.%20A.%2C%20Eisenmann%2C%20M.%2C%20Saiz%2C%20L.%20A.%2C%20Cardoso%2C%20M.%20J.%2C%20Maier-Hein%2C%20L.%2C%20%26amp%3B%20Kopp-Schneider%2C%20A.%20%282021%29.%20%26lt%3Bb%26gt%3BMethods%20and%20open-source%20toolkit%20for%20analyzing%20and%20visualizing%20challenge%20results%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BScientific%20Reports%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B11%26lt%3B%5C%2Fi%26gt%3B%281%29%2C%202369.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41598-021-82017-6%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1038%5C%2Fs41598-021-82017-6%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Methods%20and%20open-source%20toolkit%20for%20analyzing%20and%20visualizing%20challenge%20results%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Manuel%22%2C%22lastName%22%3A%22Wiesenfarth%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%20A.%22%2C%22lastName%22%3A%22Landman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Laura%20Aguilera%22%2C%22lastName%22%3A%22Saiz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20Jorge%22%2C%22lastName%22%3A%22Cardoso%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%5D%2C%22abstractNote%22%3A%22Grand%20challenges%20have%20become%20the%20de%20facto%20standard%20for%20benchmarking%20image%20analysis%20algorithms.%20While%20the%20number%20of%20these%20international%20competitions%20is%20steadily%20increasing%2C%20surprisingly%20little%20effort%20has%20been%20invested%20in%20ensuring%20high%20quality%20design%2C%20execution%20and%20reporting%20for%20these%20international%20competitions.%20Specifically%2C%20results%20analysis%20and%20visualization%20in%20the%20event%20of%20uncertainties%20have%20been%20given%20almost%20no%20attention%20in%20the%20literature.%20Given%20these%20shortcomings%2C%20the%20contribution%20of%20this%20paper%20is%20two-fold%3A%20%281%29%20we%20present%20a%20set%20of%20methods%20to%20comprehensively%20analyze%20and%20visualize%20the%20results%20of%20single-task%20and%20multi-task%20challenges%20and%20apply%20them%20to%20a%20number%20of%20simulated%20and%20real-life%20challenges%20to%20demonstrate%20their%20specific%20strengths%20and%20weaknesses%3B%20%282%29%20we%20release%20the%20open-source%20framework%20challengeR%20as%20part%20of%20this%20work%20to%20enable%20fast%20and%20wide%20adoption%20of%20the%20methodology%20proposed%20in%20this%20paper.%20Our%20approach%20offers%20an%20intuitive%20way%20to%20gain%20important%20insights%20into%20the%20relative%20and%20absolute%20performance%20of%20algorithms%2C%20which%20cannot%20be%20revealed%20by%20commonly%20applied%20visualization%20techniques.%20This%20is%20demonstrated%20by%20the%20experiments%20performed%20in%20the%20specific%20context%20of%20biomedical%20image%20analysis%20challenges.%20Our%20framework%20could%20thus%20become%20an%20important%20tool%20for%20analyzing%20and%20visualizing%20challenge%20results%20in%20the%20field%20of%20biomedical%20image%20analysis%20and%20beyond.%22%2C%22date%22%3A%222021-01-27%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1038%5C%2Fs41598-021-82017-6%22%2C%22ISSN%22%3A%222045-2322%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.nature.com%5C%2Farticles%5C%2Fs41598-021-82017-6%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222023-01-26T07%3A40%3A17Z%22%7D%7D%2C%7B%22key%22%3A%22VRKPREQG%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Maier-Hein%20et%20al.%22%2C%22parsedDate%22%3A%222020-12-01%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BMaier-Hein%2C%20L.%2C%20Reinke%2C%20A.%2C%20Kozubek%2C%20M.%2C%20Martel%2C%20A.%20L.%2C%20Arbel%2C%20T.%2C%20Eisenmann%2C%20M.%2C%20Hanbury%2C%20A.%2C%20Jannin%2C%20P.%2C%20M%26%23xFC%3Bller%2C%20H.%2C%20Onogur%2C%20S.%2C%20Saez-Rodriguez%2C%20J.%2C%20van%20Ginneken%2C%20B.%2C%20Kopp-Schneider%2C%20A.%2C%20%26amp%3B%20Landman%2C%20B.%20A.%20%282020%29.%20%26lt%3Bb%26gt%3BBIAS%3A%20Transparent%20reporting%20of%20biomedical%20image%20analysis%20challenges%26lt%3B%5C%2Fb%26gt%3B.%20%26lt%3Bi%26gt%3BMedical%20Image%20Analysis%26lt%3B%5C%2Fi%26gt%3B%2C%20%26lt%3Bi%26gt%3B66%26lt%3B%5C%2Fi%26gt%3B%2C%20101796.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2020.101796%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2Fj.media.2020.101796%26lt%3B%5C%2Fa%26gt%3B%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22BIAS%3A%20Transparent%20reporting%20of%20biomedical%20image%20analysis%20challenges%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michal%22%2C%22lastName%22%3A%22Kozubek%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anne%20L.%22%2C%22lastName%22%3A%22Martel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tal%22%2C%22lastName%22%3A%22Arbel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Eisenmann%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Allan%22%2C%22lastName%22%3A%22Hanbury%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Jannin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Henning%22%2C%22lastName%22%3A%22M%5Cu00fcller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sinan%22%2C%22lastName%22%3A%22Onogur%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Julio%22%2C%22lastName%22%3A%22Saez-Rodriguez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bram%22%2C%22lastName%22%3A%22van%20Ginneken%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annette%22%2C%22lastName%22%3A%22Kopp-Schneider%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bennett%20A.%22%2C%22lastName%22%3A%22Landman%22%7D%5D%2C%22abstractNote%22%3A%22The%20number%20of%20biomedical%20image%20analysis%20challenges%20organized%20per%20year%20is%20steadily%20increasing.%20These%20international%20competitions%20have%20the%20purpose%20of%20benchmarking%20algorithms%20on%20common%20data%20sets%2C%20typically%20to%20identify%20the%20best%20method%20for%20a%20given%20problem.%20Recent%20research%2C%20however%2C%20revealed%20that%20common%20practice%20related%20to%20challenge%20reporting%20does%20not%20allow%20for%20adequate%20interpretation%20and%20reproducibility%20of%20results.%20To%20address%20the%20discrepancy%20between%20the%20impact%20of%20challenges%20and%20the%20quality%20%28control%29%2C%20the%20Biomedical%20Image%20Analysis%20ChallengeS%20%28BIAS%29%20initiative%20developed%20a%20set%20of%20recommendations%20for%20the%20reporting%20of%20challenges.%20The%20BIAS%20statement%20aims%20to%20improve%20the%20transparency%20of%20the%20reporting%20of%20a%20biomedical%20image%20analysis%20challenge%20regardless%20of%20field%20of%20application%2C%20image%20modality%20or%20task%20category%20assessed.%20This%20article%20describes%20how%20the%20BIAS%20statement%20was%20developed%20and%20presents%20a%20checklist%20which%20authors%20of%20biomedical%20image%20analysis%20challenges%20are%20encouraged%20to%20include%20in%20their%20submission%20when%20giving%20a%20paper%20on%20a%20challenge%20into%20review.%20The%20purpose%20of%20the%20checklist%20is%20to%20standardize%20and%20facilitate%20the%20review%20process%20and%20raise%20interpretability%20and%20reproducibility%20of%20challenge%20results%20by%20making%20relevant%20information%20explicit.%22%2C%22date%22%3A%222020-12-01%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1016%5C%2Fj.media.2020.101796%22%2C%22ISSN%22%3A%221361-8415%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.sciencedirect.com%5C%2Fscience%5C%2Farticle%5C%2Fpii%5C%2FS1361841520301602%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222022-07-14T06%3A47%3A53Z%22%7D%7D%2C%7B%22key%22%3A%227KLGCCCD%22%2C%22library%22%3A%7B%22id%22%3A4725570%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Reinke%20and%20Maier-Hein%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BReinke%2C%20A.%2C%20%26amp%3B%20Maier-Hein%2C%20L.%20%28n.d.%29.%20%26lt%3Bb%26gt%3BCommon%20limitations%20of%20performance%20metrics%20in%20biomedical%20image%20analysis%26lt%3B%5C%2Fb%26gt%3B.%203.%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Common%20limitations%20of%20performance%20metrics%20in%20biomedical%20image%20analysis%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Annika%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lena%22%2C%22lastName%22%3A%22Maier-Hein%22%7D%5D%2C%22abstractNote%22%3A%22While%20the%20importance%20of%20automatic%20biomedical%20image%20analysis%20is%20increasing%20at%20an%20enormous%20pace%2C%20recent%20meta-research%20revealed%20major%20%5Cufb02aws%20with%20respect%20to%20algorithm%20validation.%20Performance%20metrics%20are%20key%20for%20objective%2C%20transparent%20and%20comparative%20performance%20assessment%2C%20but%20little%20attention%20has%20been%20given%20to%20their%20pitfalls.%20Under%20the%20umbrella%20of%20the%20Helmholtz%20Imaging%20Platform%20%28HIP%29%2C%20three%20international%20initiatives%20%5Cu2013%20the%20MICCAI%20Society%5Cu2019s%20challenge%20working%20group%2C%20the%20Biomedical%20Image%20Analysis%20Challenges%20%28BIAS%29%20initiative%2C%20as%20well%20as%20the%20benchmarking%20working%20group%20of%20the%20MONAI%20framework%20%5Cu2013%20have%20now%20joined%20forces%20with%20the%20mission%20to%20generate%20best%20practice%20recommendations%20with%20respect%20to%20metrics%20in%20medical%20image%20analysis.%20Consensus%20building%20is%20achieved%20via%20a%20Delphi%20process%2C%20a%20popular%20tool%20for%20integrating%20opinions%20in%20large%20international%20consortia.%20The%20current%20document%20serves%20as%20a%20teaser%20for%20the%20results%20presentation%20and%20focuses%20on%20the%20pitfalls%20of%20the%20most%20commonly%20used%20metric%20in%20biomedical%20image%20analysis%2C%20the%20Dice%20Similarity%20Coe%5Cufb03cient%20%28DSC%20%29%2C%20in%20the%20categories%20of%20%281%29%20mathematical%20properties%5C%2Fedge%20cases%2C%20%282%29%20task%5C%2Fmetric%20%5Cufb01t%20and%20%283%29%20metric%20aggregation.%20Being%20compiled%20by%20a%20large%20group%20of%20experts%20from%20more%20than%2030%20institutes%20worldwide%2C%20we%20believe%20that%20our%20framework%20could%20be%20of%20general%20interest%20to%20the%20MIDL%20community%20and%20will%20improve%20the%20quality%20of%20biomedical%20image%20analysis%20algorithm%20validation.%22%2C%22date%22%3A%22%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222022-07-05T10%3A05%3A56Z%22%7D%7D%5D%7D
Godau, P., Kalinowski, P., Christodoulou, E., Reinke, A., Tizabi, M., Ferrer, L., Jäger, P., & Maier-Hein, L. (2025). Navigating prevalence shifts in image analysis algorithm deployment. Medical Image Analysis, 102, 103504. https://doi.org/10.1016/j.media.2025.103504
Adler, T. J., Nölke, J.-H., Reinke, A., Tizabi, M. D., Gruber, S., Trofimova, D., Ardizzone, L., Jaeger, P. F., Buettner, F., Köthe, U., & Maier-Hein, L. (2025). Application-driven validation of posteriors in inverse problems. Medical Image Analysis, 101, 103474. https://doi.org/10.1016/j.media.2025.103474
Cimini, B. A., Bankhead, P., D’Antuono, R., Fazeli, E., Fernandez-Rodriguez, J., Fuster-Barceló, C., Haase, R., Jambor, H. K., Jones, M. L., Jug, F., Klemm, A. H., Kreshuk, A., Marcotti, S., Martins, G. G., McArdle, S., Miura, K., Muñoz-Barrutia, A., Murphy, L. C., Nelson, M. S., … Eliceiri, K. W. (2024). The crucial role of bioimage analysts in scientific research and publication. Journal of Cell Science, 137(20), jcs262322. https://doi.org/10.1242/jcs.262322
Christodoulou, E., Reinke, A., Houhou, R., Kalinowski, P., Erkan, S., Sudre, C. H., Burgos, N., Boutaj, S., Loizillon, S., Solal, M., Rieke, N., Cheplygina, V., Antonelli, M., Mayer, L. D., Tizabi, M. D., Cardoso, M. J., Simpson, A., Jäger, P. F., Kopp-Schneider, A., … Maier-Hein, L. (2024). Confidence intervals uncovered: Are we ready for real-world medical imaging AI? (arXiv:2409.17763). arXiv. https://doi.org/10.48550/arXiv.2409.17763
Rädsch, T., Reinke, A., Weru, V., Tizabi, M. D., Heller, N., Isensee, F., Kopp-Schneider, A., & Maier-Hein, L. (2024). Quality Assured: Rethinking Annotation Strategies in Imaging AI (arXiv:2407.17596). arXiv. https://doi.org/10.48550/arXiv.2407.17596
Floca, R., Bohn, J., Haux, C., Wiestler, B., Zöllner, F. G., Reinke, A., Weiß, J., Nolden, M., Albert, S., Persigehl, T., Norajitra, T., Baeßler, B., Dewey, M., Braren, R., Büchert, M., Fallenberg, E. M., Galldiks, N., Gerken, A., Götz, M., … Bamberg, F. (2024). Radiomics workflow definition & challenges - German priority program 2177 consensus statement on clinically applied radiomics. Insights into Imaging, 15(1), 124. https://doi.org/10.1186/s13244-024-01704-w
Mais, L., Hirsch, P., Managan, C., Kandarpa, R., Rumberger, J. L., Reinke, A., Maier-Hein, L., Ihrke, G., & Kainmueller, D. (2024). FISBe: A Real-World Benchmark Dataset for Instance Segmentation of Long-Range thin Filamentous Structures. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 22249–22259. https://doi.org/10.1109/CVPR52733.2024.02100
Reinke, A., Tizabi, M. D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Sudre, C. H., Acion, L., Antonelli, M., Arbel, T., Bakas, S., Benis, A., Buettner, F., Cardoso, M. J., Cheplygina, V., Chen, J., Christodoulou, E., Cimini, B. A., … Maier-Hein, L. (2024). Understanding metric-related pitfalls in image analysis validation. Nature Methods, 1–13. https://doi.org/10.1038/s41592-023-02150-0
Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M. D., Buettner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., Reyes, M., Riegler, M. A., Wiesenfarth, M., Kavur, A. E., Sudre, C. H., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Rädsch, T., … Jäger, P. F. (2024). Metrics reloaded: recommendations for image analysis validation. Nature Methods, 21(2), 195–212. https://doi.org/10.1038/s41592-023-02151-z
Kofler, F., Möller, H., Buchner, J. A., de la Rosa, E., Ezhov, I., Rosier, M., Mekki, I., Shit, S., Negwer, M., Al-Maskari, R., Ertürk, A., Vinayahalingam, S., Isensee, F., Pati, S., Rueckert, D., Kirschke, J. S., Ehrlich, S. K., Reinke, A., Menze, B., … Piraud, M. (2023). Panoptica -- instance-wise evaluation of 3D semantic and instance segmentation maps (arXiv:2312.02608). arXiv. https://doi.org/10.48550/arXiv.2312.02608
Brandenburg, J. M., Jenke, A. C., Stern, A., Daum, M. T. J., Schulze, A., Younis, R., Petrynowski, P., Davitashvili, T., Vanat, V., Bhasker, N., Schneider, S., Mündermann, L., Reinke, A., Kolbinger, F. R., Jörns, V., Fritz-Kebede, F., Dugas, M., Maier-Hein, L., Klotz, R., … Wagner, M. (2023). Active learning for extracting surgomic features in robot-assisted minimally invasive esophagectomy: a prospective annotation study. Surgical Endoscopy, 37(11), 8577–8593. https://doi.org/10.1007/s00464-023-10447-6
Tran, T. N., Adler, T. J., Yamlahi, A., Christodoulou, E., Godau, P., Reinke, A., Tizabi, M. D., Sauer, P., Persicke, T., Albert, J. G., & Maier-Hein, L. (2023). Sources of performance variability in deep learning-based polyp detection. International Journal of Computer Assisted Radiology and Surgery, 18(7), 1311–1322. https://doi.org/10.1007/s11548-023-02936-9
Wagner, M., Müller-Stich, B.-P., Kisilenko, A., Tran, D., Heger, P., Mündermann, L., Lubotsky, D. M., Müller, B., Davitashvili, T., Capek, M., Reinke, A., Reid, C., Yu, T., Vardazaryan, A., Nwoye, C. I., Padoy, N., Liu, X., Lee, E.-J., Disch, C., … Bodenstedt, S. (2023). Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark. Medical Image Analysis, 86, 102770. https://doi.org/10.1016/j.media.2023.102770
Eisenmann, M., Reinke, A., Weru, V., Tizabi, M. D., Isensee, F., Adler, T. J., Ali, S., Andrearczyk, V., Aubreville, M., Baid, U., Bakas, S., Balu, N., Bano, S., Bernal, J., Bodenstedt, S., Casella, A., Cheplygina, V., Daum, M., de Bruijne, M., … Maier-Hein, L. (2023). Why is the winner the best? (arXiv:2303.17719). arXiv. https://doi.org/10.48550/arXiv.2303.17719
Rädsch, T., Reinke, A., Weru, V., Tizabi, M. D., Schreck, N., Kavur, A. E., Pekdemir, B., Roß, T., Kopp-Schneider, A., & Maier-Hein, L. (2023). Labelling instructions matter in biomedical image analysis. Nature Machine Intelligence, 1–11. https://doi.org/10.1038/s42256-023-00625-5
Roß, T., Bruno, P., Reinke, A., Wiesenfarth, M., Koeppel, L., Full, P. M., Pekdemir, B., Godau, P., Trofimova, D., Isensee, F., Adler, T. J., Tran, T. N., Moccia, S., Calimeri, F., Müller-Stich, B. P., Kopp-Schneider, A., & Maier-Hein, L. (2023). Beyond rankings: Learning (more) from algorithm validation. Medical Image Analysis, 102765. https://doi.org/10.1016/j.media.2023.102765
Godau, P., Kalinowski, P., Christodoulou, E., Reinke, A., Tizabi, M., Ferrer, L., Jäger, P., & Maier-Hein, L. (2023). Deployment of Image Analysis Algorithms under Prevalence Shifts. https://doi.org/10.48550/ARXIV.2303.12540
Eisenmann, M., Reinke, A., Weru, V., Tizabi, M. D., Isensee, F., Adler, T. J., Godau, P., Cheplygina, V., Kozubek, M., Ali, S., Gupta, A., Kybic, J., Noble, A., de Solórzano, C. O., Pachade, S., Petitjean, C., Sage, D., Wei, D., Wilden, E., … Maier-Hein, L. (2022). Biomedical image analysis competitions: The state of current participation practice (arXiv:2212.08568). arXiv. https://doi.org/10.48550/arXiv.2212.08568
Tran, T. N., Adler, T., Yamlahi, A., Christodoulou, E., Godau, P., Reinke, A., Tizabi, M. D., Sauer, P., Persicke, T., Albert, J. G., & Maier-Hein, L. (2022). Sources of performance variability in deep learning-based polyp detection (arXiv:2211.09708). arXiv. https://doi.org/10.48550/arXiv.2211.09708
Zimmerer, D., Full, P. M., Isensee, F., Jäger, P., Adler, T., Petersen, J., Köhler, G., Ross, T., Reinke, A., Kascenas, A., Jensen, B. S., O’Neil, A. Q., Tan, J., Hou, B., Batten, J., Qiu, H., Kainz, B., Shvetsova, N., Fedulova, I., … Maier-Hein, K. (2022). MOOD 2020: A Public Benchmark for Out-of-Distribution Detection and Localization on Medical Images. IEEE Transactions on Medical Imaging, 41(10), 2728–2738. https://doi.org/10.1109/TMI.2022.3170077
Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M. D., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., Reyes, M., Riegler, M. A., Wiesenfarth, M., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Acion, L., Antonelli, M., … Jäger, P. F. (2022). Metrics reloaded: Pitfalls and recommendations for image analysis validation (arXiv:2206.01653). arXiv. https://doi.org/10.48550/arXiv.2206.01653
Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B. A., Litjens, G., Menze, B., Ronneberger, O., Summers, R. M., van Ginneken, B., Bilello, M., Bilic, P., Christ, P. F., Do, R. K. G., Gollub, M. J., Heckers, S. H., Huisman, H., Jarnagin, W. R., … Cardoso, M. J. (2022). The Medical Segmentation Decathlon. Nature Communications, 13(1), 4128. https://doi.org/10.1038/s41467-022-30695-9
Reinke, A., Tizabi, M. D., Sudre, C. H., Eisenmann, M., Rädsch, T., Baumgartner, M., Acion, L., Antonelli, M., Arbel, T., Bakas, S., Bankhead, P., Benis, A., Cardoso, M. J., Cheplygina, V., Christodoulou, E., Cimini, B., Collins, G. S., Farahani, K., van Ginneken, B., … Maier-Hein, L. (2022). Common Limitations of Image Processing Metrics: A Picture Story (arXiv:2104.05642). arXiv. https://doi.org/10.48550/arXiv.2104.05642
Schellenberg, M., Dreher, K. K., Holzwarth, N., Isensee, F., Reinke, A., Schreck, N., Seitel, A., Tizabi, M. D., Maier-Hein, L., & Gröhl, J. (2022). Semantic segmentation of multispectral photoacoustic images using deep learning. Photoacoustics, 26, 100341. https://doi.org/10.1016/j.pacs.2022.100341
Bron, E. E., Klein, S., Reinke, A., Papma, J. M., Maier-Hein, L., Alexander, D. C., & Oxtoby, N. P. (2022). Ten years of image analysis and machine learning competitions in dementia. NeuroImage, 253, 119083. https://doi.org/10.1016/j.neuroimage.2022.119083
Reinke, A. (2021, October 6). A discovery dive into the world of evaluation — Do’s don’ts and other considerations. MICCAI Educational Initiative. https://medium.com/miccai-educational-initiative/a-discovery-dive-into-the-world-of-evaluation-dos-don-ts-and-other-considerations-4189ab46fe06
Roß, T., Reinke, A., Full, P. M., Wagner, M., Kenngott, H., Apitz, M., Hempe, H., Mindroc-Filimon, D., Scholz, P., Tran, T. N., Bruno, P., Arbeláez, P., Bian, G.-B., Bodenstedt, S., Bolmgren, J. L., Bravo-Sánchez, L., Chen, H.-B., González, C., Guo, D., … Maier-Hein, L. (2021). Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge. Medical Image Analysis, 70, 101920. https://doi.org/10.1016/j.media.2020.101920
Maier-Hein, L., Wagner, M., Ross, T., Reinke, A., Bodenstedt, S., Full, P. M., Hempe, H., Mindroc-Filimon, D., Scholz, P., Tran, T. N., Bruno, P., Kisilenko, A., Müller, B., Davitashvili, T., Capek, M., Tizabi, M., Eisenmann, M., Adler, T. J., Gröhl, J., … Müller-Stich, B. P. (2021). Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room (arXiv:2005.03501). arXiv. https://doi.org/10.48550/arXiv.2005.03501
Wiesenfarth, M., Reinke, A., Landman, B. A., Eisenmann, M., Saiz, L. A., Cardoso, M. J., Maier-Hein, L., & Kopp-Schneider, A. (2021). Methods and open-source toolkit for analyzing and visualizing challenge results. Scientific Reports, 11(1), 2369. https://doi.org/10.1038/s41598-021-82017-6
Maier-Hein, L., Reinke, A., Kozubek, M., Martel, A. L., Arbel, T., Eisenmann, M., Hanbury, A., Jannin, P., Müller, H., Onogur, S., Saez-Rodriguez, J., van Ginneken, B., Kopp-Schneider, A., & Landman, B. A. (2020). BIAS: Transparent reporting of biomedical image analysis challenges. Medical Image Analysis, 66, 101796. https://doi.org/10.1016/j.media.2020.101796
Reinke, A., & Maier-Hein, L. (n.d.). Common limitations of performance metrics in biomedical image analysis. 3.
