Ranking¶

Task 1: Fast Multi-class Segmentation¶

The metrics used to rank the submission of Task 1 are the Dice Similarity Coefficient (DSC), the 95th percentile Hausdorff Distance (HD95), the inference execution time (Time), and the maximum memory usage (Mem). The ranking schema is as follows:

1. For each class and for each volume, calculate the Dice score (DSC) and the HD95. Measure also the inference time (Time) and the maximum used memory (Mem);
2. Average the DSC and the HD95 for each class across all volumes and compute the average Time across all volumes.
3. Rank all the DSC, HD95, Time, and maximum used memory independently;
4. Average the rankings obtained at point 3 for each DSC, HD95, and Time to produce the final rank. Average Time is weighted as much as the number of classes included in the dataset to balance its importance with respect to the other metrics;
5. If two or more final ranks obtained at point 4 are equal, Mem will be used to break ties;
6. If two or more ranks are still equal, it is a tie: the prize will be evenly split.

Point 3 will generate 156 rankings (77 classes * 2 metrics + Time ranking + Mem ranking). Point 4 will compute the average of 154 rankings (77 classes * 2 metrics) weighted as 1 and the single Time ranking weighted as 77. Mem ranking will be used only as a tiebreaker (point 6).

If an algorithm does not produce any result for a specific CBCT volume, the evaluation script will consider the predicted segmentation as a volume of 0s, meaning that the Dice score for that image will be equal to 0 for each class, andthe HD95 will be equal to the diagonal of the volume. Furthermore, the inference time will be set at 10 minutes for the case.

The evaluation code is publicly available.

Task 2: IAC Interactive Segmentation¶

The ranking schema of Taks 2 involves the following steps:

1. For each branch of the IAC (left and right) and for each volume, calculate the four metrics: (a) DSC@FinalClick; (b) HD95@FinalClick; (c) DSC-to-Click AUC; and (d) HD95-to-Click AUC, for a fixed number of 5 interaction steps. The result will be 4 metrics * 2 IAC branches, 8 scores in total. Measure also (e) average inference time of the five interaction steps (Time); and (f) maximum used memory (Mem), for all cases;
2. Average the four DSC- and HD95-based metrics for each IAC branch across all volumes.
3. Rank all ten metrics independently;
4. Average the eight rankings obtained at point 3 for each DSC- and HD95-based metric and the rankings obtained for Time to produce the final rank (9 rankings in total);
5. If two or more final ranks obtained at point 4 are equal, use the Mem ranking to break ties;
6. If two or more ranks are still equal, it is a tie: the prize will be evenly split.

If an algorithm does not produce any result for a specific CBCT volume, the evaluation script will consider the predicted segmentation as a volume of 0s, meaning that the Dice score for that image will be equal to 0 for each IAC for all interaction steps, and the HD95 will be equal to the diagonal of the volume. Furthermore, the average inference time will be set to 10 minutes.

The evaluation code is publicly available.

Prizes¶

The first three teams (or single participants) of each Task will be awarded the following prizes:

• 🥇 First place: €1000
• 🥈 Second place: €500
• 🥉 Third place: €300