Contest organization¶

Last year we organized a similar challenge on local reconstruction methods for diffusion MRI in the context of the IEEE International Symposium on Biomedical Imaging (ISBI 2012) conference. More information can be found at the hardi.epfl.ch website, which contains all the details about the organization of the event, the results of the competition and the full program of the workshop.

The event was very well received by the community: 13 teams registered and 25 different submissions were evaluated, 12 talks were given and 30-35 attendees were present on-site at the event. On that occasion, we collected many encouraging feedbacks, together with suggestions and comments to improve the challenge.

Taking to heart this past experience, some major improvements have been introduced in this year’s edition.

More realistic simulation of the diffusion signal. The diffusion MRI signal corresponding to a given intra-voxel configuration will be made more realistic by employing advanced models accounting for both intra- and extra- cellular water (see details).
Separate sampling classes. Three different classes have been created to better compare the algorithms based on the characteristics of the sampling scheme adopted by each participant (see details).

New evaluation system. This year’s challenge focuses on the effects of the local accuracy in the estimation of the intra-voxel fiber configuration on the quality of subsequent global connectivity analyses (see details).

Sampling classes¶

The submissions will be subdivided into three different classes, depending on the characteristics of the sampling scheme adopted by each participant:

DTI-like	6-32 samples	$b \leq 1200 \; \textrm{s}/\textrm{mm}^2$
HARDI-like	33-64 samples	$1500 \leq b \leq 4000 \; \textrm{s}/\textrm{mm}^2$
Heavyweight	65-515 samples	$b \leq 12000 \; \textrm{s}/\textrm{mm}^2$

A separate ranking will be drawn for each sampling class, therefore identifying three winners in total.

Evaluation criteria¶

Two main criteria will be considered to assess the performances of all the proposed approaches: the local accuracy in the estimation of the fiber configurations and the global impact of this accuracy on a subsequent connectivity analysis performed with a standard fiber-tracking algorithm.

At the voxel level, a score will be attributed accounting for the following metrics:
- Correct estimation of the number of fiber compartments, expressed by means of the probability of false fibre detection:
  
  $P_d = \frac{ | M_\mathrm{true} - M_\mathrm{estimated} | }{ M_\mathrm{true}}$
  
  where $M_\mathrm{true}$ and $M_\mathrm{estimated}$ are, respectively, the real and estimated number of fiber compartments inside the voxel.
- Angular precision of the estimated fiber compartments, assessed by means of the angular error (in degrees) between the estimated fiber directions and the true ones inside the voxel:
  
  $\epsilon_\theta = \arccos( | \vec{d}_\mathrm{true} \cdot \vec{d}_\mathrm{estimated}) | \, \frac{180}{\pi}$
  
  where $\vec{d}_\mathrm{true}$ is a true direction inside the voxel and $\vec{d}_\mathrm{estimated}$ is its closest estimate. The final value will be the average of the angular errors computed for all the true and estimated fiber compartments.
At the macroscopic level, we are interested to investigate the impact of the local reconstruction accuracy on the quality of subsequent global connectivity analyses performed on top of each local reconstruction method. A standard method commonly used in the literature to perform fiber-tracking will be chosen and the evaluation will be done following the Tractometer approach of [Cote2012].

Important

For determining the final ranking only the global measures will be used. The evaluation of the local accuracy are used only for descriptive purposes.

Training data and Testing data¶

Training data will be released prior of the contest itself in order for the contestants to practice with the data format and test their methods on it. The code to generate the training data set will not be released. Alternatively, we will provide on the contest website the signal simulated for several predetermined sampling schemes. You can also request (up to five times per team) to generate the signal corresponding to the training data for different sampling schemes. Please see the Training Data section to download the training data.

To evaluate each method and to draw the final ranking of the contest, however, another testing dataset will be used, but this time the ground-truth will be unknown to the contestants. The data format of this testing dataset is the same as for the training dataset; only the configuration of the fiber compartments in each voxel will change. Please see the Testing Data section to download/request the testing data.

Call for papers¶

Each team participating to the contest will be requested to write a 1-page abstract describing the approach undertaken. All papers will be gathered to form the workshop proceedings. These proceedings will be published on this website and on the official IEEE International Symposium on Biomedical Imaging (ISBI 2013) website.

Please refer to this template for typesetting your abstract. Since the results will be unknown at the time of submission of the abstract, only a technical description of the method is indeed requested.

How to pre-register¶

To express your interest in participating in this competition, just send an email to one of the organizers with the following details:

contact person and e-mail used for correspondence (NB: this e-mail will never appear on this website);
location and institution;
name of the team;
brief description of the proposed approach, if available.

The challenge is open to everyone.

References

[Cote2012]

Côté et al. Tractometer: Towards Validation of Tractography Pipelines, Medical Image Analysis (2013).