Evaluation

T1 — Temporal gene-expression distribution prediction. single-cell RNA · predicts expression.

Task

Participants will predict the gene-expression distribution of a whole heart at a developmental stage the model has never seen, given the stages before it. The prediction is a population of cells — progenitors, differentiating cells and terminal types in their real proportions — not an average cell. A model that collapses onto the mean profile forfeits most of the score.

Staging an embryo is destructive and each time point is a different animal, so a stage that was not collected cannot be recovered by re-running the experiment; and the gap between collected stages is often where the interesting transitions happen. Here nothing sits between the validation and test stages at all, so interpolation has nothing to lean on.

To keep the task tractable we release two fully observed earlier stages of real dissociated single cells — RNA across the whole transcriptome, not the 500-gene MERFISH panel Tasks 2 and 3 use. Participants may additionally train on external public single-cell data; only the evaluation stages are fixed, and any external source must be disclosed with the submission.

Task 1 from the competition proposal: single-cell UMAPs at E8.5 and E9.5 as training, each coloured and labelled by cell type, above dashed boxes marking the validation target E10.5 and the hidden test target E12.5. — Three different embryos, not one embryo followed over time — profiling destroys the embryo, so there is no cell-level correspondence between stages. What is predicted is the distribution at two stages that are never shown: E10.5 for the leaderboard and E12.5 for the final ranking.

Three different embryos, not one embryo followed over time — profiling destroys the embryo, so there is no cell-level correspondence between stages. What is predicted is the distribution at two stages that are never shown: E10.5 for the leaderboard and E12.5 for the final ranking.

Splits

Ground truth is withheld while it can still affect the board. Until the final phase a validation submission returns a leaderboard score, not the answers. At the start of the final phase the validation answers are released for every task — ranking has moved to the hidden test split by then, so they can no longer be used to probe it — and the test leaderboard opens. Test ground truth is never distributed.

Train: E8.5 · E9.5; The two real single-cell stages before the target. There is no E9.25 in this release.
Validation: E10.5; Scored through the leaderboard until the final phase, when the answers are released.
Test: E12.5; Hidden. Scored only in the final phase.
Outside the split: E7.75; The one whole-embryo stage in the release. Every stage in the split is heart: through gastrulation the embryo is small enough to profile entire, and by E8.5 it is not, so collection narrows to the organ. It ships and can be used as background, but it is not the same object as the stages it would be compared against.

Nothing is observed between E10.5 and E12.5, so there is no bracketing stage to interpolate from. And the target is mostly not the training population moved forward: E8.5 and E9.5 share only 5 cell types, so about two thirds of the cells at the later stage are types that were not there before. They did not drift into place, they differentiated — which makes this a question about which cells appear, not about how far existing ones travel. Full task page →