Evaluation

T3 — Mutant perturbation prediction. 3D MERFISH (500-gene panel) · predicts expression + 3D coordinates.

Task

Participants will predict a held-out knockout: the expression and 3D position of the cells in a mutant embryo, given wild-type development plus one observed knockout to learn the shape of a perturbation response from.

A conditional knockout line is months of work per gene, so no lab can make one for every gene it wants to understand. The three knockouts here are ordered by difficulty: training and validation use very specific genes, while the hidden test gene is broadly expressed and its effect correspondingly diffuse. Task 3 is the only one whose validation answers are released during the competition — at the start of the final phase, when the rank has moved to the hidden knockout and they can no longer be used to probe the board.

Scoring is on the perturbation effect, not the absolute state. Predicting no change at all — simply returning the matched wild type — already reaches 0.956 absolute pseudobulk correlation, so nothing but the wild-type-to-mutant change carries any information about whether a model understood the knockout.

Task 3 from the competition proposal: 3D MERFISH point clouds of a wild-type embryo and a β-catenin knockout side by side, with dashed boxes marking the held-out knockouts used for validation and for the hidden test. — One knockout is given, to learn the shape of a perturbation response from; the answer is required for a gene the model has never seen.

One knockout is given, to learn the shape of a perturbation response from; the answer is required for a gene the model has never seen.

Splits

Ground truth is withheld while it can still affect the board. Until the final phase a validation submission returns a leaderboard score, not the answers. At the start of the final phase the validation answers are released for every task — ranking has moved to the hidden test split by then, so they can no longer be used to probe it — and the test leaderboard opens. Test ground truth is never distributed.

Train: Mab21l2 KO @ E9.5; A clear phenotype from a very specific gene.
Validation: Gata4 KO @ E8.75; Also specific, at a different stage. Scored through the leaderboard until the final phase, when the answers are released.
Test: β-catenin KO @ E8.75; Broadly expressed, so its effect is diffuse — the hardest of the three. Hidden until the competition closes.
Reference: Matched WT @ E8.75 and E9.5; Every DE metric is computed against the matched wild type, not against a preceding stage.

A mutant embryo still looks mostly like a wild-type one: hand back the matched wild type unchanged and absolute expression already correlates at 0.956. So the score is computed on the difference between wild type and mutant, and a submission earns nothing for reproducing the wild type it was given — only for moving the genes the knockout actually moves. Full task page →