Evaluation

Requirements for a valid file · Task 3

score_h5ad.py applies every check below to a local file before you ever upload it. A file that breaks one of these comes back as a validation error naming what was wrong — not as a low score.

• var_names must be exactly the 500-gene MERFISH panel, in panel order. The check is element by element; a mismatch reports the expected and received counts plus the first missing and extra names. If you have the same genes in a different order, pass --allow-reorder and the scorer reindexes for you.
• .X must be a 2D cells × genes matrix, finite, and non-negative. Sparse and dense both load — the scorer densifies and casts to float32 either way — so float32 is worth writing yourself only to avoid a surprise in your own pipeline.
• Not caught by validation —Values must already be log-normalised. A raw count matrix is non-negative and finite, so it passes every check and is then scored as though it were on the log scale. Nothing will tell you; the score will simply be wrong. Do not submit counts that are normalised but not log-transformed either.
• obs["celltype"] is optional, and nothing you put in it is read. Every cell is typed by the scorer: one classifier is fitted on the held-out ground truth and then run unchanged over every submission, so a cell is typed from its own expression profile and not from the label you attached to it. Unlabelled cells are read as NA and typed the same way.
• At least 1,000 cells. Above that the count is free: no cap, and no correspondence to the target, since every metric is distributional and none assumes predicted cell i matches target cell i. The floor exists because those metrics estimate a statistic from a subsample — below roughly a thousand cells the estimate is dominated by its own sampling noise rather than by the model, so a thin submission is not scored generously, it is scored meaninglessly.
• obsm["spatial_3D"] is required, with shape cells × 3 or more and no NaN or infinity. Only the first three columns are read.
• Coordinates may be in any frame. Every spatial metric is invariant to translation and rotation, so a submission is never asked to register itself to an atlas — with one acknowledged blind spot, laterality: a mirrored embryo scores identically to a correct one.
• Predict the mutant embryo itself, not its difference from wild type. Returning the matched wild type unchanged already reaches 0.956 absolute pseudobulk correlation, so the absolute state carries almost no information about whether a model understood the knockout.
• Knock-outs ship as two biological replicates. You submit one file per target knockout and it is scored against both; the rank uses a fixed one, and the spread between the two is reported so you can see how large a score difference has to be before it means anything.

File contract · Task 3

Format

AnnData .h5ad, .X + obsm["spatial_3D"]

Condition

The held-out mutant, at its stated stage

Genes

500-gene MERFISH panel, in panel order

Coordinates

(n, 3) — any frame

Labels

Never submitted

What the scorer reads

A submission is one AnnData file. This is the layout the scorer opens it expecting for Task 3; anything not listed is free.

AnnData object with n_obs × n_vars = <your cells> × 500
    X       float32, log-normalised, finite, non-negative
    var     index = the 500-gene MERFISH panel, in panel order
    obsm    'spatial_3D'  (n_obs, 3) float32, finite — any frame
    obs     'celltype' optional and ignored

What is deliberately not constrained

• Cell count above the floor. Nothing requires n_pred = n_true — a genuine growth or proliferation model may predict a different number of cells than the target has, and only the 1,000-cell minimum is enforced. Where a count difference would confound a comparison, both sets are subsampled to a shared size first.
• Coordinate frame. Every spatial metric is invariant to translation and rotation, and outside the laterality blind spot to reflection, so no registration to the atlas is expected.
• Cell ordering and identity. No metric assumes predicted cell i corresponds to target cell i.

Score a file locally

The starter kit exposes the same scoring path the task runners use, standalone — no baseline model involved. It loads the task’s real target itself, validates your file against the task's expected gene list and order, and prints the full metric panel as JSON.

python score_h5ad.py --task T3 --target-ko gata4 --input pred.h5ad

• Optimising absolute expression loses to the floor by construction — the floor is the matched wild type, which is already almost right in absolute terms.
• Copying the training knockout’s delta onto the test gene recovers a share of the effect and gets the direction wrong; the panel scores direction explicitly.