Evaluation

Requirements for a valid file · Task 2

score_h5ad.py applies every check below to a local file before you ever upload it. A file that breaks one of these comes back as a validation error naming what was wrong — not as a low score.

• var_names must be exactly the 500-gene MERFISH panel, in panel order. The check is element by element; a mismatch reports the expected and received counts plus the first missing and extra names. If you have the same genes in a different order, pass --allow-reorder and the scorer reindexes for you.
• .X must be a 2D cells × genes matrix, finite, and non-negative. Sparse and dense both load — the scorer densifies and casts to float32 either way — so float32 is worth writing yourself only to avoid a surprise in your own pipeline.
• Not caught by validation —Values must already be log-normalised. A raw count matrix is non-negative and finite, so it passes every check and is then scored as though it were on the log scale. Nothing will tell you; the score will simply be wrong. Do not submit counts that are normalised but not log-transformed either.
• obs["celltype"] is optional, and nothing you put in it is read. Every cell is typed by the scorer: one classifier is fitted on the held-out ground truth and then run unchanged over every submission, so a cell is typed from its own expression profile and not from the label you attached to it. Unlabelled cells are read as NA and typed the same way.
• At least 1,000 cells. Above that the count is free: no cap, and no correspondence to the target, since every metric is distributional and none assumes predicted cell i matches target cell i. The floor exists because those metrics estimate a statistic from a subsample — below roughly a thousand cells the estimate is dominated by its own sampling noise rather than by the model, so a thin submission is not scored generously, it is scored meaninglessly.
• obsm["spatial_3D"] is required, with shape cells × 3 or more and no NaN or infinity. Only the first three columns are read.
• Coordinates may be in any frame. Every spatial metric is invariant to translation and rotation, so a submission is never asked to register itself to an atlas — with one acknowledged blind spot, laterality: a mirrored embryo scores identically to a correct one.
• One file per setting. The heart and embryo settings are scored separately — score each with --setting heart or --setting embryo, and submit them as separate entries.

File contract · Task 2

Format

AnnData .h5ad, .X + obsm["spatial_3D"]

Genes

500-gene MERFISH panel, in panel order

Coordinates

(n, 3) — any frame

Cells

Need not match the target count

Setting

heart or embryo — scored separately

What the scorer reads

A submission is one AnnData file. This is the layout the scorer opens it expecting for Task 2; anything not listed is free.

AnnData object with n_obs × n_vars = <your cells> × 500
    X       float32, log-normalised, finite, non-negative
    var     index = the 500-gene MERFISH panel, in panel order
    obsm    'spatial_3D'  (n_obs, 3) float32, finite — any frame
    obs     'celltype' optional and ignored

What is deliberately not constrained

• Cell count above the floor. Nothing requires n_pred = n_true — a genuine growth or proliferation model may predict a different number of cells than the target has, and only the 1,000-cell minimum is enforced. Where a count difference would confound a comparison, both sets are subsampled to a shared size first.
• Coordinate frame. Every spatial metric is invariant to translation and rotation, and outside the laterality blind spot to reflection, so no registration to the atlas is expected.
• Cell ordering and identity. No metric assumes predicted cell i corresponds to target cell i.

Score a file locally

The starter kit exposes the same scoring path the task runners use, standalone — no baseline model involved. It loads the task’s real target itself, validates your file against the task's expected gene list and order, and prints the full metric panel as JSON.

python score_h5ad.py --task T2 --setting heart --input pred.h5ad

• Coordinates are per-embryo local and not registered across time points, so no shared frame can be assumed between input stages — and none is required of the output.
• Regressing coordinates against an MSE target optimises absolute position in a frame the metrics deliberately ignore, while leaving the relational structure they do score unconstrained.