Evaluation

Requirements for a valid file · Task 1

score_h5ad.py applies every check below to a local file before you ever upload it. A file that breaks one of these comes back as a validation error naming what was wrong — not as a low score.

• var_names must be exactly the whole transcriptome — 32,285 genes — in the released order. The check is element by element; a mismatch reports the expected and received counts plus the first missing and extra names. If you have the same genes in a different order, pass --allow-reorder and the scorer reindexes for you.
• .X must be a 2D cells × genes matrix, finite, and non-negative. Sparse and dense both load — the scorer densifies and casts to float32 either way — so float32 is worth writing yourself only to avoid a surprise in your own pipeline.
• Not caught by validation —Values must already be log-normalised. A raw count matrix is non-negative and finite, so it passes every check and is then scored as though it were on the log scale. Nothing will tell you; the score will simply be wrong. Do not submit counts that are normalised but not log-transformed either.
• obs["celltype"] is optional, and nothing you put in it is read. Every cell is typed by the scorer: one classifier is fitted on the held-out ground truth and then run unchanged over every submission, so a cell is typed from its own expression profile and not from the label you attached to it. Unlabelled cells are read as NA and typed the same way.
• At least 1,000 cells. Above that the count is free: no cap, and no correspondence to the target, since every metric is distributional and none assumes predicted cell i matches target cell i. The floor exists because those metrics estimate a statistic from a subsample — below roughly a thousand cells the estimate is dominated by its own sampling noise rather than by the model, so a thin submission is not scored generously, it is scored meaninglessly.
• No coordinates. Task 1 uses scRNA-seq, so the cells are dissociated and no position is available for any of them — obsm is not read at all.

File contract · Task 1

Format

AnnData .h5ad, .X = [n_cells, n_genes]

Genes

Whole transcriptome — 32,285 genes, in the released order

Cells

Need not match the target count

Coordinates

Not applicable — single-cell modality

Labels

Never submitted

What the scorer reads

A submission is one AnnData file. This is the layout the scorer opens it expecting for Task 1; anything not listed is free.

AnnData object with n_obs × n_vars = <your cells> × 32285
    X       float32, log-normalised, finite, non-negative
    var     index = the transcriptome gene list, in the released order
    obs     'celltype' optional and ignored

What is deliberately not constrained

• Cell count above the floor. Nothing requires n_pred = n_true — a genuine growth or proliferation model may predict a different number of cells than the target has, and only the 1,000-cell minimum is enforced. Where a count difference would confound a comparison, both sets are subsampled to a shared size first.
• Cell ordering and identity. No metric assumes predicted cell i corresponds to target cell i.

Score a file locally

The starter kit exposes the same scoring path the task runners use, standalone — no baseline model involved. It loads the task’s real target itself, validates your file against the task's expected gene list and order, and prints the full metric panel as JSON.

python score_h5ad.py --task T1 --input pred.h5ad

• Task 1 is whole-transcriptome. Assuming the 500-gene MERFISH panel is the most common schema error here.
• Predicting one average cell over and over still scores on the two gene-level metrics, which only compare means. It scores near nothing on the two that compare populations — cell-state distribution and gene-gene co-variation — and those are half the task. Submit a spread of cells, not a consensus cell.