NakedSignal OS

Submission check

Validate a predictions file against the submission contract before you send it. Nothing here is scored.

How submission check works

Your browser read the file, split it on commas, and ran seven checks over it: the declaration block, the declared sealed set, the columns, the row count per partition, the case-id format, the value range and the row sums. The columns it expected were generated from the sealed set’s own published class list, and the row counts from its published partition sizes, so this page cannot ask you for a shape the set does not have.

The one check that is real against a published value is the set hash: the string you declare is compared, character for character, with the set_hash in the seal manifest for that generation. That is what catches a run against a set that has since been retired and published in full — a perfect file and a worthless result.

What this page cannot do, and does not pretend to: check that your case ids are the right ones, and score anything at all. Case identifiers are salted HMACs, published nowhere; the seals publish the per-case hashes, and a case hash cannot be recomputed from an id without the operator-only content hash. So membership is checked in the harness, not here. And scoring needs labels, which are sealed — which is the entire point of sealing them.

Specifications

The submission contract — columns, declaration block, sealed sets

One CSV. A declaration block, then one row per case for every partition of the sealed set you were served. The columns are generated from that set’s own published class list, so they cannot drift from the set you are predicting on.

Declaration — `# key: value` lines, above the header
  • # generation: …
  • # set_hash: …
  • # code_fingerprint: …
  • # seed: …
  • # device: …

The seed and the code fingerprint are not paperwork: a result nobody can re-run is not a result. `generation` and `set_hash` say which sealed set you were served, and that hash is the one thing on this page checked against a value we have published.

Columns — 10, in this order
  • case_id32 lowercase hex, unique
  • partitionranked | anchor
  • p_basophilfloat in [0, 1], 6 dp
  • p_eosinophilfloat in [0, 1], 6 dp
  • p_erythroblastfloat in [0, 1], 6 dp
  • p_immature_granulocytefloat in [0, 1], 6 dp
  • p_lymphocytefloat in [0, 1], 6 dp
  • p_monocytefloat in [0, 1], 6 dp
  • p_neutrophilfloat in [0, 1], 6 dp
  • p_plateletfloat in [0, 1], 6 dp

Generated from gen2’s published class list, in its published order. Every row’s probabilities must sum to 1 within 0.000001.

Sealed sets we have published a seal for
generationset_hashcasespartitionsstatus
gen1641ab45baeaf…1,400ranked 1,000 · anchor 400retired 2026-07-22, published in full
gen2ad30d214daa2…1,201ranked 801 · anchor 400accepting submissions until 2026-09-20
public00fb7b7e7345…1,000arm 1,000control arm, published in full by design

Read from plan_d/results/heldout/manifests/*.seal.json at build time. A seal publishes the set’s hash, its size and its class list. It does not publish the case identifiers, and that is why your browser cannot tell you whether your ids are the right ones.

What the specification says, quoted
§5.1 The adapter contract
A model enters the standard through an adapter exposing exactly:

```
fit(x_train, y_train, x_val, y_val, n_classes)   ->  fitted
predict_proba(x)                                 ->  (n, k) row-stochastic
describe()                                       ->  identity and configuration
```

The scorer knows nothing else about the model. This is what allows a competitor's model, a partner's model and the operator's own model to be scored on identical terms, with no adapter holding privileged access to anything.
§5.2 Submission integrity
For private evaluation, the model is executed against sealed data by the operator, or in a sealed environment with logged I/O. The submitter never receives the test cases. Model weights need not be disclosed; the interface must be.
§6.1 The result record
One record per (task, model), carrying: task and model identity, spec version, code fingerprint, seed, device, split sizes, class count, training cap status, split origin, runtime, per-dimension raw measurements and scores, the composite index with its coverage, and any error.

Verbatim from plan_d/docs/MEDEVAL-1-SPEC.md. The whole document is on Specifications.