While engineering builds the device, the clinical team collects cases from the sites, checks them against the inclusion and exclusion criteria, and tracks enrollment against the subgroup targets. Annotators and expert readers produce the reference standard following the protocol, and disagreements go through the planned adjudication. The test set is held back from the start, with access limited and logged, so the developers never see it.
On the map: Collect data at sites. Expert annotation and adjudication. Hold back the sequestered test set.
Rows are disciplines, columns are stages 1 to 8. Select a square to open that task.
Back to the full map
Keep the test set sequestered behind engineering's access controls, and have quality qualify annotators and sites before labeling starts. Ground truth set by unqualified readers is hard to defend.
Development data can be labeled by one trained expert or a cheaper pre-annotation step with expert review. The test reference comes from independent qualified readers, usually US board-certified, who did not label the training data. Readers get a written instruction, a training session and practice cases before they start.
Split by site and patient before any labeling, so no site or patient appears on both sides. Store the test set where only the regulatory assessment team can reach it, run evaluations through a script someone else executes, and log every access with date, reason and result.
When the total count and every pre-set subgroup quota are both met. Keep collecting through the build if possible, so a reserve tranche exists if a subgroup comes up short or FDA asks for more data.
See the open positions and apply.