The sequestered test set stays closed until the model version is frozen and the protocol and statistical analysis plan are signed. The team then runs the locked model on the test cases, collects the reference standard from independent experts, and runs the reader study if the claim needs one. The statistician executes the pre-specified analysis, primary and secondary endpoints with confidence intervals, and the planned subgroup breakdowns.
On the map: Unblind after the model is locked. Run the reader study. Statistical analysis with subgroups.
Rows are disciplines, columns are stages 1 to 8. Select a square to open that task.
Back to the full map
Unblind only after engineering confirms the model is locked on the verified build. Agree subgroup analyses with regulatory in advance, so results don't invite questions about after-the-fact slicing.
Only after the model weights and software version are fixed and recorded, and the protocol and analysis plan are approved. Access controls keep the developers away from the test data until then. Any model change after unblinding is a protocol deviation that needs a written justification or a fresh test set, because FDA expects no post-hoc tuning on test data.
The claim decides it. A measurement or quantification output is compared against expert reference values in a standalone study. A device that changes how readers interpret images needs an MRMC study with a washout between sessions. Many programs run both and keep standalone accrual going during the washout so extra cases are on hand if FDA asks.
For continuous outputs the readers' values are usually averaged, and inter-reader agreement (ICC or a reader-to-reader Bland-Altman) is reported to justify the average. For segmentations the device is compared with each reader separately. The rule is written into the protocol before unblinding.
The list comes from the plan: sex, age, race, ethnicity, site, scanner maker, protocol, and known confounders. Subgroups are powered only when a subgroup claim is made. Otherwise the team reports counts, estimates and intervals and looks for any subgroup whose estimate falls clearly outside the overall result.
See the open positions and apply.