Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices in South Korea

1. Why AI-based medical devices need trial design “extras”

Clinical performance studies for AI/ML-based medical devices can look familiar (prospective vs. retrospective, single-site vs. multi-site, diagnostic accuracy vs. clinical outcome), but they introduce additional failure modes: data leakage, hidden bias, site-to-site variability in inputs, and performance changes when the real-world population differs from the development dataset. The South Korean guidance therefore emphasizes strengthened controls for data quality, reference standard (ground truth) quality, and workflow alignment so that results reflect real use and remain interpretable for regulatory review. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)

The document also uses terminology commonly discussed internationally, such as locked algorithms (the same input yields the same output) when describing how the software is fixed during a clinical evaluation. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS) See also: IMDRF “Machine Learning-enabled Medical Devices: Key Terms and Definitions”

2. Start with a precise claim: device role, output, and intended use

A defensible clinical trial starts with a clear description of:

• The intended purpose / intended use and clinical question (screening, triage, diagnosis, prognosis, monitoring, etc.)
• The role in clinical decision-making (standalone output vs. decision support for clinicians)
• The output type (binary classification, multi-class classification, continuous score, detection, segmentation, prioritization)

These elements determine the most appropriate endpoints, comparator(s), and study design. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)

3. Select a clinical evidence approach: prospective or retrospective

The guidance discusses two broad study approaches.

3.1 Prospective clinical performance study

A prospective study is useful when you need to demonstrate performance in a controlled, forward-looking setting, such as:

• standardizing data capture and clinical workflow,
• ensuring blinding procedures can be implemented, and
• minimizing confounding from changes in clinical practice over time. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)

3.2 Retrospective clinical performance study using existing data

A retrospective study can be suitable when high-quality existing data are available and the evidence question is primarily diagnostic performance rather than downstream clinical outcomes. The guidance still expects strong controls against optimistic bias, including strict separation of training/validation/test data and transparent rules for selecting cases and handling exclusions. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS) Related regulatory context on using real-world data/evidence: FDA “Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices”

4. Data strategy: the foundation of credibility for AI trials

4.1 Inclusion/exclusion criteria and representativeness

Define patient eligibility criteria to match the intended use population, and justify how your dataset reflects the disease spectrum and the real-world mix of cases. The guidance highlights that biased sampling can inflate performance and reduce generalizability. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)

4.2 Independence of datasets and leakage control

To avoid overly optimistic results, ensure the clinical test dataset is independent from any development data. Document and prevent common leakage risks, such as duplicated images, multiple visits from the same patient, or overlapping samples across datasets. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)

4.3 Input quality controls and multi-site variability

AI performance can be sensitive to acquisition conditions. Predefine minimum quality criteria (e.g., device settings, image resolution, specimen handling, operator steps) and, where multiple devices or sites are involved, plan analyses that can detect clinically meaningful performance differences across sites and subgroups. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)

5. Ground truth (reference standard): defining “truth” in a reproducible way

The guidance provides practical options for reference standards (ground truth), including:

• Pathology results (e.g., biopsy) when appropriate
• Established clinical diagnostic criteria
• Imaging findings interpreted by qualified experts
• Expert panel consensus / adjudication to manage reader variability
• Longitudinal clinical follow-up when immediate confirmation is not feasible

The key expectation is to justify that the reference standard is clinically valid for the intended claim and to manage uncertainty (e.g., inter-rater disagreement) transparently. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)

6. Comparator choice, blinding, and workflow realism

6.1 Comparator choice aligned to the claim

Comparators should reflect how the device will be used in practice, such as:

• standard-of-care clinician interpretation (single reader, multiple readers, or adjudicated reading),
• an existing non-AI device or software, or
• alternative clinical pathways (for example, with and without AI assistance).

Because an assistive tool is evaluated differently from a standalone diagnostic, clearly state whether and when clinicians can view the AI output. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)

6.2 Blinding and randomization

Where feasible, blind readers to clinical outcomes and reference standard results to reduce bias. In prospective studies, randomization can be used to balance case mix or mitigate order effects in workflows. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)

7. Endpoints and statistical analysis

7.1 Performance endpoints and success criteria

Select endpoints that map to the device output and clinical use case, such as sensitivity, specificity, AUC, PPV/NPV, agreement metrics, time-to-decision, or downstream clinical outcomes when relevant. Predefine primary vs. secondary endpoints and the success criteria (including non-inferiority or superiority margins when applicable). Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)

7.2 Statistical plan fit for diagnostic-performance evidence

The guidance emphasizes using statistical methods appropriate for diagnostic performance evaluation, including confidence intervals and handling of correlated data (for example, multiple findings per patient or multi-reader multi-case setups). FDA’s statistical guidance on diagnostic tests can be a helpful reference when structuring reporting and analysis plans. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS) See: FDA “Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests”

8. Sponsor checklist (what reviewers expect to see)

• Algorithm version control: lock and document the software/algorithm version used in the trial; describe preprocessing and traceability from input to output.
• Data governance: describe sourcing, curation, de-identification, and audit trails; justify representativeness and manage missing/indeterminate cases.
• Reference standard: justify the ground truth and define adjudication steps for ambiguous cases.
• Human factors in workflow: specify when AI outputs are shown, how clinicians are trained, and what actions users are expected to take.
• Pre-specified analysis: register endpoints, success criteria, and subgroup analyses (site/device variability and key subpopulations). Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)

9. Takeaway

For AI-based medical devices, clinical evidence quality depends heavily on a defensible data strategy and a clinically valid reference standard. The South Korean guidance provides a practical structure to design studies whose results are credible, reproducible, and aligned with the device’s intended role in care. Source: Guidance on Clinical Trials Design of Artificial Intelligence (AI)-based Medical Devices (MFDS)