Physician (MD/DO) - Medical AI Evaluation

About the role

Fits licensed doctors with broad diagnostic experience who want to judge AI output against real clinical standards. Each item gets three blind reads before a physician adjudicates.

At a glance

What you will do

• Review EHR-style vignettes with a question, a proposed answer and a trap answer
• Score reasoning quality, including vignette accuracy, completeness and whether the answer is given away
• Judge answer correctness and whether the item can be graded, plus the quality of the trap
• Check that the reasoning chain reaches the answer from the facts alone, and write short rationales

What they look for

• MD or DO with an active, unrestricted US licence, checked with the issuing board
• Board certification or completed residency, with recent direct patient care, attending level preferred
• Broad multi-system experience in internal, family, emergency or hospital medicine
• Solid clinical reasoning across differential diagnosis, next steps and evidence-based guidelines
• Written English at C1 or above

A typical week

• Read assigned vignettes and score them independently before discussion
• Write brief rationales for each judgement
• Keep to submission deadlines during active project phases, roughly 10 to 20 hours a week