Physician (MD/DO) Medical AI Evaluation

About the role

Practicing physicians review clinical vignettes to test and improve medical AI systems. The work is project-based and part-time, with payment for each accepted task.

What you will do

• Assess EHR-based clinical scenarios that include a question, a suggested answer and a misleading option
• Rate the quality of clinical reasoning in benchmark items, including accuracy and whether answers are given away
• Verify that reasoning chains reach conclusions using only the facts in the vignette and correct faulty traces
• Write brief justifications and take part in independent blind reviews followed by physician adjudication
• Cover diagnosis and treatment tasks across many body systems, with general and internal medicine as the focus

What they look for

• Board certification or completed residency, with recent hands-on patient care, ideally at attending level
• Broad diagnostic and treatment experience across systems, such as internal, family, emergency or hospital medicine
• Solid clinical reasoning skills in differential diagnosis, next steps and evidence-based guidelines
• Written English at C1 level or above