Physician (MD/DO) Medical AI Evaluation

About the role

Practicing physicians review clinical vignettes to test and improve medical AI systems. The work focuses on general and internal medicine reasoning quality.

What you will do

• Assess EHR-based vignettes paired with questions, proposed answers and misleading trap answers
• Rate benchmark items for accuracy, completeness, answer correctness and trap quality
• Verify that reasoning chains reach conclusions from vignette facts alone and correct traces
• Write brief rationales and take part in blind reads followed by physician adjudication

What they look for

• MD or DO degree plus an active unrestricted US medical license
• Board certification or completed residency with recent direct patient care
• Broad diagnostic and treatment experience across multiple systems
• Strong clinical reasoning in differential diagnosis and evidence-based management
• Written English at C1 level or above

Nice to have

• Background in medical annotation or quality assurance