Physician (MD/DO) Medical AI Evaluation

About the role

Practicing physicians review clinical EHR vignettes to test and improve medical AI systems. The work is project-based and part-time, focused mainly on general and internal medicine.

What you will do

• Assess vignettes that pair a question with a proposed answer and a misleading trap answer
• Rate benchmark items for accuracy, completeness, answer correctness, gradeability and trap quality
• Verify that reasoning traces reach the answer using only the vignette facts, and write brief rationales
• Take part in three separate blind reads followed by physician adjudication

What they look for

• MD or DO degree plus an active unrestricted US medical license
• Board certification or completed residency, with recent direct patient care
• Broad diagnostic and treatment experience across multiple body systems
• Solid clinical reasoning in differential diagnosis, next steps and evidence-based guidelines
• Written English at C1 level or above