Physician (MD/DO), Medical AI Evaluation

About the role

Practicing physicians review clinical EHR vignettes to test and improve medical AI systems. The work is project-based and part-time, with a focus on general and internal medicine.

What you will do

• Assess clinical vignettes paired with questions, proposed answers and trap answers
• Score benchmark items for accuracy, completeness, answer correctness and trap quality
• Verify that reasoning chains reach the answer from vignette facts alone and correct reasoning traces
• Write brief rationales for the assessments
• Take part in three independent blind reads followed by physician adjudication

What they look for

• Board certification or completed residency, with recent direct patient care
• Broad diagnostic and treatment experience across body systems
• Strong clinical reasoning in differential diagnosis, next steps and evidence-based guidelines
• Written English at C1 level or above