Physician (MD/DO) Medical AI Evaluation

About the role

Practicing physicians review clinical vignettes to test and improve medical AI systems. The work is project-based and part-time, with tasks scored on clinical reasoning quality.

What you will do

• Assess EHR-based clinical vignettes that pair a question with a proposed answer and a misleading trap answer
• Rate benchmark items for accuracy, completeness, answer correctness, gradeability and trap quality
• Verify that reasoning chains reach the answer using only vignette facts, and correct traces where needed
• Write brief rationales and take part in blind reads followed by physician adjudication

What they look for

• Board certification or completed residency, with recent direct patient care, ideally at attending level
• Broad diagnostic and treatment experience across body systems, such as internal, family, emergency or hospital medicine
• Solid clinical reasoning in differential diagnosis, next-step management and evidence-based guidelines
• Strong written English at C1 level or above