Physician (MD/DO) Medical AI Evaluation

About the role

Practising physicians review clinical EHR vignettes and benchmark items to assess the quality of medical AI reasoning. Work is project-based and part-time, with tasks scored for accuracy and gradeability.

What you will do

• Evaluate clinical vignettes paired with questions, proposed answers and trap answers across diagnosis and treatment
• Score clinical reasoning quality, including vignette accuracy, completeness and whether the answer is given away
• Check that reasoning chains reach the answer from vignette facts alone and correct reasoning traces
• Write short rationales for assessments

What they look for

• Board certification or completed residency with recent direct patient care, attending level preferred
• Broad multi-system diagnostic and treatment experience in internal, family, emergency or hospital medicine
• Strong clinical reasoning in differential diagnosis, next-step management and evidence-based guidelines
• Strong written English at C1 or above