Physician (MD/DO) Medical AI Evaluation

About the role

Practicing physicians review clinical vignettes and reasoning to test and improve medical AI systems. Work is project-based and part-time, with independent blind reads followed by physician adjudication.

What you will do

• Assess EHR-based clinical vignettes that pair a question with a proposed answer and a misleading trap answer
• Rate benchmark items for vignette accuracy, completeness, answer correctness and trap quality
• Verify that reasoning chains reach conclusions from vignette facts alone and correct faulty traces
• Write brief rationales and take part in blind reads followed by physician adjudication
• Cover diagnosis and treatment tasks across many body systems, from cardiovascular to musculoskeletal

What they look for

• Board certification or completed residency, with recent direct patient care, attending level preferred
• Broad multi-system diagnostic and treatment experience in internal, family, emergency or hospital medicine
• Strong clinical reasoning in differential diagnosis, next-step management and evidence-based guidelines
• Written English at C1 level or above