Physician (MD/DO) - Medical AI Evaluation

About the role

Practicing physicians review clinical vignettes and reasoning to assess and improve medical AI systems. The work is project-based and part-time, with tasks completed remotely on a per-task basis.

What you will do

• Assess clinical EHR vignettes paired with questions, proposed answers and trap answers
• Score benchmark items for vignette accuracy, completeness and answer correctness
• Judge whether the vignette reveals its answer and how good the trap is
• Verify that reasoning chains reach the answer from vignette facts alone
• Correct reasoning traces and write brief rationales
• Work through three independent blind reads before physician adjudication

What they look for

• An MD or DO with an active, unrestricted US medical license verified by the issuing board
• Board certification or completed residency with recent direct patient care
• Broad multi-system diagnostic and treatment experience in internal, family, emergency or hospital medicine
• Strong clinical reasoning in differential diagnosis, next-step management and evidence-based guidelines
• Written English at C1 level or above

Nice to have

• Experience with medical annotation or quality assurance