Physician (MD/DO), Medical AI Evaluation

About the role

Practising physicians review clinical vignettes and judge the quality of reasoning in a medical AI evaluation programme. The work is part time and project based.

What you will do

• Assess clinical vignettes with a question, a proposed answer and a trap answer
• Score items for accuracy, completeness and whether the answer is given away
• Judge answer correctness, gradeability and the quality of the trap
• Check that reasoning reaches the answer from the vignette facts alone
• Correct reasoning traces and write brief rationales
• Take part in three independent blind reads followed by physician adjudication

What they look for

• Board certification or completed residency with recent direct patient care
• Broad diagnostic and treatment experience across body systems
• Strong clinical reasoning including differential diagnosis and next steps
• Written English at C1 level or above
• Based in the US and open to part time, non permanent projects

Nice to have

• Experience in medical annotation or quality assurance