Physician (MD/DO) Medical AI Evaluation

About the role

Practising physicians review clinical vignettes and score the reasoning quality of medical AI benchmark items. The work helps improve AI tools used in clinical settings.

What you will do

• Evaluate clinical vignettes paired with questions, proposed answers and trap answers across diagnosis and treatment
• Score vignette accuracy, completeness, answer correctness and trap quality
• Check whether reasoning chains reach the answer from vignette facts alone and correct reasoning traces
• Write short rationales for assessments

What they look for

• Board certification or completed residency with recent direct patient care
• Broad multi system diagnostic and treatment experience in internal, family, emergency or hospital medicine
• Strong clinical reasoning including differential diagnosis and evidence based guidelines
• Strong written English at C1 level or above