Physician (MD/DO) Medical AI Evaluation

About the role

Practicing physicians review clinical vignettes and reasoning traces to judge the quality of a medical AI evaluation program, working on a part-time project basis.

What you will do

• Assess clinical vignettes paired with questions, proposed answers and trap answers across diagnosis and treatment
• Score benchmark items for vignette accuracy, completeness, answer correctness and trap quality
• Check whether reasoning chains reach the answer from vignette facts alone and correct the traces
• Write short rationales and take part in blind reads followed by physician adjudication

What they look for

• Medical degree (MD or DO) plus an active unrestricted US license verified with the issuing board
• Board certification or completed residency with recent direct patient care, attending level preferred
• Broad multi-system diagnostic and treatment experience in internal, family, emergency or hospital medicine
• Strong clinical reasoning in differential diagnosis, next-step management and evidence-based guidelines
• Written English at C1 level or above

Nice to have

• Experience with medical annotation or quality assurance