Researcher, AI Evaluations
- San Francisco, CA
- Full-time
- Senior
- Posted 7 days ago
or apply from Claude / ChatGPT / Muse
Senior research role on an AI insights team, measuring model and agent quality through hands-on studies and rubrics; remote in the US or from a US hub.
About the role
An individual contributor post with no reports, sitting between AI research and product. Findings are meant to shape model choices, release calls and the quality bar users actually feel.
At a glance
- Pay
- Not stated in the posting
- Location
- San Francisco, CA
- Seniority
- Senior
- Employment
- Full-time
- Skills
- LLM-as-judge, Braintrust, LangSmith, DeepEval
- Posted
- 9 Nov 2026
- Last checked
- 16 Nov 2026
What you will do
- Own evaluation research for the highest-priority AI research workstreams, from quality criteria and study design to analysis and recommendat
- Create rubrics, representative evaluation datasets, benchmarks and quality thresholds grounded in user needs and design judgement
- Work with AI research and modelling platform engineers to fold validated methods into repeatable evaluations and regression tests
- Explain quality, failure patterns and uncertainty in ways that inform model selection, product priorities and release decisions
- Link user research to model development, sharing reusable methods, reviewing colleagues' work and mentoring peers
What they look for
- Seven or more years in research or applied research, with a record of leading complex evaluation projects alone and showing measurable impac
- Hands-on experience evaluating AI products and turning findings into real model and product improvements
- Solid qualitative and quantitative research ability, comfortable interpreting uncertainty and drawing practical conclusions about model beha
- Strong problem solving and stakeholder management, able to handle ambiguity, weigh risks and balance speed against rigour
How Keywordise applies for you
Alex Demo
[email protected]
From your CV
Yes, fully remote
- Sent
- 09:41
- Attached
- Your tailored CV
- 1Upload your CVAdd your CV once. It takes about a minute.
- 2Tailored to this jobKeywordise writes a CV and a cover letter for this job from your real experience.
- 3The form fills itselfName, email, experience and screening questions, field by field.
- 4Application sentYou get a receipt and follow it in your dashboard.
Apply with Keywordise AI7-day free trial, no card.
Apply from your AI assistant
Connect Keywordise to your assistant once, then send it the prompt. It applies for you.
Claude
- In Claude, open Settings, then Connectors, and choose Add custom connector.
- Name it Keywordise and paste the connector URL.
- Click Connect, log in to Keywordise and approve. Then send the prompt in a new chat.
https://keywordise.com/mcpDisconnect any time in your Keywordise settings.
ChatGPT
- In ChatGPT, open Settings, then Connectors. On plans that need it, turn on developer mode first.
- Add a new connector with the connector URL and choose OAuth. Leave the client ID and secret empty.
- Log in to Keywordise and approve. Then send the prompt in a new chat.
https://keywordise.com/mcpDisconnect any time in your Keywordise settings.
Muse
- Open Muse (muse.ai or the app) and send it the message below.
- If Muse asks how to sign in, answer OAuth. Open the sign-in link it sends, log in to Keywordise and approve.
- Send the prompt. Muse asks before it sends anything.
Add a custom connector called Keywordise. It is a remote MCP server at https://keywordise.com/mcp and uses OAuth. Give me the sign-in link.Muse is available in the US only for now.
Apply to Keywordise job vwrz6xjcnwQuestions about this job
Where is this job based?
The posting lists Researcher, AI Evaluations as a role based in San Francisco, CA, United States.
What does this job pay?
No pay range is stated in the posting.
What experience is needed?
It is a senior role. The posting asks for: seven or more years in research or applied research, with a record of leading complex evaluation projects alone and showing measurable impac; hands-on experience evaluating AI products and turning findings into real model and product improvements; solid qualitative and quantitative research ability, comfortable interpreting uncertainty and drawing practical conclusions about model beha. Skills it names include LLM-as-judge, Braintrust, LangSmith, DeepEval.
How do I apply with Keywordise AI?
Click Apply with Keywordise AI, create an account and upload your CV. Your CV and cover letter are written for this job from what you have really done, then Keywordise applies for you. Trial: 7 days, no card.
More science and research jobs in San Francisco, CA
- BBridgeBioSenior Scientist / Associate Director, Drug Product Manufacturing
- San Francisco, CA
- A large AI research organizationResearch Scientist, Discovery (Engineering & Applied Sciences)
- San Francisco, CA
- An AI safety research organizationResearch Scientist, Life Sciences
- San Francisco, CA
FigmaResearcher, Design- San Francisco, CA
Mochi HealthMedical Writer and Producer- San Francisco, CA
- GSGolden StateDirector, Global Partnership Development
- San Francisco, CA