AI Agent Evaluation Engineer
- Remote, North America
- Full-time
- Senior
- Posted 4 days ago
or apply from Claude / ChatGPT / Muse
Defining and measuring quality for automated customer service phone agents handling millions of calls; fully remote.
About the role
Someone who does well at the intersection of hands-on conversation analysis, evaluation design, and automated testing systems will excel here.
At a glance
- Pay
- Not stated in the posting
- Remote scope
- Remote, North America
- Seniority
- Senior
- Employment
- Full-time
- Skills
- Python
- Posted
- 12 Nov 2026
- Last checked
- 16 Nov 2026
What you will do
- Design and maintain evaluation rubrics across adherence, outcomes, and conversation experience
- Write annotation guidelines and run calibration sessions to measure inter-rater agreement
- Review production calls weekly to find failures, cluster them, and build automated checks
- Build and operate evaluation pipelines combining automated checks, LLM judges, and human review
What they look for
- Experience designing evaluation systems for conversational agents or language models
- Background in linguistic analysis, quality assurance, or prompt engineering
- Familiarity with LLM-based evaluation methods and golden set creation
- Strong analytical skills for tracing failures to prompts, tools, or speech models
- Ability to work cross-functionally with engineering and product teams
Nice to have
- Experience evaluating multilingual audio conversations and accents
Benefits
- Competitive compensation package with equity options
How Keywordise applies for you
Alex Demo
[email protected]
From your CV
Yes, fully remote
- Sent
- 09:41
- Attached
- Your tailored CV
- 1Upload your CVAdd your CV once. It takes about a minute.
- 2Tailored to this jobKeywordise writes a CV and a cover letter for this job from your real experience.
- 3The form fills itselfName, email, experience and screening questions, field by field.
- 4Application sentYou get a receipt and follow it in your dashboard.
Apply with Keywordise AI7-day free trial, no card.
Apply from your AI assistant
Connect Keywordise to your assistant once, then send it the prompt. It applies for you.
Claude
- In Claude, open Settings, then Connectors, and choose Add custom connector.
- Name it Keywordise and paste the connector URL.
- Click Connect, log in to Keywordise and approve. Then send the prompt in a new chat.
https://keywordise.com/mcpDisconnect any time in your Keywordise settings.
ChatGPT
- In ChatGPT, open Settings, then Connectors. On plans that need it, turn on developer mode first.
- Add a new connector with the connector URL and choose OAuth. Leave the client ID and secret empty.
- Log in to Keywordise and approve. Then send the prompt in a new chat.
https://keywordise.com/mcpDisconnect any time in your Keywordise settings.
Muse
- Open Muse (muse.ai or the app) and send it the message below.
- If Muse asks how to sign in, answer OAuth. Open the sign-in link it sends, log in to Keywordise and approve.
- Send the prompt. Muse asks before it sends anything.
Add a custom connector called Keywordise. It is a remote MCP server at https://keywordise.com/mcp and uses OAuth. Give me the sign-in link.Muse is available in the US only for now.
Apply to Keywordise job 56bnfly357Questions about this job
Is this a remote job, and where can I work from?
Yes, it is listed as remote, aimed at candidates in North America.
What does this job pay?
No pay range is stated in the posting.
What experience is needed?
It is a senior role. The posting asks for: experience designing evaluation systems for conversational agents or language models; background in linguistic analysis, quality assurance, or prompt engineering; familiarity with LLM-based evaluation methods and golden set creation. Skills it names include Python.
How do I apply with Keywordise AI?
Press the Apply with Keywordise AI button and add your CV. Keywordise tailors your CV and cover letter to this posting and sends the application for you, with a free 7-day trial and no card.
More remote engineering jobs
- NNVIDIASenior Libraries Engineer, AI and HPC
- Remote, Europe
- A global professional services firmGlobal Security Operations Engineer
- Remote, Australia
- HHumanaSenior Decision Intelligence Engineer
- Remote, worldwide
- MMotionalSenior Engineer
- USD 168,121 to 214,200 per year
- Remote, United States
- A freelance AI training platformFreelance Mechanical CFD Engineer, AI Trainer
- Remote, Germany
- A technology platform for AI training projectsSenior CFD Engineer (OpenFOAM / ANSYS Fluent), Freelance AI Trainer
- Remote, United States