CrestaHiring

AI Agent Evaluation Engineer

  • Remote, North America
  • Full-time
  • Senior
  • Posted 4 days ago
In one line

Defining and measuring quality for automated customer service phone agents handling millions of calls; fully remote.

About the role

Someone who does well at the intersection of hands-on conversation analysis, evaluation design, and automated testing systems will excel here.

At a glance

Pay
Not stated in the posting
Remote scope
Remote, North America
Seniority
Senior
Employment
Full-time
Skills
Python
Posted
12 Nov 2026
Last checked
16 Nov 2026

What you will do

  • Design and maintain evaluation rubrics across adherence, outcomes, and conversation experience
  • Write annotation guidelines and run calibration sessions to measure inter-rater agreement
  • Review production calls weekly to find failures, cluster them, and build automated checks
  • Build and operate evaluation pipelines combining automated checks, LLM judges, and human review

What they look for

  • Experience designing evaluation systems for conversational agents or language models
  • Background in linguistic analysis, quality assurance, or prompt engineering
  • Familiarity with LLM-based evaluation methods and golden set creation
  • Strong analytical skills for tracing failures to prompts, tools, or speech models
  • Ability to work cross-functionally with engineering and product teams

Nice to have

  • Experience evaluating multilingual audio conversations and accents

Benefits

  • Competitive compensation package with equity options

How Keywordise applies for you

  1. 1Upload your CVAdd your CV once. It takes about a minute.
  2. 2Tailored to this jobKeywordise writes a CV and a cover letter for this job from your real experience.
  3. 3The form fills itselfName, email, experience and screening questions, field by field.
  4. 4Application sentYou get a receipt and follow it in your dashboard.

Apply with Keywordise AI7-day free trial, no card.

Apply from your AI assistant

Connect Keywordise to your assistant once, then send it the prompt. It applies for you.

Claude

  1. In Claude, open Settings, then Connectors, and choose Add custom connector.
  2. Name it Keywordise and paste the connector URL.
  3. Click Connect, log in to Keywordise and approve. Then send the prompt in a new chat.
Connector URL
https://keywordise.com/mcp

Disconnect any time in your Keywordise settings.

ChatGPT

  1. In ChatGPT, open Settings, then Connectors. On plans that need it, turn on developer mode first.
  2. Add a new connector with the connector URL and choose OAuth. Leave the client ID and secret empty.
  3. Log in to Keywordise and approve. Then send the prompt in a new chat.
Connector URL
https://keywordise.com/mcp

Disconnect any time in your Keywordise settings.

Muse

  1. Open Muse (muse.ai or the app) and send it the message below.
  2. If Muse asks how to sign in, answer OAuth. Open the sign-in link it sends, log in to Keywordise and approve.
  3. Send the prompt. Muse asks before it sends anything.
Message for Muse
Add a custom connector called Keywordise. It is a remote MCP server at https://keywordise.com/mcp and uses OAuth. Give me the sign-in link.

Muse is available in the US only for now.

Prompt
Apply to Keywordise job 56bnfly357
How to connect

Questions about this job

Is this a remote job, and where can I work from?

Yes, it is listed as remote, aimed at candidates in North America.

What does this job pay?

No pay range is stated in the posting.

What experience is needed?

It is a senior role. The posting asks for: experience designing evaluation systems for conversational agents or language models; background in linguistic analysis, quality assurance, or prompt engineering; familiarity with LLM-based evaluation methods and golden set creation. Skills it names include Python.

How do I apply with Keywordise AI?

Press the Apply with Keywordise AI button and add your CV. Keywordise tailors your CV and cover letter to this posting and sends the application for you, with a free 7-day trial and no card.

More remote engineering jobs

All remote engineering jobs
CrestaRemote, North America
Apply with Keywordise AI