AI Rubric Reviewer
- USD 40 to 75 per hour
- Remote, United States
- Contract
- Mid-level
- Posted 2 days ago
or apply from Claude / ChatGPT / Muse
Fully remote contract role reviewing AI-drafted tasks and model attempts to ensure fairness and accuracy in evaluation.
AI Rubric Reviewer is a mid-level contract remote role in the United States at Mercor. Pay: USD 40 to 75 per hour. Posted 17 Nov 2026. Key requirements: ability to read long agent transcripts and identify specific errors; precise writing skills and a commitment to verifying data.
About the role
The work focuses on assessing grading rubrics for LLM evaluations and analyzing long agent transcripts to identify errors and improve training data quality.
At a glance
- Pay
- USD 40 to 75 per hour
- Remote scope
- Remote, United States
- Seniority
- Mid-level
- Employment
- Contract
- Posted
- 17 Nov 2026
- Last checked
- 19 Nov 2026
What you will do
- Assess if grading rubrics align with prompts and correct any unfair criteria.
- Analyze long agent transcripts to identify errors and provide concise feedback.
What they look for
- Ability to read long agent transcripts and identify specific errors.
- Precise writing skills and a commitment to verifying data.
Benefits
- Compensation of $40 to $75 per hour
How Keywordise applies for you
Alex Demo
[email protected]
From your CV
Yes, fully remote
- Sent
- 09:41
- Attached
- Your tailored CV
- 1Upload your CVAdd your CV once. It takes about a minute.
- 2Tailored to this jobKeywordise writes a CV and a cover letter for this job from your real experience.
- 3The form fills itselfName, email, experience and screening questions, field by field.
- 4Application sentYou get a receipt and follow it in your dashboard.
Apply with Keywordise AI7-day free trial, no card.
Apply from your AI assistant
Connect Keywordise to your assistant once, then send it the prompt. It applies for you.
Claude
- In Claude, open Settings, then Connectors, and choose Add custom connector.
- Name it Keywordise and paste the connector URL.
- Click Connect, log in to Keywordise and approve. Then send the prompt in a new chat.
https://keywordise.com/mcpDisconnect any time in your Keywordise settings.
ChatGPT
- In ChatGPT, open Settings, then Connectors. On plans that need it, turn on developer mode first.
- Add a new connector with the connector URL and choose OAuth. Leave the client ID and secret empty.
- Log in to Keywordise and approve. Then send the prompt in a new chat.
https://keywordise.com/mcpDisconnect any time in your Keywordise settings.
Muse
- Open Muse (muse.ai or the app) and send it the message below.
- If Muse asks how to sign in, answer OAuth. Open the sign-in link it sends, log in to Keywordise and approve.
- Send the prompt. Muse asks before it sends anything.
Add a custom connector called Keywordise. It is a remote MCP server at https://keywordise.com/mcp and uses OAuth. Give me the sign-in link.Muse is available in the US only for now.
Apply to Keywordise job ecinyresvmQuestions about this job
Is this a remote job, and where can I work from?
The posting lists it as remote for people in the United States.
What does this job pay?
The posting states USD 40 to 75 per hour.
What experience is needed?
It is a mid-level role. The posting asks for: ability to read long agent transcripts and identify specific errors; precise writing skills and a commitment to verifying data.
How do I apply with Keywordise AI?
Click Apply with Keywordise AI, create an account and upload your CV. Your CV and cover letter are written for this job from what you have really done, then Keywordise applies for you. Trial: 7 days, no card.
More remote business and finance jobs
- CCrowdStrikeSenior Finance Transformation Manager
- Remote, United States
CieloSpecialized Talent Sourcer- Remote, Philippines
Brown & Brown, IncUnderwriting Assistant- Remote, United States
- ZZZINC Zillow, Inc.Senior Sales Strategy Manager
- USD 112,800 to 180,200 per year
- Remote, United States
- SFStitch FixBusiness Operations & Workforce Management Analyst, Styling
- Remote, United States
- FFullsteamCustomer Onboarding & Training Specialist
- Remote, United States