Remote job · IT and software
Senior Data Scientist, AI Evaluation
In one line: Senior Data Scientist, AI Evaluation is a remote role in the United States at a global brokerage infrastructure company.
- Remote (US)
- A global brokerage infrastructure company
- Posted 7 Oct 2026
Apply with Keywordise AI
Upload your CV and Keywordise tailors your CV and cover letter to this job and applies for you. 7-day free trial, no card.
Apply with Keywordise AIPrefer your AI assistant? Apply from Claude or ChatGPT
Add Keywordise to your assistant as a connector, sign in, then send it this prompt. It applies for you through Keywordise.
https://keywordise.com/mcpApply to Keywordise job b6igvlyx7eAbout the role
You will define how model and agent correctness is measured, owning the quality bar independently of the teams building those systems. The work turns vague quality questions into trusted scoring methods and evaluation loops.
At a glance
- PayNot stated in the posting
- Remote scopeRemote, United States
- Posted7 Oct 2026
- Last checked18 Oct 2026
What you will do
- Set ground truth, metrics and scoring approaches for models and agents
- Create repeatable evaluation cycles that track quality and flag regressions before launch
- Work with engineering and analytics engineering to put evaluation tooling into production
- Turn evaluation findings into concrete recommendations for improving systems
- Define evaluation guidelines, documentation and review practices
- Guide and align colleagues on evaluation standards
What they look for
- Strong quantitative background
- Track record of rigorous, measurable work in evaluation, experimentation or model validation
- Ability to translate ambiguous quality questions into concrete measurement designs
- Comfort partnering with product, engineering and business stakeholders
Questions about this job
Is this a remote job, and where can I work from?
The posting lists it as remote for people in the United States.
What does this job pay?
No pay range is stated in the posting.
What experience is needed?
The posting asks for: strong quantitative background; track record of rigorous, measurable work in evaluation, experimentation or model validation; ability to translate ambiguous quality questions into concrete measurement designs.
How do I apply with Keywordise AI?
Press the Apply with Keywordise AI button and add your CV. Keywordise tailors your CV and cover letter to this posting and sends the application for you, with a free 7-day trial and no card.
How it works
- 1Upload your CVSign up with Google or your email and add your CV. It takes about a minute.
- 2We tailor your applicationKeywordise writes a CV and a cover letter for this job from your real experience.
- 3We apply for youYour application goes to the employer and you follow it in your dashboard.
More remote IT and software jobs
- Junior CRM DeveloperAn online grocery retailer · Remote (UK)
- Senior Agentic Platform EngineerA global advertising technology company · Remote (Netherlands)
- Technical Support ConsultantA global customer and technical support services provider · Remote, Europe
- Space Tech Co-Founder and Head of SalesA startup venture builder for founders · Remote (Denmark)
- Junior Technical Support Specialist (French-speaking, Remote)A business process outsourcing provider · Remote (Portugal)
- Senior Quality Assurance EngineerA geospatial property intelligence technology company · Remote (US)
Browse more: Remote IT and software jobs · Remote jobs in the United States · Remote IT and software jobs in the United States
Explore more: remote jobs, the Keywordise Job Market Index, the blog.