Remote job · IT and software
Senior AI/ML Test and Evaluation Engineer
In one line: Senior AI/ML Test and Evaluation Engineer is a remote role in the United States at an open-source AI platform company.
- USD 145,000 to 250,000 per year
- Remote (US)
- An open-source AI platform company
- Posted 25 Aug 2026
Apply with Keywordise AI
Upload your CV and Keywordise tailors your CV and cover letter to this job and applies for you. 7-day free trial, no card.
Apply with Keywordise AIPrefer your AI assistant? Apply from Claude or ChatGPT
Add Keywordise to your assistant as a connector, sign in, then send it this prompt. It applies for you through Keywordise.
https://keywordise.com/mcpApply to Keywordise job mdnf5pkjugAbout the role
Build and run the benchmarking and evaluation capability at the heart of an AI platform, judging models and agentic workflows and reporting to senior stakeholders on what is ready to deploy.
At a glance
- PayUSD 145,000 to 250,000 per year
- Remote scopeRemote, United States
- Posted25 Aug 2026
- Last checked18 Oct 2026
What you will do
- Create and operate a test and evaluation platform for AI models and agentic systems
- Build evaluation methods that blend human expert and automated judging across several input and output types
- Design and improve interfaces so customers can describe their workflows in an intuitive way
- Develop repeatable ways to compare performance against current operational baselines
- Document limitations and expose failure modes, including ones nobody asked about
- Produce reports for senior stakeholders that inform decisions on which capabilities to field
What they look for
- Hands-on engineering experience on an open-source toolchain
- Ability to design evaluation harnesses that pair automated metrics with structured expert judgment
- Skill in building methodologies that hold up when rerun later with a different model
- Experience designing user interfaces for workflow expression
- U.S. citizenship and willingness to obtain and keep a security clearance
- Ability to travel up to 15 percent for the hybrid role
Questions about this job
Is this a remote job, and where can I work from?
The posting lists it as remote for people in the United States.
What does this job pay?
The posting states USD 145,000 to 250,000 per year.
What experience is needed?
The posting asks for: hands-on engineering experience on an open-source toolchain; ability to design evaluation uses that pair automated metrics with structured expert judgment; skill in building methodologies that hold up when rerun later with a different model.
How do I apply with Keywordise AI?
Press the Apply with Keywordise AI button and add your CV. Keywordise tailors your CV and cover letter to this posting and sends the application for you, with a free 7-day trial and no card.
How it works
- 1Upload your CVSign up with Google or your email and add your CV. It takes about a minute.
- 2We tailor your applicationKeywordise writes a CV and a cover letter for this job from your real experience.
- 3We apply for youYour application goes to the employer and you follow it in your dashboard.
More remote IT and software jobs
- IT Support Specialist Level 2A US-based organisation with international teams · Remote (Peru)
- NOC EngineerA GPU cloud infrastructure provider · Remote, worldwide
- AI Content EngineerA data quality software company · Remote (Germany)
- Staff Software EngineerA cloud cost management software company · Remote, worldwide
- Solutions Architect, Digital Native Business (Fintech)A data and AI platform company · Remote (US)
- IFS Techno-Functional Consultant / IFS Platform SpecialistAn IT services provider for enterprise ERP · Remote (Brazil)
Browse more: Remote IT and software jobs · Remote jobs in the United States · Remote IT and software jobs in the United States
Explore more: remote jobs, the Keywordise Job Market Index, the blog.