AlteryxHiring

Lead Software Development Engineer in Test

  • USD 165,000 to 185,000 per year
  • Remote, United States
  • Lead
  • Posted 8 days ago
Apply with Keywordise AI

or apply from Claude / ChatGPT / Muse

In one line

Lead test engineer for an AI assistant product, owning the test setup that measures how well the agent performs; US based with a salary range and bonus.

About the role

A small group works with engineering, product and data science to judge agent quality through evaluation and observability. Good work means catching regressions early and turning production failures into lasting coverage.

At a glance

Pay
USD 165,000 to 185,000 per year
Remote scope
Remote, United States
Seniority
Lead
Skills
Python, Selenium, Pytest, Locust, Allure, LangChain, LangSmith
Posted
21 Oct 2026
Last checked
29 Oct 2026

What you will do

  • Turn requirements into a test strategy and detailed test cases
  • Write manual and automated tests across unit, component, integration and end-to-end levels
  • Build and grow scalable test frameworks for feature and AI workflow testing

What they look for

  • Experience judging AI agents on multi-step tasks, tool choice, tool-call accuracy and recovery from failure
  • Knowledge of agent frameworks such as LangChain, able to debug context handling, tool runs and agent state
  • Hands-on use of LangSmith or similar for evaluation datasets, custom evaluators and experiment comparisons
  • Strong Python engineering for evaluation tooling, metrics pipelines and reporting, plus applied statistics to separate real gains from noise
  • Technical lead experience: guiding evaluation practice, mentoring engineers and wiring evaluations into CI/CD

Nice to have

  • JavaScript or TypeScript experience
  • Modelling experience

Benefits

  • Target annual bonus of 15% of base salary

How Keywordise applies for you

  1. 1Upload your CVAdd your CV once. It takes about a minute.
  2. 2Tailored to this jobKeywordise writes a CV and a cover letter for this job from your real experience.
  3. 3The form fills itselfName, email, experience and screening questions, field by field.
  4. 4Application sentYou get a receipt and follow it in your dashboard.

Apply with Keywordise AI7-day free trial, no card.

Apply from your AI assistant

Connect Keywordise to your assistant once, then send it the prompt. It applies for you.

ClaudeConnectorChatGPTConnectorMuseby Meta

Claude

  1. In Claude, open Settings, then Connectors, and choose Add custom connector.
  2. Name it Keywordise and paste the connector URL.
  3. Click Connect, log in to Keywordise and approve. Then send the prompt in a new chat.
Connector URL
https://keywordise.com/mcp

Disconnect any time in your Keywordise settings.

ChatGPT

  1. In ChatGPT, open Settings, then Connectors. On plans that need it, turn on developer mode first.
  2. Add a new connector with the connector URL and choose OAuth. Leave the client ID and secret empty.
  3. Log in to Keywordise and approve. Then send the prompt in a new chat.
Connector URL
https://keywordise.com/mcp

Disconnect any time in your Keywordise settings.

Muse

  1. Open Muse (muse.ai or the app) and send it the message below.
  2. If Muse asks how to sign in, answer OAuth. Open the sign-in link it sends, log in to Keywordise and approve.
  3. Send the prompt. Muse asks before it sends anything.
Message for Muse
Add a custom connector called Keywordise. It is a remote MCP server at https://keywordise.com/mcp and uses OAuth. Give me the sign-in link.

Muse is available in the US only for now.

Prompt
Apply to Keywordise job 4bknuomnwl
How to connect

Questions about this job

Can I do this job remotely, and from which countries?

The posting lists it as remote for people in the United States.

What does this job pay?

The posting states USD 165,000 to 185,000 per year.

What experience is needed?

It is a lead role. The posting asks for: experience judging AI agents on multi-step tasks, tool choice, tool-call accuracy and recovery from failure; knowledge of agent frameworks such as LangChain, able to debug context handling, tool runs and agent state; hands-on use of LangSmith or similar for evaluation datasets, custom evaluators and experiment comparisons. Skills it names include Python, Selenium, Pytest, Locust.

How do I apply with Keywordise AI?

Select Apply with Keywordise AI, sign up and upload your CV. Keywordise writes a CV and a cover letter for this job from your real experience and applies for you. The 7-day free trial needs no card.

More remote IT and software jobs

All remote IT and software jobs
AlteryxRemote, United States
Apply with Keywordise AI