Senior Software Development Engineer in Test, AI

About the role

Own quality assurance for generative AI and agentic systems, building testing strategies, evaluation frameworks and metrics for LLM-powered applications so AI features stay reliable, accurate and trustworthy.

What you will do

• Design test plans, evaluation frameworks and quality metrics for LLM applications and agentic workflows
• Build end-to-end automated test suites in Python covering evaluations, APIs, UI and performance
• Create automated prompt testing and regression suites that catch unintended model behavior changes
• Define multi-dimensional criteria for accuracy, relevance, safety, consistency and latency
• Run adversarial red-team testing to find hallucinations, bias and security weaknesses
• Deploy real-time monitoring and alerting to flag quality degradation in production

What they look for

• Deep understanding of generative AI, including LLMs, prompt engineering and AI application patterns
• Strong hands-on Python coding skills
• Expertise with test automation frameworks and scripting such as Python, JavaScript, Selenium and Pytest
• Solid grounding in functional, integration, regression, performance and security testing
• Experience with API testing tools like Postman and REST Assured, plus backend testing