Senior AI Engineer for Observability
- Toronto, ON
- Senior
- Posted 3 weeks ago
or apply from Claude / ChatGPT / Muse
Senior role on a data analytics team, building monitoring and evaluation systems for AI models and agent workflows in a hybrid setup.
About the role
Non-deterministic AI models and agent workflows are hard to debug once they run in production, so the team builds the tracing, evaluation and governance layer around them.
At a glance
- Pay
- Not stated in the posting
- Location
- Toronto, ON
- Seniority
- Senior
- Skills
- Python, SQL, LogQL, PromQL, Prometheus, Grafana, Datadog, OpenTelemetry
- Posted
- 22 Oct 2026
- Last checked
- 15 Nov 2026
What you will do
- Build tracing for LLM calls, capturing prompts, tool calls, spans and intermediate outputs with OpenTelemetry.
- Create automated evaluation pipelines that check relevance, groundedness, hallucination and prompt adherence.
- Watch for model drift, data quality loss, latency spikes and error rates across production pipelines.
- Set up observability pipelines and dashboards in Prometheus, Grafana and Datadog to track SLIs and SLOs.
- Define guardrails for autonomous agents, including when they may fix problems on their own.
What they look for
- Seven to ten years in site reliability engineering, plus a degree in computer science, engineering or a related field.
- Hands-on AI observability work: automated detection, investigation and remediation of model issues.
- Python for custom evaluations, and SQL, LogQL or PromQL for querying telemetry, logs and metrics.
- Familiarity with model architectures, evaluation metrics, token usage and spotting concept or data drift.
Nice to have
- Experience in a heavily regulated industry.
How Keywordise applies for you
Alex Demo
[email protected]
From your CV
Yes, fully remote
- Sent
- 09:41
- Attached
- Your tailored CV
- 1Upload your CVAdd your CV once. It takes about a minute.
- 2Tailored to this jobKeywordise writes a CV and a cover letter for this job from your real experience.
- 3The form fills itselfName, email, experience and screening questions, field by field.
- 4Application sentYou get a receipt and follow it in your dashboard.
Apply with Keywordise AI7-day free trial, no card.
Apply from your AI assistant
Connect Keywordise to your assistant once, then send it the prompt. It applies for you.
Claude
- In Claude, open Settings, then Connectors, and choose Add custom connector.
- Name it Keywordise and paste the connector URL.
- Click Connect, log in to Keywordise and approve. Then send the prompt in a new chat.
https://keywordise.com/mcpDisconnect any time in your Keywordise settings.
ChatGPT
- In ChatGPT, open Settings, then Connectors. On plans that need it, turn on developer mode first.
- Add a new connector with the connector URL and choose OAuth. Leave the client ID and secret empty.
- Log in to Keywordise and approve. Then send the prompt in a new chat.
https://keywordise.com/mcpDisconnect any time in your Keywordise settings.
Muse
- Open Muse (muse.ai or the app) and send it the message below.
- If Muse asks how to sign in, answer OAuth. Open the sign-in link it sends, log in to Keywordise and approve.
- Send the prompt. Muse asks before it sends anything.
Add a custom connector called Keywordise. It is a remote MCP server at https://keywordise.com/mcp and uses OAuth. Give me the sign-in link.Muse is available in the US only for now.
Apply to Keywordise job 4hf7uaj5ooQuestions about this job
Where is this job based?
The posting lists Senior AI Engineer for Observability as a role based in Toronto, ON, Canada.
What does this job pay?
The posting does not state a salary, so there is no figure to quote.
What experience is needed?
It is a senior role. The posting asks for: seven to ten years in site reliability engineering, plus a degree in computer science, engineering or a related field; hands-on AI observability work: automated detection, investigation and remediation of model issues; python for custom evaluations, and SQL, LogQL or PromQL for querying telemetry, logs and metrics. Skills it names include Python, SQL, LogQL, PromQL.
How do I apply with Keywordise AI?
Click Apply with Keywordise AI, create an account and upload your CV. Your CV and cover letter are written for this job from what you have really done, then Keywordise applies for you. Trial: 7 days, no card.
More IT and software jobs in Toronto, ON
- AAppnovationTechnical Lead
- Toronto, ON
- A global luxury travel providerIntern - Analytics
- Toronto, ON
- A global technology corporationOffice Administrator
- CAD 50,000 to 60,000 per year
- Toronto, ON
- CCityliticsData Engineer
- Toronto, ON
- GGuidepointSenior Software Engineer - Search & AI
- Toronto, ON
- MLManu LifeVP, GWAM Technology Operations & Platforms
- Toronto, ON