Data Scientist, Agent Evaluations & Quality

Overview

About the Role This company is building an AI executive assistant that operates across email, calendars, meetings, and business software. As a Data Scientist, Agent Evaluations & Quality , you will own the measurement system that determines whether the assistant is genuinely improving in ambiguous, real-world environments. You'll partner directly with AI Agent Capabilities engineers to generate t

What you'll do

• About the Role This company is building an AI executive assistant that operates across email, calendars, meetings, and business software.
• As a Data Scientist, Agent Evaluations & Quality , you will own the measurement system that determines whether the assistant is genuinely improving in ambiguous, real-world environments.
• You'll partner directly with AI Agent Capabilities engineers to generate the evidence that shapes product decisions, model choices, and release quality.
• This is a high-ownership, deeply technical role at the intersection of applied data science, LLM evaluation, and product quality, ideal for someone who thrives on turning hard, open-ended quality questions into rigorous, actionable answers.
• What You'll Do Architect and maintain automated evaluation pipelines that measure agent quality across product surfaces.
• Translate agent capabilities into explicit pass, partial-pass, and failure criteria for complex multi-step tasks.
• Build representative gold datasets and regression suites covering real workflows, edge cases, and adversarial scenarios.
• Define meaningful metrics, task success, tool-selection accuracy, instruction adherence, factual consistency, latency, cost, and reliability.
• Design deterministic and model-based graders, calibrate LLM-as-a-judge systems, and track grader agreement.
• Compare models, prompts, and implementations using rigorous offline experiments and production evidence.

Requirements

• Practical experience with Engineering.
• Experience level: 4+ yrs.
• Strong written and verbal communication in English.
• Comfortable working remotely with distributed teams.

Skills