Principal Machine Learning Engineer - AI Agent Quality

About the role

Atlassian is looking for a Principal Machine Learning Engineer to set the technical direction for AI agent quality in Studio, a platform where customers build, run and manage AI agents under the Rovo brand. It is a hands-on technical leadership role focused on measuring and improving the quality of agents that work in Jira and Confluence, automation rules or autonomously. The team is based in Sydney and Melbourne with remote options.

Key responsibilities

• Own the agent quality architecture: define what is measured, how production traffic is sampled and judged, and how results roll up without hiding failure modes behind one score.
• Design for cost and accuracy: decide which quality questions use deterministic checks, behavioral heuristics or contextual model-based judgment, and build the evaluation system around those choices.
• Architect evaluation and experimentation: define benchmarks, datasets, evaluation environments and observability, and guide teams in reproducible experiments and calibrated evaluators.
• Make quality part of shipping: set up regression gates, pre-release checks and drift detection on production traffic.
• Lead hard quality problems hands-on, such as agent selection and routing, tool-use reliability and quality for agents running in automations with no user feedback.
• Set a multi-quarter roadmap with product and data science, partner with central AI teams and model providers, and mentor engineers through design reviews.

Required profile

• 6+ years in ML/AI engineering, with deep expertise in LLMs, NLP, information retrieval or recommendation.
• Proven track record of building and shipping LLM-powered or agentic systems used by real customers and improving them through measurement.
• Strong product judgment: finding the real problem, defining measurable outcomes and choosing between deterministic, classical ML and LLM-based approaches.
• Record of influencing technical direction across teams and mentoring senior engineers.
• Sound judgment on trade-offs across quality, latency, cost, reliability, privacy and access control.

Required skills

• Strong evaluation and experimental skills: designing benchmarks, assessing label and judge quality, recognizing leakage and confounding, and connecting offline results to product impact.
• Software and data engineering skills, including data pipelines, model serving and production observability.
• Regular use of AI tools in own work and turning new research into production results.

Nice to have: experience building agent platforms, trajectory-level evaluation of multi-step agents, enterprise SaaS experience, and a master's or PhD in a computer science or machine learning field.