Senior Machine Learning Engineer - Rovo Agents Quality

About the role

Atlassian is looking for a Senior Machine Learning Engineer to join the Rovo Agents quality team in Studio, a Sydney-based team with remote options. Studio is where customers build, run and manage AI agents under the Rovo brand, and the team builds the systems that measure how well AI agents perform and use them to make them better. The team works with product, data science and Atlassian's central AI teams on applied ML in production.

Key responsibilities

• Design, build, ship and operate the evaluation and optimization systems for Rovo agents.
• Build offline evaluation suites and benchmarks reflecting realistic agent tasks, using real usage patterns, synthetic scenarios and simulated tools, and run them as regression tests.
• Build online quality metrics on production traffic, covering sampling strategy, behavioral signals such as tool errors, retry loops, failed handoffs and abandoned sessions, and model-based judging of selected agent trajectories.
• Calibrate judges against human ratings and independent outcome signals, and evolve evals so they do not saturate as agents improve.
• Use evaluation results and production signals to improve agent quality across prompts, context selection, tool use, retrieval and agent routing.
• Improve quality while reducing cost and latency across production LLM systems.
• Own projects from problem definition through experimentation, deployment and ongoing measurement; contribute to design and code reviews, coach engineers and help software engineers write good evals.

Required profile

• Bachelor's or master's degree, preferably in Computer Science or a related field, or equivalent practical experience.
• 4+ years of related industry experience in machine learning, including hands-on experience designing, evaluating and operating LLM-based or agentic systems.
• Experience building and operating ML systems in production, including practices to improve quality, reliability and cost.
• Regular use of AI tools in daily engineering work, with the judgment to validate their outputs.
• A strong ownership mindset and the ability to lead ambiguous technical work from problem definition through delivery.
• Clear communication, able to explain ML trade-offs to engineers, product managers and leaders.

Required skills

• Strong coding in Python and/or Kotlin, writing performant, production-quality code using AI.
• Experience designing experiments, building evaluation datasets and benchmarks, and analyzing model and system behavior.
• Familiarity with SQL, distributed data processing and cloud data environments.
• Nice to have: agent evaluation experience (trajectory analysis, human evaluation, LLM-as-a-judge), tool-using agents or RAG applications, retrieval/ranking/recommendation experience, and enterprise SaaS experience.