Data Scientist - AI & Model Validation

About the role

The Commonwealth Bank is hiring a Data Scientist for its AI & General Model Validation (AIGMV) team in Sydney. The team independently tests and challenges key machine learning, generative AI and agentic AI use cases across the Group, assessing whether AI solutions perform as intended by reviewing their data, methodology, assumptions and outcomes. The role works closely with model developers, product owners and business stakeholders while keeping the independence expected of a validation function.

Key responsibilities

• Deliver defined components of AI and ML validation, including technical analysis, testing and evaluation.
• Review data, methodologies, assumptions and model outcomes, investigating unexpected behaviour and potential weaknesses.
• Design and run evaluations for generative AI and agentic systems, including RAG, tool use, guardrails, routing, memory and LLM-as-a-Judge approaches.
• Build reproducible test harnesses, reusable evaluations and automation to make validation more efficient and consistent.
• Navigate unfamiliar codebases, reproduce evaluation pipelines and inspect trace data to understand model and agent behaviour.
• Communicate complex technical findings clearly to technical and non-technical stakeholders and share knowledge to develop AI validation practices, tooling and standards.

Required profile

• A curious, hands-on Data Scientist with sound analytical judgement, attention to detail and the confidence to investigate unexpected results.
• Experience across several of the listed areas helps; candidates who feel strong across some but not all areas are welcome.
• Strong communication skills and the ability to explain complex analysis and technical risks clearly.
• Candidates already in the Commonwealth Bank Group must apply through the bank's internal application process.

Required skills

• Advanced Python, GitHub, version control and CI/CD; SQL, R or TypeScript exposure is also valuable.
• Frameworks such as LangChain, LangGraph, OpenAI Agents SDK, Pydantic and platforms such as Langfuse or HoneyHive.
• Generative AI and agentic architectures, including RAG pipelines, tool use, guardrails and multi-agent routing.
• Hands-on experience building or evaluating AI solutions, including synthetic data; transformer fine-tuning is beneficial.
• Experience with AI coding agents such as Claude Code or Codex.
• Working knowledge of traditional machine learning (regression, random forests, XGBoost) and evaluation methods.