Principal Data Engineer

About the role

Medibank is looking for a Principal Data Engineer to lead the technical vision and engineering excellence of its next-generation data platforms. The role defines and leads the technical strategy, architecture, standards and delivery of enterprise data platforms, partnering with architects, engineers, data scientists and business leaders to build scalable, secure data solutions for analytics, machine learning and customer experiences. It also involves mentoring engineering teams and helping them adopt modern platform practices.

Key responsibilities

• Define the technical roadmap for the Databricks platform, enterprise data pipelines and AI/ML capabilities, and act as a senior technical advisor to engineering leaders and business stakeholders.
• Lead the design of scalable lakehouse architectures and enterprise data platforms on Databricks, and build reusable frameworks, accelerators and reference implementations.
• Guide teams on data modelling, data quality, pipeline orchestration, metadata management and AI/ML lifecycle management, and run technical workshops and onboarding sessions.
• Help lead a Data Platform Centre of Excellence and support delivery across globally distributed and offshore teams.
• Champion DataOps and MLOps practices (CI/CD, automated testing, data lineage, observability) and develop tools and automation to reduce technical debt.

Required profile

• 10+ years in data engineering, software engineering or a related field.
• 5+ years with Databricks or large-scale Spark platforms, and 5+ years delivering cloud-native data solutions.
• Demonstrated experience defining technical strategy and influencing enterprise architecture decisions.
• Experience leading engineering teams across multiple regions and geographies is highly regarded.

Required skills

• Databricks (Delta Lake, Unity Catalog, MLflow, Databricks SQL), PySpark, Spark SQL and lakehouse architecture.
• Data modelling (dimensional modelling, Data Vault, feature stores) and orchestration tools such as Airflow, Databricks Workflows or Azure Data Factory.
• MLOps: model monitoring, automated retraining, CI/CD for ML, model governance.
• CI/CD tooling (GitHub Actions, Azure DevOps or Jenkins) and Infrastructure as Code (Terraform, ARM/Bicep or CloudFormation).
• Python and SQL, REST APIs, Kafka/Kinesis, microservices; Azure services (ADLS, Synapse, Event Hub, Key Vault) and BI tools (Power BI, Tableau) are highly regarded.