Databricks Data Engineer Consultant

About the role

An experienced Databricks Data Engineer is sought to join a Data & AI team. You will design, develop and maintain scalable data pipelines and transformation workflows on Azure Databricks, supporting advanced analytics and data engineering initiatives.

Key responsibilities

• Design, develop and optimise scalable data ingestion pipelines using Azure Data Factory, Databricks and Apache Spark.
• Build and maintain Databricks notebooks with PySpark for data preparation, transformation and enrichment to support Data Science use cases.
• Collaborate with Data Scientists to deliver clean, structured datasets on time.
• Implement data quality checks, validation rules and monitoring.
• Integrate structured, semi-structured and unstructured data from multiple sources.
• Ensure data security, governance and compliance with banking regulations.
• Optimise Spark jobs for performance and cost-efficiency in Azure.
• Take part in code reviews, design discussions and agile ceremonies, and document pipelines and workflows.

Required profile

• At least 3 years of hands-on experience in Azure Databricks.
• Strong analytical and problem-solving skills, excellent communication and stakeholder management, and ability to work in cross-functional teams.
• Agile mindset and experience in Scrum/Agile environments.
• Bachelor's or Master's degree in a related field preferred; Australian citizenship required and security clearance eligibility needed.
• Location flexible across Brisbane, Sydney and Melbourne.

Required skills

• Apache Spark, especially PySpark
• Azure Data Factory, Azure Synapse, Azure Blob Storage and Delta Lake
• ETL/ELT, data modelling and data warehousing
• Python and/or Scala
• CI/CD with Azure DevOps, Git and Terraform (preferred)
• SQL, NoSQL and data lake architectures
• Data governance, security and compliance in financial services