Data Engineer

About the role

As a Data Engineer you will support senior team members and stakeholders in enhancing regulatory processes through data profiling, analytics, and robust engineering solutions. You will play a key role in governing, interpreting, and transforming data into actionable intelligence.

Key responsibilities

• Collaborate within a high‑performing Scrum team to deliver BAU work and project milestones, interpreting requirements and managing risks.
• Design, build, and maintain scalable data pipelines in Databricks using Python, PySpark, and Spark SQL within a metadata‑driven framework.
• Develop and operate Databricks notebooks and jobs, optimise performance and cost through cluster configuration and Spark/Delta Lake tuning.
• Implement CI/CD practices, source control, automated testing, peer code review and CAB‑approved releases.
• Manage data security with Unity Catalog, including permissions, row filters, column masks and secrets management.
• Participate in rotating incident management, troubleshooting, root‑cause analysis and documentation of operational incidents.

Required profile

• At least four years of data engineering experience with hands‑on Databricks or equivalent cloud platform.
• Bachelor’s degree in a relevant field is desirable.
• Experience with Agile Scrum, SDLC and data lifecycle processes.

Required skills

• Databricks notebooks, Jobs/Workflows, classic compute clusters and Spark performance tuning.
• Python, PySpark, advanced SQL (Spark SQL, T‑SQL).
• Delta Lake (MERGE, upserts, schema evolution, OPTIMIZE, VACUUM).
• Unity Catalog security and governance.
• Azure ADLS Gen2, Azure Key Vault.
• Data modelling (relational, dimensional, medallion architecture) and RDBMS concepts.
• CI/CD, Git‑based workflows, unit testing and debugging.

What we offer

• Salary package of $135,995.90 plus superannuation.
• Hybrid working environment – up to 2 days remote per week.
• Leave loading 17.5% and monthly RDO with paid Christmas closure.
• Wellbeing programs, discounted fitness, health insurance and financial services.
• Professional development support and salary packaging options.