Principal Data Engineer (Apache Spark, dbt, Airflow)

About the role

ASX, a global securities exchange, is looking for a Principal Data Engineer to lead the design and delivery of its data platform. The role covers ingestion, storage, transformation and serving layers, and involves guiding platform data engineers embedded in workload teams.

Key responsibilities

• Evolving the end-to-end data platform design, including ingestion, storage, transformation and serving layers across the organisation.
• Leading the design of data lake and data mesh architectures, ensuring scalability, fault tolerance and cost efficiency.
• Designing, building and maintaining reusable frameworks, libraries and internal developer tooling to standardise data pipeline development.
• Identifying and engaging stakeholders such as architects, product owners, workload engineers and business SMEs.
• Ensuring data pipelines and engineering processes follow engineering standards, have process controls and support compliance reporting.
• Driving DataOps adoption, including CI/CD for data pipelines, infrastructure-as-code, automated testing and observability.
• Supporting data privacy, lineage tracking and regulatory controls with security and data office teams.
• Managing the deliveries and stakeholders of several platform data engineers embedded in workload teams.

Required profile

• 10+ years of data engineering experience.
• Proven track record designing and operating large-scale data platforms and pipelines in AWS.
• Deep expertise in distributed systems, batch and streaming data processing and modern data architecture patterns.
• Knowledge of data lake, data warehouse and data lakehouse patterns, and experience with complex data structures such as JSON, Parquet, Avro and XML.
• Nice to have: at least 3 years in a lead data engineering role, experience implementing data management capabilities in data platform frameworks, and familiarity with Confluence, JIRA or ServiceNow.
• Candidates must be legally authorised to work in Australia permanently; background checks are part of onboarding.

Required skills

• Enterprise data pipeline tooling: Apache Spark, dbt, Airflow
• AWS cloud engineering
• Data infrastructure automation with Terraform
• Design governance and DataOps principles