Data Engineer – AWS Kinesis & Databricks (AU Citizen)

About the role

Accenture is looking for a hands-on Data Engineer in Canberra to build reliable, real-time data pipelines using Amazon Kinesis and Databricks. You will work with modern AWS data technologies to develop and optimise streaming pipelines that move high-volume data from Kinesis into the Databricks Lakehouse using Spark Structured Streaming and Delta tables.

Key responsibilities

• Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
• Develop streaming solutions in Databricks using Spark Structured Streaming.
• Ingest and manage streaming data within Delta tables.
• Optimise pipelines for performance, latency, reliability and cost.
• Build monitoring and alerting using AWS CloudWatch.
• Work with batch data engineers so that streaming and batch pipelines integrate effectively.
• Develop clean, testable Python and SQL code.
• Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.

Required profile

• Hands-on production experience with Amazon Kinesis Data Streams and/or Kinesis Data Firehose; Kinesis experience is essential.
• Australian citizenship with an active security clearance, required because of the secure nature of the role.
• Strong troubleshooting and problem-solving skills.

Required skills

• Strong Python and SQL skills.
• Experience with Apache Spark Structured Streaming.
• Hands-on experience with Databricks, ideally including Delta Lake and Unity Catalog.
• Good knowledge of AWS services, particularly IAM, S3 and CloudWatch.
• Understanding of scalable and reliable data-pipeline design.
• Experience with Kafka is beneficial.

What we offer

• 18 weeks of paid parental leave.
• Long and short-term career break opportunities.
• Structured career development program and local and international career opportunities.
• Certified Family Inclusive Workplace.
• Flexible work arrangements.