Data Engineer (Kinesis & Databricks)

About the role

This permanent position is based in Canberra or Melbourne and starts in December. You will be part of a sovereign delivery team building a high‑volume, real‑time data pipeline using Amazon Kinesis and Databricks.

Key responsibilities

• Build and run the real‑time layer: Kinesis Data Streams and Firehose feeding Databricks via Spark Structured Streaming into Delta tables.
• Own latency, reliability and cost, implement monitoring in CloudWatch and ensure streaming pipelines integrate cleanly with batch pipelines.

Required profile

• Hands‑on production experience with Kinesis Data Streams and/or Kinesis Data Firehose.
• Experience with Spark Structured Streaming in production and Databricks (Delta Lake).
• Strong Python and SQL coding skills and solid AWS fundamentals (IAM, S3, CloudWatch).
• Australian citizen with active Baseline clearance (NV1 also welcome).

Required skills

• Kinesis Data Streams
• Kinesis Data Firehose
• Spark Structured Streaming
• Databricks
• Delta Lake
• Python
• SQL
• AWS IAM
• AWS S3
• AWS CloudWatch
• Unity Catalog (nice to have)
• Kafka or MSK (nice to have)

What we offer

• Salary up to $140,000 AUD inclusive of superannuation.
• Consultant level permanent role starting 1 December.
• Location in Canberra or Melbourne with flexible work arrangements and structured career development paths.