Senior Data Engineer - Spark & Scala

About the role

A secure enterprise environment supporting critical infrastructure is looking for an experienced Spark and Scala Data Engineer. The role is advertised for Melbourne, with locations also listed for Adelaide, Brisbane, Sydney and Canberra. It is a rolling 3-month contract, with one day per fortnight onsite.

Key responsibilities

• Design and develop scalable data pipelines using Spark and Scala.
• Build and support batch and real-time data processing solutions.
• Analyse requirements and provide fit-for-purpose technical designs.
• Develop APIs, data services and reusable engineering frameworks.
• Deliver high-quality, well-tested code and perform code reviews.
• Support CI/CD, automation and production operations.
• Collaborate with stakeholders to deliver projects on time and within budget.
• Mentor junior engineers and contribute to continuous improvement.

Required profile

• Extensive experience in data engineering or big data development.
• Experience building real-time and batch data pipelines and large-scale big data platforms.
• Strong SQL and data analysis skills.
• Experience with CI/CD, automated testing and Agile delivery.
• Strong communication and stakeholder management skills.

Required skills

• Scala, Java and Apache Spark
• Kafka, Hive, HBase, HDFS and the Hadoop ecosystem
• CI/CD and automated testing