Senior Data Engineer

About the role

Datium Insights is looking for a Senior Data Engineer to work closely with the Head of Data Engineering and lead a team of Data Engineers who build, maintain and support the company's Data Lake and Data Warehouses. The role is hands-on, about 60% focused on ETL frameworks and pipelines, new development and operations, and 40% on mentoring and leading engineers to deliver high-quality results.

Key responsibilities

• Liaise with key business stakeholders, facilitate design workshops and meetings, break work into chunks and assign it to Data Engineers, and review requirements and documentation.
• Review non-functional requirements against budget, design and document Airflow and other frameworks, and design and document pipelines within the overall architecture.
• Develop and maintain ETL pipelines, complex transformation logic and pipeline development standards; peer review code and approve pull requests.
• Manage code deployment and promotion between environments, and provide guidance and coaching to Data Engineers.
• Ensure test cases are created, organise release testing cycles, and report on test results and issues.
• Write complex SQL scripts and stored procedures, understand and maintain SSIS ETL pipelines, and expand the Airflow framework.
• Maintain security and access to the warehouse, enforce coding standards, and ensure data quality meets SLAs.
• Act as escalation point for system issues, triage errors, run regression tests and keep pipelines running within agreed SLAs.

Required profile

• 7+ years of experience in Data Engineering and/or Software Engineering.
• 3+ years leading small to medium teams of 4-10 people.
• 3+ years of recent Python development experience, or equivalent in Java, C# or Scala.
• Good written and verbal communication.

Required skills

• Advanced knowledge of SQL
• Python, with expert scripting skills and use of standard libraries
• Airflow and SSIS ETL pipelines