Data Engineer – Web Scraping

About the role

We are seeking a Data Engineer specialized in web scraping to build scalable data collection solutions and pipelines that feed business‑critical insights. The role is based in Australia and works closely with analysts, engineers and cross‑functional teams to deliver reliable datasets for strategic decision‑making.

Key responsibilities

• Collaborate with analysts and stakeholders to understand data requirements and deliver tailored data solutions.
• Design, develop, and maintain web scrapers for structured and unstructured sources.
• Clean, transform, validate, and manipulate large datasets using Python and Pandas.
• Build and maintain data ingestion pipelines into databases or data warehouses.
• Schedule, monitor, and optimise scraping workflows with Apache Airflow.
• Develop quality control checks to ensure data integrity and availability.
• Investigate and resolve pipeline issues and production incidents.
• Design and enhance internal tools, automation frameworks and platform capabilities.
• Work with cross‑functional engineering teams to implement scalable data processing workflows.

Required profile

• Bachelor’s or Master’s degree in Computer Science or a related technical discipline.
• 2–4 years of professional software development experience.
• Strong programming skills in Python and solid knowledge of SQL/databases.
• Experience with data cleaning, transformation and analysis using Pandas.
• Hands‑on experience with web technologies (HTML, JavaScript, APIs) and scraping frameworks such as Selenium or Scrapy.
• Proven ability to process, clean and transform large datasets.
• Proactive problem‑solving mindset.

Required skills

• Python
• SQL
• Pandas
• Selenium
• Scrapy
• XPath
• Fiddler
• Postman
• Apache Airflow
• HTML
• JavaScript
• APIs
• Data pipelines
• Databases / Data warehouses