Data Engineer

Data Engineer - Apache Airflow - Long Term Contract Opportunity

This role is integral to delivering innovative data solutions for a global IT consultancy, enabling clients across diverse industries to harness data for strategic decision-making. You will design and maintain scalable data infrastructure, ensuring seamless data flow and robust governance to support transformative business outcomes. Working with international teams, your contributions will drive client success by delivering reliable, high-quality data systems.

Key Responsibilities

• Develop and optimise data pipelines using PySpark and Airflow to process large-scale datasets.
• Manage containerised environments with Docker and Kubernetes for scalable deployments.
• Implement CI/CD workflows using GitHub Actions to streamline development processes.
• Perform maintenance and troubleshoot issues in data systems to ensure reliability.
• Uphold data quality, integrity, and governance to meet client and regulatory standards.
• Collaborate with cross-functional teams to deliver tailored data solutions.

Essential Qualifications

• Experience in data engineering or a related field.
• Proficiency in PySpark, Airflow, Unix, Docker, Kubernetes, and GitHub Actions.
• Strong understanding of data quality, integrity, and governance principles.
• Effective problem-solving and communication skills.

Desirable Qualifications

• Experience with cloud platforms (e.g., AWS, Azure, or GCP).
• Knowledge of advanced data governance frameworks.