DevOps Specialist

Project Overview

A major financial services organisation is modernising its on-premise data lake to build a secure, compliant, and future-proof data platform. The current environment is based on Cloudera-supported Apache technologies (Spark, NiFi, Airflow) and will be migrated to a modern, cloud-native architecture on Google Cloud Platform (GCP).

This role involves hands-on delivery as part of a DataOps team, contributing directly to the migration, containerisation, and implementation of managed cloud services using infrastructure-as-code best practices.

Key Responsibilities

• Support migration from on-premise data lake to GCP
• Develop and optimise data pipelines using Apache Spark
• Implement containerised solutions (Docker, Kubernetes)
• Build infrastructure using Terraform (IaC)
• Work with ETL processes and database management
• Contribute to CI/CD and automation practices
• Ensure compliance, scalability, and reliability of the data platform

Required Skills

• Apache Spark
• Docker & Kubernetes
• Terraform (Infrastructure as Code)
• Apache NiFi
• Python & SQL
• Database management & ETL processes
• Cloudera ecosystem
• Scala

Nice to Have

• Google Cloud Platform (especially Dataproc)
• Apache Airflow
• CI/CD pipelines
• Observability & monitoring tools
• YAML, Linux & Windows environments
• Elasticsearch & Kibana
• CLI proficiency
• Experience with modern cloud-native architectures