Senior DevOps Engineer - Databricks

Required Profiles

1

Required Qualifications

Technical Skills

• 8+ years of experience in DevOps, Cloud Engineering, or Cloud Architecture roles
• Strong hands-on experience with at least one major cloud platform (AWS, Azure, or GCP)
• Proven experience provisioning and managing Databricks workspaces
• Expert-level knowledge of CI/CD tools (e.g., Jenkins, GitLab CI, Azure DevOps, GitHub Actions, CircleCI)
• Strong proficiency in Infrastructure as Code (Terraform, CloudFormation, ARM Templates, Ansible, or similar)
• Experience with containerisation and orchestration (Docker, Kubernetes)
• Proficiency in scripting languages (Python, Bash, PowerShell, or similar)
• Understanding of networking, security, and compliance in cloud environments

Professional Experience

• Background in Cloud Engineering or Cloud Architecture
• Experience working in Agile/Scrum environments
• Track record of implementing DevOps practices in enterprise settings
• Experience with database technologies and data platforms

Soft Skills

• Strong problem-solving and analytical abilities
• Excellent communication and collaboration skills
• Ability to work effectively with cross-functional teams
• Self-motivated with a continuous learning mindset
• Ability to mentor junior team members

Preferred Qualifications

• Cloud certifications (AWS Certified Solutions Architect, Azure Solutions Architect Expert, Google Cloud Professional Cloud Architect, or equivalent)
• Databricks certifications
• Experience with monitoring and observability tools (Prometheus, Grafana, ELK Stack, Splunk, Datadog)
• Knowledge of security best practices and DevSecOps
• Experience with GitOps workflows
• Familiarity with service mesh technologies (Istio, Linkerd)
• Experience in regulated industries (Financial Services, Healthcare, Public Service)

Key Responsibilities

Cloud Infrastructure & Architecture

• Design, implement, and manage scalable cloud infrastructure solutions on major cloud platforms (AWS)
• Architect and provision Databricks workspaces for data engineering and analytics workloads
• Ensure cloud infrastructure follows best practices for security, performance, and cost optimisation
• Provide technical leadership in cloud architecture decisions and infrastructure design

CI/CD Pipeline Development

• Design and build robust Continuous Integration/Continuous Delivery (CI/CD) pipelines
• Implement automated testing, deployment, and release management processes
• Optimise build and deployment workflows for efficiency and reliability
• Integrate security scanning and quality gates into CI/CD pipelines

Infrastructure as Code (IaC)

• Develop and deploy Infrastructure as Code using tools such as Terraform, CloudFormation, ARM Templates, or Pulumi
• Maintain version-controlled infrastructure configurations
• Implement automated infrastructure provisioning and configuration management
• Ensure infrastructure consistency across development, staging, and production environments

DevOps Best Practices

• Champion DevOps culture and practices across development teams
• Implement monitoring, logging, and alerting solutions
• Drive automation initiatives to improve operational efficiency
• Collaborate with application development teams to optimise deployment processes