Cloud Engineer

Job Title: Azure Databricks Engineer

Job Summary:

We are looking for an experienced Azure Databricks Engineer to join our team. The ideal candidate will have hands-on experience in designing, developing, and managing scalable data solutions using Azure Databricks. The role involves working collaboratively with cross-functional teams to build and maintain robust data pipelines and analytics solutions.

Key Responsibilities:

• Design and Implement Data Pipelines: Develop, deploy, and optimize data pipelines and ETL/ELT processes using Azure Databricks.
• Collaborate with Data Teams: Work closely with data analysts, data scientists, and data engineers to gather requirements and design solutions.
• Data Modeling and Management: Create and maintain data models that ensure data quality and governance.
• Optimize Performance: Tune Databricks clusters and notebooks to maximize efficiency and minimize cost.
• Integrate with Azure Ecosystem: Connect and integrate Databricks with other Azure services like Azure Data Lake Storage (ADLS), Azure Synapse Analytics, Azure SQL Database, and Azure Data Factory.
• Develop Scalable Solutions: Ensure solutions are scalable, maintainable, and aligned with best practices for data engineering.
• Support and Troubleshooting: Monitor jobs, handle exceptions, and resolve issues related to data pipelines and processing.
• Documentation: Create and maintain technical documentation for data workflows and solutions.

Technical Skills:

• Strong proficiency in Apache Spark and its applications within Azure Databricks.
• Experience with Python/Scala/Spark SQL for data processing.
• Expertise in integrating Databricks with Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), and Azure Synapse.
• Knowledge of CI/CD pipelines using tools like Azure DevOps or GitHub Actions.
• Proficiency with data modeling concepts and SQL for building and querying data structures.

Additional Desirable Skills

• Certifications: Azure certifications such as Microsoft Certified: Azure Data Engineer Associate or Databricks Certification.
• Experience with machine learning workflows and frameworks within Databricks.
• Familiarity with Big Data technologies and distributed computing.
• Education: Bachelor’s degree in computer science, Engineering, Data Science, or related field (or equivalent work experience).

Salary: 12,000 - 30,000 depending on experience