Data Engineer

Responsibilities

 Implement data transformations and analytics using Python and PySpark.

 Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and

Databricks.

 Configure and manage Databricks Unity Catalog for secure data governance and access control.

 Orchestrate workflows using Apache Airflow for scheduling and automation.

 Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.

 Set up and maintain CI/CD pipelines for data workflows and deployments.

 Manage version control and branching strategies using Git.

 Write and execute unit tests to ensure code quality and reliability.

 Monitor, troubleshoot, and optimize data processes for performance and cost efficiency.

Required

 Programming: Proficiency in Python and PySpark for data engineering tasks.

 Python - good understanding of basics (like OOP, functions, imports, exceptions, file handling). Pandas

library knowledge

 Azure Knowledge: Strong experience with Azure Data Factory, Azure Data Lake, Azure Synapse, or related

services.

 Databricks: Hands-on experience in building and managing workflows in Databricks, including Unity Catalog.

 Orchestration: Experience with Apache Airflow for workflow scheduling and automation.

 CI/CD: Familiarity with tools like Azure DevOps or Jenkins for automated deployments.

 Version Control: Expertise in Git for code management.

 Testing: Strong understanding of unit testing frameworks for data pipelines.

 Agile Methodology: Experience working in Agile teams and using tools like Jira or Azure Boards.