Senior Data Engineer

Job Title - Senior Data Engineer

Location - Dubai (Onsite)

Duration - 6+ Months

Job Description Databricks Senior Data Engineer

• A Databricks Data Engineer would design, build, and maintain data pipelines and workflows using Databricks, a cloud-based platform for big data processing. Will collaborate with data scientists and analysts to understand data needs and ensure data quality, security, and compliance. Is responsible for data integration, transformation, and loading processes, often leveraging tools like PySpark and SQL.

Key Responsibilities of a Databricks Data Engineer:

• Provides the implementation strategy and executes backup, recovery, and technology solutions to perform analysis.
• Performs ETL tool capabilities with the ability to pull data from various sources and perform a load of the transformed data into a database or business intelligence platform.
• Codes using programming language used for statistical analysis and modeling such as Python/Spark
• Data Pipeline Engineering: Design, develop, and deploy robust and scalable data pipelines using Databricks, incorporating data extraction from diverse sources (databases, APIs, streaming platforms), transformation and cleansing using Spark, and loading into target systems (data lakes, data warehouses, etc.). Optimize pipeline performance for efficiency and cost-effectiveness.
• Databricks Ecosystem Expertise: Utilize the full capabilities of the Databricks platform, including Databricks SQL, Delta Lake, Databricks Runtime, and Databricks Workflows, to orchestrate complex data workflows and ensure data quality and pipeline reliability.
• Cloud Integration: Seamlessly integrate Databricks with cloud services (Azure, AWS, or GCP) for storage, compute, and security, leveraging services like Azure Data Lake Storage, AWS S3, Google Cloud Storage, and more.
• Data Architecture and Modeling: Contribute to the design and implementation of robust data models and architectures for data lakes and data warehouses, ensuring data consistency, integrity, and accessibility for various use cases.
• Data Quality Assurance: Implement rigorous data quality checks and validation procedures throughout the data pipeline to maintain high accuracy and reliability. Adhere to data governance policies and best practices.
• Collaboration and Communication: Work closely with data scientists, analysts, business intelligence specialists, and business stakeholders to understand their data requirements and provide technical expertise on Databricks and data engineering best practices

Skills

• Expertise in Databricks Lakehouse Platform ,ELT With Spark SQL and Python, Incremental Data Processing, Production Pipelines, Data Governance, Auto Loader, lakeflow-connect
• Expertise in the programming languages used for statistical modeling and analysis, data warehousing and Cloud solutions, and building data pipelines.
• Proficient in developing notebooks in Data bricks using Python and Spark and Spark SQL.
• Strong understanding of a cloud services platform (e.g., GCP, or AZURE, or AWS) and all the data life cycle stages. Azure is preferred.
• Proficient in using Azure Data Factory and other Azure Datalake, Serverless.
• Preferred to have knowledge of Delta lake, Lakehouse and Unity Catalog concepts.
• Strong understanding of cloud-based data lake systems and data warehousing solutions.
• Strong understanding of the data interconnections between organizations’ operational and business functions.
• Strong understanding of the data life cycle stages - data collection, transformation, analysis, storing the data securely, providing data accessibility
• Strong understanding of the data environment to ensure that it can scale for the following demands: Throughput of data, increasing data pipeline throughput, analyzing large amounts of data, Real-time predictions, insights and customer feedback, data security, data regulations, and compliance.
• Strong knowledge of algorithms and data structures, as well as data filtering and data optimization.
• Strong understanding of analytic reporting technologies and environments (e.g., Power BI, Looker, Qlik, etc.)
• DeltaLake Unity Catalog, R, Scala, Azure Logic Apps, Cloud Services Platform (e.g., GCP, or AZURE, or AWS)
• Good understanding of concepts and technologies underpinning modern data stack (storage, compute, governance, MLOps, etc.) with focus on Databricks platform
• Experience in data modelling and designing/implementing scalable and efficient ETL/ELT pipelines on Databricks platform
• Knowledge of SQL, Python and Spark
• Understanding of ML/AI and MLOps experience on Databricks platform (MLflow)
• Understanding and experience in solving data governance challenges utilizing Unity catalog (or related technology) in the context of Databricks platform
• Experience with optimizing all aspects of Databricks workloads
• Design, develop, and maintain scalable data pipelines and ETL processes using Azure Databricks, Data Factory, and other Azure services.
• Implement and optimize Spark jobs, data transformations, and data processing workflows in Databricks.
• Develop and integrate custom machine learning models using Azure Machine Learning, MLflow, and other relevant libraries.
• Leverage Azure DevOps and CI/CD best practices to automate the deployment and management of data pipelines and infrastructure.
• Conducting troubleshooting on data models.