Senior Data Engineer

We have an immediate opportunity for "Senior Data Engineer " . Please let me know your interest with your updated word copy of your resume

Job Title - Senior Data Engineer

Location - Doha- Qatar (Onsite)

Duration - 6+ Months

Job Description Databricks Senior Data Engineer

A Databricks Data Engineer would design, build, and maintain data pipelines and workflows using Databricks, a cloud-based platform for big data processing. Will collaborate with data scientists and analysts to understand data needs and ensure data quality, security, and compliance. Is responsible for data integration, transformation, and loading processes, often leveraging tools like PySpark and SQL.

Key Responsibilities of a Databricks Data Engineer:

Provides the implementation strategy and executes backup, recovery, and technology solutions to perform analysis.

Performs ETL tool capabilities with the ability to pull data from various sources and perform a load of the transformed data into a database or business intelligence platform.

Codes using programming language used for statistical analysis and modeling such as Python/Spark

Data Pipeline Engineering: Design, develop, and deploy robust and scalable data pipelines using Databricks, incorporating data extraction from diverse sources (databases, APIs, streaming platforms), transformation and cleansing using Spark, and loading into target systems (data lakes, data warehouses, etc.). Optimize pipeline performance for efficiency and cost-effectiveness.

Databricks Ecosystem Expertise: Utilize the full capabilities of the Databricks platform, including Databricks SQL, Delta Lake, Databricks Runtime, and Databricks Workflows, to orchestrate complex data workflows and ensure data quality and pipeline reliability.

Cloud Integration: Seamlessly integrate Databricks with cloud services (Azure, AWS, or GCP) for storage, compute, and security, leveraging services like Azure Data Lake Storage, AWS S3, Google Cloud Storage, and more.

Skills

Expertise in Databricks Lakehouse Platform ,ELT With Spark SQL and Python, Incremental Data Processing, Production Pipelines, Data Governance, Auto Loader, lakeflow-connect

Expertise in the programming languages used for statistical modeling and analysis, data warehousing and Cloud solutions, and building data pipelines.

Proficient in developing notebooks in Data bricks using Python and Spark and Spark SQL.

Strong understanding of a cloud services platform (e.g., GCP, or AZURE, or AWS) and all the data life cycle stages. Azure is preferred.

DeltaLake Unity Catalog, R, Scala, Azure Logic Apps, Cloud Services Platform (e.g., GCP, or AZURE, or AWS)

Good understanding of concepts and technologies underpinning modern data stack (storage, compute, governance, MLOps, etc.) with focus on Databricks platform

Experience in data modelling and designing/implementing scalable and efficient ETL/ELT pipelines on Databricks platform

Knowledge of SQL, Python and Spark

Interested candidates send me your CV along with below details:

Expected Salary:

Visa/Work permit:

Notice period:

Current location: