Data Engineer

Duration: 8 Months

Job Summary:

We are seeking a motivated and skilled Data Engineer with expertise in Databricks to join our Next Gen MES team in Limerick, Ireland. The ideal candidate will have hands-on experience in building and optimizing data pipelines, ensuring data quality, and leveraging Databricks to facilitate data integration and analytics. This role requires a strong understanding of ETL processes, data architecture, and a commitment to delivering high-quality data solutions.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.

• Collaborate with data analysts and data scientists to understand data needs and provide appropriate data solutions.

• Optimize existing ETL processes to enhance performance and efficiency.

• Ensure data quality by implementing validation and testing procedures.

• Work with Big Data technologies and frameworks to manage large datasets.

• Assist in data modeling and design of data warehouses to support business intelligence initiatives.

• Monitor and troubleshoot data pipeline performance issues.

• Participate in code reviews and contribute to best practices and standards in data engineering.

• Document data processes, architecture, and workflows for knowledge sharing across the team.

• Stay updated with industry trends and emerging technologies related to data engineering.

• Drive innovation in this area of data engineering.

• Strong communication and collaboration skills to work effectively in a team environment.

Qualifications:

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field.

• Experience with cloud platforms, ideally Azure, AWS also suitable.

• 3+ years of experience in data engineering or a related role.

• Strong proficiency in Databricks and Apache Spark for processing large datasets.

• Proficient in SQL and scripting languages (e.g., Python, Scala).

• Familiarity with data modeling concepts, ETL tools, and data warehousing solutions.

• Knowledge of data governance and data quality principles.

• Excellent problem-solving skills and attention to detail.

Preferred Qualifications:

• Experience with machine learning concepts and data science frameworks.

• Understanding of modern data architectures and data lake concepts.

• Familiarity with version control systems (e.g., Git) and CI/CD pipelines.