Data Engineer

Scope

We are seeking a highly experienced Senior Data Engineer to join our client and play a critical role in shaping our data infrastructure and driving innovative solutions. This individual will work closely with a talented Data Science team and contribute to cutting-edge AI projects. The ideal candidate is an expert in building and optimizing data pipelines, designing scalable architectures, and ensuring seamless data flow to support complex analytics and AI initiatives

Responsibilities

Data Architecture and Pipelines:

• Design, implement, and optimize robust data pipelines to handle large-scale data ingestion, transformation, and storage.
• Work with real-time and batch data processing frameworks such as Apache Spark, Apache Kafka, and Apache NiFi.

Collaboration with Data Science and AI Teams:

• Partner with data scientists and AI engineers to build efficient data workflows and deliver high-quality data for model development and analytics.
• Develop infrastructure to support machine learning and AI model deployment.

Data Integration and Storage:

• Manage and optimize Data Lakes and data warehouses, ensuring seamless integration and accessibility.
• Leverage Apache Iceberg for scalable, high-performance table formats in large-scale analytics.

IBM Technology Stack:

• Work extensively with IBM products such as IBM DataStage, IBM Db2, IBM Watson, and other relevant tools.
• Integrate and optimize IBM technologies to enhance data management and AI-driven solutions.

Graph and NoSQL Databases:

• Design and maintain data models using Neo4j for graph database applications.
• Manage and query NoSQL databases like MongoDB to support diverse data structures.

Data Governance and Security:

• Ensure compliance with data security standards and best practices.
• Establish frameworks for data quality, metadata management, and data governance.

Performance Optimization:

• Troubleshoot and resolve data-related issues to maintain high system performance and reliability.
• Continuously improve system architecture to enhance scalability and efficiency.

Required skills

• AI Integration: Demonstrated ability to collaborate with Data Science teams on AI and ML projects.
• Programming: Strong skills in Python, Java, or Scala for data processing and pipeline development.
• Cloud Expertise: Familiarity with cloud platforms (AWS, Azure, GCP) is a plus.

Required experience

• Minimum of 7 years in data engineering or a related role, with proven experience in data-intensive environments and AI projects.
• Extensive experience with IBM products (e.g., IBM DataStage, IBM Db2, IBM Watson).
• Strong knowledge of Data Lakes and Apache Iceberg.
• Proficiency with Apache Spark, Apache Kafka, and Apache NiFi for large-scale data processing.
• Expertise in Neo4j and other graph databases.
• Hands-on experience with MongoDB and NoSQL databases.

Required education

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.