Data Engineer

Job Summary:

We are seeking a highly skilled Data Engineer to design, develop, and maintain scalable ETL/ELT pipelines and data infrastructure solutions. In this role, you will work with cross-functional teams to process structured and unstructured data, ensure data quality, and optimize performance for large datasets. You will be an integral part of building data solutions that drive critical business decisions in the energy domain.

Roles and Responsibilities:

• ETL/ELT Pipeline Development:
• Design, develop, and maintain scalable ETL/ELT pipelines to process structured and unstructured data efficiently.
• Data Integration:
• Collaborate with business teams to understand data sources and design effective data integration strategies that meet business needs.
• Performance Optimization:
• Optimize database performance, ensuring efficient query execution and minimal processing time for large datasets.
• Data Quality & Validation:
• Implement and maintain data quality frameworks, performing data validation processes to ensure the accuracy, consistency, and reliability of data.
• Automation and Transformation:
• Develop and maintain data transformation scripts and automated workflows to enhance data processing efficiency.
• Data Infrastructure Deployment & Monitoring:
• Support the deployment, management, and monitoring of data infrastructure and cloud-based storage solutions, ensuring high availability and reliability.
• Collaboration with Data Architects:
• Work closely with the Data Architect to establish and enforce best practices for data modeling, warehousing, and system design.
• Continuous Improvement:
• Stay up-to-date with the latest technologies and trends in data engineering, bringing innovative solutions to improve our data pipelines and infrastructure.

Qualification and Experience:

• Education:
• BE/BTech or Science graduates in Computer Science, Information Management, or a related field.
• Experience:
• Minimum of 7 years of hands-on experience as a Data Engineer, with specific experience in the energy domain or related industries.
• Technical Skills:
• Proficiency in SQL and database technologies (e.g., PostgreSQL, MySQL, Redshift, Snowflake).
• Hands-on experience with ETL/ELT tools (e.g., Apache NiFi, Talend, Informatica, Airflow).
• Strong knowledge of cloud platforms (AWS, Azure, GCP).
• Experience with programming languages like Python, Java, or Scala.
• Familiarity with big data technologies (Hadoop, Spark, Kafka, etc.) is a plus.
• Other Skills:
• Strong problem-solving skills and attention to detail.
• Ability to work effectively in a collaborative, cross-functional team environment.
• Excellent communication skills, both written and verbal.