Senior Data Engineer

JD

Responsibilities

• Collaborate with several stakeholders to deeply understand the needs of data practitioners to deliver at scale.
• Data Engineers help to define, build and maintain Data Platform.
• Work on emerging technologies, building distributed applications.
• Development effort End-to-End for on-time delivery of high-quality solutions that conform to requirements, conform to the architectural vision, and comply with all applicable standards.
• Present technical solutions, capabilities, considerations, and features in business terms.
• Help to further develop critical initiatives, such as Data Discovery, Data Lineage and Data Quality.
• Build data systems, pipelines, analytical tools and programs.
• Conduct complex data analysis and report on results.

Skills and Attributes

• Strong technical expertise in data modeling, data ingestion, distributed processing, data warehousing, and ETL processes.
• Experience working with Apache Kafka and Spark Streaming.
• Exposure to connecting and querying different data sources, including NoSQL databases (e.g., MongoDB), MySQL, Elasticsearch, and Kafka topics.
• Proficiency in SQL, Python, PySpark, Spark SQL and a solid understanding of the Databricks platform.
• Ability to build large-scale batch and real-time data pipelines.
• Knowledge of best practices for optimizing Databricks cluster configurations (size, type, memory) based on data requirements and delivering optimized code.
• Experience in maintaining, monitoring, and handling enhancement requests for data pipelines.
• Familiarity with visualization tools such as Kibana and QlikView.
• Exposure to Docker, Kubernetes, and GitLab.
• Experience with JIRA processes and Confluence for documentation and project management.

Qualification & Requirements

• 8+ Years of Experience.
• Degree in Computer Science, Data Science, Mathematics, IT, or a related field.
• 5 years of Experience as a data engineer or similar role.