Lead Data Engineer

Lead Data Engineer

Department: Product Architecture

Reports to: Manager, Data Engineering

The Lead Data Engineer will drive the evolution of our data platform toward a data mesh architecture. In this role, the lead engineer will lead the development of domain-oriented data products exposed via GraphQL APIs, and enable near real-time reporting through AWS DMS replication to Aurora PostgreSQL. The engineer will also be responsible for building and optimizing ETL/ELT pipelines using AWS Glue and PySpark, ensuring scalable, high-performance data processing across our platform.

Responsibilities:

• Design and implement a data mesh architecture using GraphQL APIs to expose domain-owned data products.
• Build and manage a modern AWS-based data lake using S3, Glue, Lake Formation, Athena, and Redshift.
• Develop and optimize ETL/ELT pipelines using AWS Glue and PySpark to support batch and streaming data workloads.
• Implement AWS DMS pipelines to replicate data into Aurora PostgreSQL for near real-time analytics and reporting.
• Define and enforce standards for data governance, quality, observability, and API design.
• Lead and mentor a team of data engineers, fostering a culture of innovation and ownership.
• Partner with product, engineering, and analytics teams to deliver robust, reusable data solutions.
• Drive automation and CI/CD practices for data infrastructure and pipelines.
• Stay current with emerging technologies and industry trends to continuously evolve the platform.

Requirements:

• Bachelor’s degree in a technical field such as Computer Science or Mathematics
• At least 7 years of experience in data engineering, with at least 2 years in a lead or senior role.
• Experience working with GraphQL APIs for data access.
• Deep expertise in AWS Glue and PySpark for scalable ETL/ELT development.
• Strong experience with AWS data lake technologies (S3, Glue, Lake Formation, Athena, Redshift).
• Hands-on experience with AWS DMS and Aurora PostgreSQL for real-time data replication.
• Solid understanding of data mesh principles and decentralized data architecture.
• Proficiency in Python, SQL, and infrastructure-as-code tools (e.g., Terraform, CloudFormation).
• Experience with data modeling, orchestration tools (e.g., Airflow), and CI/CD pipelines.
• Excellent communication and leadership skills.

Preferred Qualifications:

• Master’s degree, especially with a focus on data engineering, distributed systems, or cloud architecture.
• Experience with event-driven architectures (e.g., Kafka, Kinesis).
• Familiarity with data cataloging and metadata management tools.
• Knowledge of data privacy and compliance standards (e.g., GDPR, HIPAA).
• Background in agile development and DevOps practices.