Big Data Engineer

We are seeking a big data engineer for one of our clients. The successful candidate will be responsible for overseeing the creation and maintenance of our database infrastructure, including collecting and maintaining data, ensuring the integrity of our data, and creating and training data models.

Responsibilities

Below are some of the responsibilities of a big data engineer:

• Design the architecture of our big data platform
• Build lakehouse PoC and make lakehouse technology decisions (based on our requirements)
• Build a scalable and performant data model satisfying our existing use cases and future plans using object storage (S3-compatible)
• Build code architecture and deployment infrastructure for data pipelines development using orchestrator (Airflow) and Spark
• Prepare documentation, knowledge transfer sessions and run books for data lake and data platform
• Mentor engineering team during work process
• Maintain our data pipeline and data lake platform
• Customize and oversee integration tools, warehouses, databases, and analytical systems
• Configure and provide availability for data-access tools

Job Qualifications and Skill Sets

Below are the qualifications expected of a big data engineer:

• Minimum 5 years of relevant data engineering experience
• Experience designing data model which utilizes object (S3-like) storage and used for massive parallel processing using Spark
• Experience running lake house technologies (DeltaLake, Iceberg) in production on big datasets (TB of data)
• Experience with pipeline orchestrators (Airflow/Dagster).
• Experience writing and operating Spark pipelines at scale (hundreds of cores, TBs or RAM)
• Experience with JVM based languages (Kotlin would be a plus)
• Experience with Python
• Excellent command of English both verbal and written

EEO Statement

Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.