Senior Data Engineer

We are DIGIPAL, a software delivery agency specialising in building high-performing product design and software development teams for clients across Europe and North America. We work with a wide range of organisations from ambitious startups to multinational corporations. We have a unique career opportunity for you to work on building exciting digital products for international clients.

WHO WE ARE LOOKING FOR:

We are looking for a Senior Data Engineer with strong software engineering skills who is passionate about working with large-scale sensor data and autonomous vehicle technologies, comfortable working with modern data platforms such as Databricks, and capable of turning raw sensor data into high-quality datasets used for simulation and Machine Learning pipelines.

WHAT YOU WILL DO:

• Analyze large volumes of real-world sensor data from autonomous vehicle fleets to detect edge cases and unusual driving scenarios.
• Develop complex SQL, Python, and PySpark queries to filter, aggregate, and transform raw sensor data.
• Identify and curate datasets suitable for AV simulations and ML training pipelines.
• Build and maintain data processing pipelines and ETL workflows for large-scale sensor datasets.
• Work with time-series data from cameras, LiDAR, and radar sensors, processing datasets that can reach ~1 TB per hour.
• Use internal tools and browser-based platforms to support:
• Data discovery
• Dataset search
• Auto-labeling workflows
• Process structured and semi-structured data (e.g., object detection outputs, arrays of structs).
• Design and implement data-mining techniques to discover meaningful driving scenarios.
• Collaborate with engineering teams and researchers to improve data quality, coverage, and usability.
• Contribute to the development and improvement of internal data analytics tools and workflows.

WHAT WE EXPECT FROM YOU:

• 5+ years of experience in Data Engineering, with a background in software engineering
• Advanced SQL skills, with the ability to write complex analytical queries.
• Advanced Python programming skills.
• Hands-on experience with Databricks or similar big data platforms.
• Strong experience with Spark / PySpark for large-scale data processing.
• Experience working with large-scale data pipelines and ETL processes.
• Familiarity with machine learning workflows, particularly in preparing datasets for training.
• Solid understanding of advanced data analytics techniques.
• Experience working with time-series data.
• Experience or interest in Autonomous Vehicles (AV), sensor data, or robotics is a strong advantage.
• Ability to work in a cross-functional, international environment with distributed teams.

OUR OFFER:

• Remote work setup
• Long term B2B Contract
• Attractive compensation package
• Opportunities for professional growth and development
• Collaborate with top minds from around the globe
• Build world-class digital products for European and US markets