As a lead data engineer you will be leading the data engineering efforts in a product team. You will work together with product/solution architecture to provide technical necessities to design and develop end-to-end data ingestion pipelines and well tested and monitored data services. You will assess the technical dependency between different functional components and define a resolution. You will also provide technical guide and coach to the junior/medior data engineers in the team, set technical standards and best practices.

?Your responsibilities:

• You will be designing, developing, testing, documenting the data collection framework. The data collection consists of (complex) data pipelines from (IoT) sensors and low/high level control components to our Data Science platform.
• You will build monitoring solution of data pipeline which enables data quality improvement.
• You will develop scalable data pipelines to transform and aggregate data for business use, following software engineering best practices. For these data pipelines you will make use of the best frameworks available for data processing like Spark and Splunk.
• You develop our data services for customer sites towards a product, using (test & deployment) automation, componentization, templates and standardization in order to reduce delivery time of our projects for customers. The product provides insights in the performance of our material handling systems at customers all around the globe.
• You design and build a CI/CD pipeline, including (integration) test automation for data pipelines. In this process you strive for an ever-increasing degree of automation.
• You will work with infrastructure engineer to extend storage capabilities and types of data collection (e.g. streaming)
• You have experience in developing APIs.
• You will coach and train the junior data engineer with the state of art big data technologies.