Data Engineer – Unified Data Layer (UDL) (f/m/div.)
Core
Build resilient data pipelines connecting on-premises laboratory equipment, test benches, and edge systems to Microsoft Azure, parsing complex proprietary binary formats and transforming raw data into optimized Lakehouse tables.
Role type
Senior Data Engineer (Scientific Data & IoT)
Builds
Automated data ingestion pipelines, normalized Lakehouse tables (Delta Lake/Parquet), and data connectors for scientific instruments.
Domain
Industrial IoT / Scientific Research / Manufacturing
Deliverable
production ML models
Required skills
Python (binary data handling, NumPy, Pandas, PySpark), Bash, Apache Spark (Databricks), SQL, Terraform, Git, Linux, REST APIs, Kafka/Event Hubs
Preferred skills
HDF5, NetCDF, Protobuf, OPC UA, MQTT, Azure Data Factory, Vectorial databases
Technologies
Azure, Databricks, Delta Lake, Parquet, Kafka, Terraform, Python, Bash, Linux
Responsibilities
Develop and maintain data pipelines for heterogeneous sensor streams and experimental batch runs; Implement parsers for undocumented binary and proprietary file formats; Clean and transform raw data into partitioned Lakehouse tables; Collaborate with scientists to decipher domain-specific data semantics.
Seniority
Senior, hands-on IC