AI DATA ENGINEER
Core
Design and develop large-scale data processing pipelines and apply machine learning techniques to manage structured and unstructured data across fashion, geospatial, industrial, and production fields.
Role type
AI Data Engineer
Builds
End-to-end data pipelines and deployed ML models
Domain
Data Engineering and Artificial Intelligence
Deliverable
production ML models | infrastructure
Required skills
Apache Spark (Scala, Spark SQL), Python, Scala, distributed processing (RDD, DataFrame, MapReduce), Azure Data Factory, ETL pipeline design, machine learning (regression, clustering, anomaly detection, forecasting), model deployment
Preferred skills
data visualization and reporting tools
Technologies
Apache Spark, Scala, Spark SQL, Python, Microsoft Azure, Azure Data Factory
Responsibilities
Design and develop large-scale data processing pipelines; Implement and optimize distributed processing; Develop custom UDFs for data transformations; Implement and manage data ingestion and processing architectures on Azure; Build end-to-end pipelines for structured and unstructured data; Apply machine learning techniques for forecasting and anomaly detection; Support model deployment ensuring scalability and reliability; Create reports and dashboards for communicating results
Seniority
Mid-level, hands-on IC
