AI Data Engineer - AI Training data
Core
Building and maintaining Python-based data processing pipelines to create high-quality, validated training datasets for AI models from raw structured and unstructured data.
Role type
Senior IC AI Data Engineer
Builds
Large-scale AI training datasets and data processing workflows
Domain
AI/ML data engineering, Cloud (Azure)
Deliverable
production ML models
Required skills
Python, Azure Blob Storage, Azure Data Lake Storage, pandas, NumPy, SQL, Git, Linux, distributed processing, data quality validation
Preferred skills
OCR pipelines, Azure Databricks, Hugging Face Datasets, PySpark, dataset lineage, CI/CD
Technologies
Python, Azure, pandas, NumPy, JSON/JSONL, CSV, Parquet, PySpark, Azure Databricks, Azure Machine Learning, Azure Data Factory
Responsibilities
Design and optimize Python data pipelines for large-scale ingestion and transformation; Build datasets from scratch via cleaning, validation, and preparation; Develop OCR and text processing workflows; Monitor batch jobs and implement recovery mechanisms; Create automated data quality checks; Manage bulk data synchronization with Azure storage; Optimize cloud resource consumption and performance.
Seniority
Senior, hands-on IC
