Data Engineer - Python, AI
Core
Build NLP pipelines and large-scale data processing systems for AI and analytics use cases.
Role type
mid-level IC data engineer (Python, AI/NLP)
Builds
scalable ingestion and data transformation pipelines, NLP pipelines, and Flask-based APIs for model inference
Domain
AI/ML, NLP, Data Engineering
Deliverable
production ML models
Required skills
Python, PySpark, Pandas, NLP libraries (Flair, BERT, HuggingFace), Flask, MLflow, CI/CD, Redis, Autosys JILs, Linux shell scripting
Preferred skills
Polars, Dask, PyTorch, TensorFlow, Docker, Kubernetes, FastAPI, Airflow, Prefect, AWS boto3
Technologies
PySpark, Pandas, PyArrow, Flair, BERT, LLM frameworks, Flask, MLflow, GitHub, LightSpeed Enterprise, Redis, Autosys JILs, ITRS Geneos, ECS, boto3, PyTest, unittest
Responsibilities
Develop and optimize ETL/data processing jobs; Build and maintain NLP pipelines; Develop scalable ingestion and data transformation pipelines; Build and maintain Flask-based APIs for model inference; Integrate caching and fast lookups using Redis; Manage and deploy ML models using MLflow; Support CI/CD workflows; Create and maintain Autosys JILs for job scheduling; Write unit tests and improve automation test coverage; Monitor application and system health using ITRS Geneos
Seniority
Mid-level, hands-on IC
