Lead - Data Engineering
Core
Architect high-performance data infrastructure and real-time pipelines enabling agentic AI to reason over complex enterprise environments.
Role type
Senior IC AI Data Engineer
Builds
Real-time data pipelines, RAG systems, vector databases, semantic layers, and knowledge graphs for autonomous AI agents.
Domain
Enterprise AI, Data Engineering, Observability
Deliverable
production ML models
Required skills
Python, ELK stack (ElasticSearch, Logstash, Kibana), ETL processes, data mining, relational and NoSQL databases (PostgreSQL, DB2, MongoDB), CI/CD pipelines, vector databases (Pinecone, Milvus, Weaviate)
Preferred skills
Data modeling, social coding (GitHub, Visual Studio), professional data engineering certifications
Technologies
Python, ELK, Pinecone, Milvus, Weaviate, PostgreSQL, DB2, MongoDB, GitHub, Visual Studio
Responsibilities
Design and scale RAG pipelines for unstructured IT logs and documentation; ensure health and performance of vector databases; build automated data guardrails to detect noise, bias, or PII; develop CI/CD pipelines for data infrastructure; resolve data quality issues at the source.
Seniority
Senior, hands-on IC