Senior Data Engineer
Core
Architect and build AI-native data pipelines, semantic layers, and agent retrieval infrastructure to support knowledge and cognitive agents in real-time.
Role type
Senior IC data engineer (AI-native systems)
Builds
End-to-end ingestion, transformation, and serving pipelines for agentic consumption; vector stores and embedding pipelines; KPI calculation services.
Domain
Consumer goods / AI-native data platforms
Deliverable
production ML models | infrastructure
Required skills
Python (PySpark), SQL, Big Data architecture (Lakehouse, Data Mesh), vector databases, RAG pipeline design, LLM orchestration frameworks, CI/CD, data governance
Preferred skills
Azure ecosystem, semantic layer design, agent orchestration tooling
Technologies
Spark, Databricks, Azure AI Search, pgvector, FAISS, LangChain, LangGraph, LlamaIndex, Airflow, Azure Data Factory
Responsibilities
Design end-to-end ingestion, transformation, and serving pipelines optimized for agentic consumption; standardize and contextualize data for semantic layer design; encode KPI calculations in reusable back-end services; define data acquisition strategies for AI agents; implement vector stores and RAG patterns at production scale; design permissioning and audit logging for AI data access; lead full application lifecycle on Azure; enforce engineering standards for data quality and agent output evaluation.
Seniority
Senior, hands-on IC