Senior Data Engineer
Core
Build and maintain optimized, highly available data pipelines to facilitate deeper analysis and reporting for AI and algorithm teams.
Role type
Senior Data Engineer
Builds
Data pipelines, data search capabilities, and access APIs across edge computing and cloud environments.
Domain
Infrastructure monitoring, optical fibers, AI/ML data infrastructure
Deliverable
production ML models
Required skills
Data pipeline orchestration (Apache Airflow, Prefect), Event-driven architectures (Kafka, RabbitMQ), SQL and NoSQL databases (MySQL, Elasticsearch, MongoDB, PostgreSQL, ClickHouse), Python, Version control (Git), Containerization (Docker), CI/CD pipelines
Preferred skills
Streaming technologies, Distributed computing (Hadoop, MapReduce, Apache Spark), Kubernetes, Data visualization (Grafana, ELK Stack, Kibana)
Responsibilities
Design and implement data collection processes for various data types; Develop and implement data search capabilities and access APIs; Design, develop, and optimize data and metadata storage solutions; Analyze existing data schemas and processes to identify and implement optimization solutions; Monitor and analyze cloud data usage to optimize storage and operational costs; Collaborate with Software, AI, and Algorithm teams on system integration and data accessibility.
Seniority
Senior, hands-on IC
