Data Engineer
Core
Design, build, and optimize scalable data pipelines and architectures to support big data platforms, analytics, and machine learning initiatives.
Role type
Senior Data Engineer (Big Data & Cloud)
Builds
Large-scale data pipelines, ETL packages, data warehouses, and big data platforms (Hadoop, Spark, Kafka).
Domain
Big Data, Cloud Infrastructure, Data Engineering
Required skills
Advanced SQL, Big Data tools (Hadoop, Spark, Kafka), ETL development (Abinitio), Cloud services (AWS EC2, EMR, RDS, Redshift), Python/Java/Scala/C++, NoSQL databases (Postgres, Cassandra), Data pipeline orchestration (Airflow, Azkaban, Luigi), Data migration, Performance tuning, Root cause analysis.
Preferred skills
Stream processing (Storm, Spark-Streaming), Object-oriented scripting, Internet technologies (SaaS, IaaS, PaaS), Data governance expertise.
Responsibilities
Create and maintain optimal data pipeline architecture; Design and implement scalable ETL packages; Oversee large-scale Hadoop platforms; Enable and run data migrations across databases; Develop scripts for database maintenance and performance tuning; Support data scientists with data preparation for ML models; Analyze complex data systems to recommend improvements; Troubleshoot data issues and present solutions.
Seniority
Senior, hands-on IC
