Big Data Engineer
Core
Design, develop, and deploy big data and analytics IT solutions, focusing on high-volume batch and near real-time data processing pipelines.
Role type
Senior Big Data Engineer
Builds
High-volume batch and (near) real-time data processing pipelines, data warehousing solutions, and analytical platforms.
Domain
Big Data, Data Warehousing, Analytics
Deliverable
production ML models | product features | infrastructure
Required skills
Data engineering, Data warehousing, Data ingestion (Sqoop, Flume), Big Data processing (Spark/Scala), Data storage (HDFS, HBase, Hive), ETL development, SQL optimization, Streaming frameworks (Kafka, Spark Streaming, Storm), Data profiling, Requirements engineering, Solution architecture
Preferred skills
Building Spark SQL and Spark DataFrame API-based applications, Managing structured and unstructured data types
Technologies
Sqoop, Flume, Spark, Scala, HDFS, HBase, Hive, Kafka, Spark Streaming, Storm
Responsibilities
Integrate data into analytical platforms, Develop and optimize ETL processes, Architect and deploy big data solutions, Manage structured and unstructured data types, Implement databases and data access middleware
Seniority
Senior, hands-on IC
