Data Engineer
Core
Design, build, and maintain scalable Python/Scala applications and high-performance Spark clusters for Big Data processing.
Role type
Intermediate IC data engineer (Big Data)
Builds
Reusable, reliable application systems and low-latency data processing components
Domain
Big Data, Distributed Systems, Cloud
Deliverable
production ML models | product features
Required skills
Python, Scala, Apache Spark, HDFS, MapReduce, YARN, Hive, SQL, UNIX/Linux shell scripting, Kafka, NoSQL databases, Git, Jenkins, Maven
Preferred skills
Agile/Scrum, Unit Testing, SDLC, Cloud/container technologies, Real-time data processing
Technologies
Python, Scala, Spark, HDFS, MapReduce, YARN, Hive, Kafka, HBase, Impala, Avro, Parquet, JSON, CSV, Git, Jenkins, Maven
Responsibilities
Design and build efficient Python/Scala code; Develop high-performance Spark clusters; Process data using Hive, Impala, and HBase; Identify and resolve application bottlenecks; Translate use cases into functional applications; Perform testing of software prototypes.
Seniority
Intermediate, hands-on IC
