Java Spark developer
Core
Design, develop, and maintain high-performance, large-scale data processing pipelines and distributed applications using Java and Apache Spark.
Role type
Senior IC Java Spark developer
Builds
Resilient batch and real-time streaming data architectures, microservices, and distributed processing modules
Domain
Big Data / Distributed Systems
Deliverable
production ML models
Required skills
Core Java (8/11/17+), Apache Spark (RDDs, DataFrames, Structured Streaming), Distributed Ecosystem (HDFS, YARN), Messaging & Streaming (Kafka, RabbitMQ), SQL & NoSQL, Build Tools (Maven, Gradle), Testing (JUnit, Mockito)
Preferred skills
Cloud Platforms (AWS EMR, Azure Databricks, GCP Dataproc), Lakehouse Technologies (Iceberg, Delta Lake, Hudi), Containerization (Docker, Kubernetes), Workflow Schedulers (Airflow, Luigi)
Technologies
Apache Spark, Apache Kafka, Kubernetes, AWS EMR, Azure Databricks, Google Cloud Dataproc, Parquet, ORC, Avro, Hive, Cassandra, HBase, MongoDB, Delta Lake, Snowflake, PostgreSQL, Oracle, MySQL, Jenkins, GitLab CI, GitHub Actions
Responsibilities
Design and implement scalable data ingestion and ETL/ELT pipelines; Profile, debug, and optimize Spark jobs for performance; Design data storage schemas and implement data governance; Configure and maintain CI/CD pipelines
Seniority
Mid-Senior, hands-on IC
