Big Data Engineer with PySpark
Core
Design, develop, and maintain scalable Big Data pipelines and data-processing solutions using Apache Spark and PySpark.
Role type
Senior Big Data Engineer
Builds
Batch and real-time data pipelines, ETL/ELT processes, and streaming applications
Domain
Big Data, Cloud Data Platforms
Deliverable
production ML models
Required skills
Apache Spark, PySpark, Python, Scala, SQL, Hadoop, HDFS, Hive, MapReduce, Apache Kafka, ETL/ELT pipeline development, distributed computing, Git, CI/CD
Preferred skills
AWS (S3, EMR, Glue, Redshift, Lambda), Azure (ADLS, Databricks, Azure Data Factory, Synapse), GCP (BigQuery, Dataproc, Cloud Storage), Databricks, Snowflake, Apache Airflow, Docker, Kubernetes, data modeling, data governance, data quality frameworks
Technologies
Apache Spark, PySpark, Python, Scala, SQL, Hadoop, HDFS, Hive, MapReduce, Apache Kafka, Airflow, AWS, Azure, GCP, Databricks, Snowflake, Docker, Kubernetes, Git, CI/CD
Responsibilities
Design and optimize ETL/ELT pipelines for large-scale processing; Build batch and real-time applications using Apache Spark and PySpark; Write complex SQL queries and optimize processing jobs; Develop real-time streaming pipelines using Kafka and Spark Streaming; Troubleshoot pipeline failures and improve scalability; Collaborate with data architects, data scientists, and engineering teams; Apply data security, governance, and access-control best practices
Seniority
Senior, hands-on IC