Spark Data Engineer
Core
Building and managing data pipelines using Apache Spark and Python for national security projects.
Role type
Senior Spark Data Engineer
Builds
Data pipelines processing various file formats (csv, xml, json, avro, parquet) on AWS
Domain
National Security / Defense
Deliverable
production ML models | infrastructure
Required skills
Apache Spark, Python, AWS, SQL (RDBMS), Linux, shell scripting, unit testing
Preferred skills
NiFi, Apache Airflow, Java, Scala, EMR/Spark cluster administration, Spark performance tuning, cloud security solutions, configuration management
Technologies
Apache Spark, Python, AWS, Postgres, Oracle, MySQL, Linux, NiFi, Apache Airflow, EMR, Java, Scala
Responsibilities
Building and managing data pipelines, processing data in multiple formats, writing and validating unit tests, administering Spark clusters, tuning Spark job performance
Seniority
Senior, hands-on IC