Data Engineer
Core
Build and maintain large-scale data pipelines for a Federal Government client, focusing on ingestion, curation, and extraction of data sets.
Role type
Data Engineer
Builds
Data pipelines and data lake integrations for government clients
Domain
Big data processing and analytics
Deliverable
production ML models | product features
Required skills
Spark, Kafka, Scala, Python, Java, Hadoop HDFS, Amazon S3, data modeling, schema design, Apache Airflow, Git
Preferred skills
Monitoring and logging tools, workflow management
Technologies
Apache Spark, Apache Kafka, Hadoop HDFS, Amazon S3, Apache Airflow, Git
Responsibilities
Maintain and develop Spark pipelines to process large-scale data sets; Ingest tables into data lake from Kafka using batch or real-time processing; Run Spark jobs for ingestion, curation, and extraction; Alter schemas and configs to meet business requirements; Troubleshoot and resolve bugs in data processing pipelines; Monitor performance and troubleshoot issues using logging and monitoring tools; Document best practices and develop standards for data processing pipelines.
Seniority
Mid-level, hands-on IC