Software Engineer
Core
Designing, implementing, and optimizing data ingress/egress pathways for scalable analytics workloads using distributed processing.
Role type
Senior Software Engineer (Data Engineering)
Builds
Efficient, scalable data processing pipelines and analytics workloads
Domain
Big Data / Distributed Systems
Deliverable
production ML models
Required skills
Apache Spark, Python, Java, Apache Airflow, SQL (MySQL, PostgreSQL), Linux CLI, Bash scripting, Git, Data wrangling (pandas, NumPy)
Preferred skills
HPC Job Scheduling (Slurm), Atlassian Tool Suite (JIRA, Confluence)
Technologies
Apache Spark, Apache Airflow, Python, Java, MySQL, MariaDB, PostgreSQL, Parquet, JSON, CSV, XML, Git, Slurm, JIRA, Confluence, Jupyter Notebook
Responsibilities
Design dataflow architectures, implement data transport mechanisms, optimize data processing pipelines, orchestrate complex workflows, query and aggregate large datasets
Seniority
Senior, hands-on IC