Custom Software Engineer
Core
Design, code, and enhance scalable data pipelines and ETL processes to collect, process, and store large volumes of structured and unstructured data.
Role type
Senior Data Engineer
Builds
Scalable data pipelines, ETL processes, and data solutions for business teams
Domain
Data Engineering / Cloud Data Platforms
Deliverable
production ML models | infrastructure
Required skills
Python, PySpark, Java, Scala, SQL, NoSQL, Apache Airflow, AWS Glue, Google Dataflow, Azure Data Factory, Hadoop, Spark, Kafka, Docker, Kubernetes, Redshift, BigQuery, Snowflake
Preferred skills
Machine learning pipelines, data science workflows, data visualization tools (Tableau, Power BI)
Technologies
AWS, GCP, Azure, Spark, Hadoop, Kafka, Redshift, BigQuery, Snowflake, Airflow, Luigi, Docker, Kubernetes, Tableau, Power BI
Responsibilities
Design, develop, and maintain scalable data pipelines and ETL processes; Optimize data architecture for performance, reliability, and scalability; Implement data quality checks and monitor data integrity; Collaborate with data scientists, analysts, and business teams to understand data requirements; Manage and optimize databases and data warehouses; Develop automation tools for data ingestion, transformation, and integration; Document data flows, architecture, and processes; Ensure compliance with data governance and security policies.
Seniority
Senior, hands-on IC