Data Engineer
Core
Designing and developing a data platform to fulfill business needs, including building data pipelines, maintaining real-time data processing applications, and ensuring data quality.
Role type
Data Engineer
Builds
Data pipelines, real-time data processing applications, and data visualizations
Domain
Cloud data engineering and analytics
Deliverable
production ML models
Required skills
Snowflake, Apache Airflow, AWS Glue, Apache Iceberg, SQL, Python or Scala, AWS services (S3, Lambda, API Gateways, DMS, RDS), Linux/Cloud development
Preferred skills
Data warehousing concepts (Inmon, Kimball, Data Vault), AWS MSK and EMR, Docker, Kubernetes, Elasticsearch, Kafka, Git, Terraform, Prometheus, Grafana, CloudWatch, PowerShell, Unix scripting
Technologies
Snowflake, Apache Airflow, AWS Glue, Apache Iceberg, Kafka, Spark Structured Streaming, Kafka Streams, AWS MSK Connect, AWS S3, AWS Lambda, AWS API Gateways, AWS DMS, AWS RDS, Elasticsearch, Docker, Kubernetes, Terraform, Prometheus, Grafana, CloudWatch
Responsibilities
Design and develop data platforms; maintain and evolve existing data platforms; build processes to ingest data from Kafka, APIs, and databases; develop and maintain real-time data processing applications; apply transformations to data streams; perform data modeling; ensure data quality; troubleshoot incidents; document processes and share knowledge
Seniority
Mid-level, hands-on IC
