Senior Data Engineer South Africa
Core
Designing, building, and optimizing data architecture and pipelines on AWS, Microsoft Fabric, or Databricks to support engineering capabilities across Africa.
Role type
Senior Data Engineer
Builds
Data platforms and pipelines
Domain
Cloud data engineering (AWS, Azure, Databricks)
Deliverable
production ML models | product features | infrastructure
Required skills
Python, SQL, Linux, Git, Terraform, CloudFormation, PySpark, AWS Kinesis, AWS Glue, AWS Redshift, AWS Athena, AWS IAM, AWS VPC, Apache Airflow, AWS Step Functions, Docker, CI/CD, data modeling, ETL, big data technologies (Hadoop, Spark), real-time data processing, data integration patterns, cloud security best practices
Preferred skills
Experience with a second cloud vendor (e.g., Azure), Docker/Containers/CICD pipelines, open-source contributions, machine learning algorithms, enterprise integration patterns, DevOps practices
Technologies
AWS (S3, KDS, KDF, DataSync, StorageGateway, TransferFamily, Snowball, DynamoDB, RDS, MySQL, PostgreSQL, Aurora, Redshift, Elasticsearch, OpenSearch, Redis, Memcached, Lambda, Step Functions, Airflow, Glue, LakeFormation, Athena, KMS, SSE, TLS), Microsoft Fabric, Databricks, Kafka, MSK, Hadoop, Hive, EMR, Parquet, CSV, Avro, JSON
Responsibilities
Develop ETL processes integrating with source and destination systems; develop data pipelines using Python, Spark, and SQL; orchestrate data pipelines; implement data infrastructure using IAC; manage cloud storage and data formats; ensure data quality and security; design and implement data solutions
Seniority
Senior, hands-on IC
