Data Engineer – AWS Kinesis & Databricks - AU Citizen
Core
Build and support real-time data pipelines using Amazon Kinesis and Databricks to move high-volume streaming data into a Lakehouse.
Role type
Senior Data Engineer (Real-time Streaming)
Builds
Real-time data pipelines, streaming solutions, Delta tables
Domain
Cloud Data Engineering (AWS)
Required skills
Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, Python, SQL, Apache Spark Structured Streaming, Databricks, Delta Lake, AWS CloudWatch, AWS IAM, AWS S3
Preferred skills
Kafka
Responsibilities
Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose; Develop streaming solutions in Databricks using Spark Structured Streaming; Ingest and manage streaming data within Delta tables; Optimise pipelines for performance, latency, reliability and cost; Build monitoring and alerting using AWS CloudWatch; Work with batch data engineers to ensure streaming and batch pipelines integrate effectively; Develop clean, testable Python and SQL code; Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.
