Data Engineer
Core
Designing and developing data pipelines for market data processing and anomaly detection on a modern cloud-based Big Data platform.
Role type
Senior Data Engineer (Python/Big Data)
Builds
Data ingestion, transformation, and analysis pipelines for financial market data
Domain
Financial services / Big Data / Cloud
Deliverable
production ML models | product features
Required skills
Python, PySpark, AWS (S3, Glue, Lambda, Step Functions), Big Data architecture, Data Lake design, SQL, CI/CD, Data Quality frameworks
Preferred skills
Data visualization, API development, Financial market data experience
Technologies
Python, PySpark, AWS, Great Expectations, Git
Responsibilities
Design and develop data pipelines in Python (PySpark), ingest and normalize large-scale market data, implement data lake/medallion architectures, optimize processing performance and cloud costs, manage data quality, expose data to business consumers, improve frameworks (monitoring, CI/CD, tests), write technical documentation
Seniority
Senior, hands-on IC
