Data Engineer (Credit Bureau)
Core
Design and build a metadata-driven, event-based distributed data processing platform for a new Consumer Credit Bureau, managing over 100 million objects with high throughput and sub-500ms latency.
Role type
Senior IC Data Engineer (Cloud-Native)
Builds
Business-critical data pipelines combining batch and streaming approaches, highly available low-latency APIs, and a sophisticated matching algorithm for data updates.
Domain
Financial Services / Consumer Credit Reporting / Cloud Infrastructure
Deliverable
production ML models
Required skills
Python, Apache Spark, AWS (S3, Iceberg, Aurora, DynamoDB, Lambda, Fargate, ECS, Airflow, Kinesis, Glue), Data Pipeline Architecture, Event-Driven Design, Metadata Management, Distributed Systems, API Development, Code Review, Technical Mentoring
Preferred skills
AI tools (Perplexity, Cursor, CoPilot), Whiteboarding, Collaborative Decision Making
Technologies
Python, Spark, Airflow, Iceberg, DynamoDB, AWS Glue, S3, Aurora, Lambda, Fargate, ECS, Kinesis
Responsibilities
Design and build metadata-driven, event-based distributed data processing platforms; Build and scale data pipelines combining batch and streaming approaches; Develop highly available and low latency APIs; Contribute actively to codebase and participate in peer reviews; Drive development from refinement sessions to post-deployment observability; Employ AI tools to aid in all aspects of work.
Seniority
Senior, hands-on IC with mentorship responsibilities
