Software Engineer - Data Engineering
Core
Design and implement end-to-end near real-time data movement solutions and build robust data platforms for Caterpillar's digital ecosystem.
Role type
Senior Data Engineering Software Engineer
Builds
End-to-end near real-time data pipelines, large data lakes, and data warehouses for OLTP & OLAP workloads.
Domain
Industrial technology, cloud data infrastructure, big data
Deliverable
production ML models | product features | dashboards & analysis
Required skills
Python (OOP), SQL/NoSQL, AWS services (Kafka, Kinesis, Lambda, DynamoDB, RDS), CI/CD tools, data pipeline architecture, concurrency management, test-driven development
Preferred skills
Snowflake, Azure DevOps, Jenkins, GoCD, AWS CloudFormation, Gen AI concepts
Technologies
Python, SQL, NoSQL, Kafka, Kinesis, AWS SQS, AWS SNS, Lambda, API Gateway, DynamoDB, Aurora, PostgreSQL, Azure DevOps, Jenkins, GoCD, AWS CloudFormation
Responsibilities
Design and develop complex systems and infrastructure components; maintain high standards of software quality; mentor junior developers; perform integrated and customer acceptance testing; communicate with end users to direct development and debugging.
Seniority
Senior, hands-on IC with mentorship responsibilities
