Lead Software Engineer- Data Engineer
Core
Lead the development of Caterpillar's next-generation Digital Manufacturing Data Platform for large-scale data ingestion, transformation, and analytics across manufacturing, supply chain, and engineering ecosystems.
Role type
Senior IC Data Engineering Lead (Platform & Architecture)
Builds
Scalable AWS-based data platforms, Snowflake data warehouses, and real-time ingestion pipelines for manufacturing data.
Domain
Manufacturing / Industrial IoT / Cloud Data Engineering
Required skills
Data engineering, AWS data ecosystem (S3, Glue, Lambda, EMR, Redshift), Snowflake architecture and optimization, Python, SQL, data ingestion pipelines (ETL/ELT), distributed systems, CI/CD, Agile leadership, stakeholder collaboration, AI/ML pipeline integration.
Preferred skills
Real-time streaming (Kafka, Spark Streaming), Graph databases (Neo4j, Neptune), Vector databases (Milvus, OpenSearch), NVIDIA ecosystem (RAPIDS), GenAI workflows.
Technologies
AWS (S3, Glue, Lambda, EMR, Redshift), Snowflake, Python, SQL, Kafka, Spark Streaming, Jenkins, GitHub, Azure DevOps, Neo4j, Milvus, OpenSearch, cuDF, cuML, cuGraph.
Responsibilities
Lead and mentor a team of data engineers; Architect scalable and secure data platforms on AWS; Design robust data ingestion frameworks for batch and near real-time pipelines; Enforce software engineering principles and CI/CD standards; Partner with product managers and SMEs to translate business needs into scalable data solutions.
Seniority
Senior, hands-on IC with leadership responsibilities
