Lead Big Data Engineer
Core
Designing and developing scalable data processing pipelines and ETL processes to handle large-scale enterprise datasets and support strategic business outcomes.
Role type
Lead Big Data Engineer
Builds
Scalable data processing pipelines, ETL processes, and data-intensive applications for enterprise data management.
Domain
Enterprise data management, cloud analytics, distributed computing
Deliverable
production ML models | product features | infrastructure
Required skills
Distributed computing frameworks, ETL development, Cloud platforms (AWS/Azure/GCP), Modern programming languages (Python/Scala/Java/C#), SQL, Stream processing technologies, Data-intensive application development
Preferred skills
Stream processing technologies (Apache Pulsar/Kinesis), Distributed processing frameworks (Spark/Hadoop), Data integration tools (NiFi/Talend), CI/CD pipelines, Containerization (Docker/Kubernetes)
Technologies
Apache Spark, Databricks, Snowflake, Azure Synapse, AWS, Azure, Google Cloud Platform, Apache Pulsar, Amazon Kinesis, Docker, Kubernetes, OpenShift
Responsibilities
Design and develop scalable data processing pipelines using distributed computing frameworks; Build and maintain ETL processes; Develop data-intensive applications and services; Implement stream processing solutions; Collaborate with cross-functional teams to translate business requirements; Provide production support and troubleshooting for data systems.
Seniority
Senior, hands-on IC with leadership responsibilities
