Software Engineer, Data Core
Core
Design, build, and maintain data processing pipelines and architecture to process hundreds of millions of data sources for a unified utilities map.
Role type
Senior IC data engineering software engineer (via careerplan.io/jobs/B6-00F-AD-C69-software-engineer-data-core-at-4m-analytics-ltd)
Builds
Data processing pipelines, ETL workflows, databases, and geospatial data processing components
Domain
Utilities mapping, geospatial data, big data infrastructure
Deliverable
production ML models
Required skills
Python, big data technologies (Hadoop, Spark, Kafka), cloud platforms (AWS, GCP, Azure), data architecture design, CI/CD implementation, geospatial data processing, scalability optimization
Technologies
Python, Hadoop, Spark, Kafka, AWS, Google Cloud, Azure
Responsibilities
Design and implement data architecture for effective storage and retrieval; Develop and optimize Python-based applications for data processing; Drive scalability and performance optimization of data systems; Create and manage data infrastructure components including ETL workflows; Implement CI/CD processes for data processing and model training; Implement geospatial data processing techniques for the unified utilities map
Seniority
Senior, hands-on IC
