Software Engineer, Data Platform

## About the role You will be responsible for developing our core data platform. This includes building scalable data ingestion connectors/pipelines, developing efficient APIs and ensuring the robustness and security of data and cloud based infrastructure. You will build the large scale infrastructure that powers up crawling, ranking and retrieval systems. Having a thorough and accurate understanding of data is at the core of this work. It's cutting-edge software engineering at fore-front of AI. ## Responsibilities - Designing and implementing scalable data ingestion and processing pipelines - Develop high-performance RESTful APIs to facilitate seamless data access. - Architect resilient data storage solutions using modern cloud technologies. - Manage cloud infrastructure (AWS, GCP, or Azure) to support high-volume data processing. - Implement automated monitoring, logging, and alerting systems to ensure high availability, performance and reliability of data pipelines and systems. - Evaluate and implement new technologies, frameworks, and approaches to improve efficiency and outcomes. ## Hard Requirements - 4+ years of experience as a software engineer, with a focus on data platforms or infrastructure. - Expertise in Python and SQL - Mastery of distributed systems and proficiency in NoSQL and SQL tools. - Proven experience deploying and managing mission-critical ETL pipelines with diverse and large-scale data sources. - Experience with orchestration frameworks like Airflow - Proficiency in AWS or other cloud environments, with practical knowledge of their services. - Demonstrated ability to design and monitor secure and scalable systems, especially in production-grade environments. - Experience with containerization (Docker) and orchestration (Kubernetes) to streamline deployments. - Familiarity with LLMs and their potential applications ## Bonus Requirements - Experience in stream processing frameworks and architectures. - Background in financial services or other data-intensive industries. - Hands-on experience with telemetry and monitoring tools.