Quantexa Data Engineer
Core
Design, develop, and optimize big data solutions and search applications using Apache Spark, Scala, and Elasticsearch to support data analytics and machine learning initiatives.
Role type
Senior Data Engineer (Big Data & Search)
Builds
Scalable data processing pipelines, Spark applications, and Elasticsearch search/indexing solutions
Domain
Finance / Compliance (AML) / Big Data
Deliverable
production ML models | product features
Required skills
Apache Spark (RDDs, DataFrames, SQL), Scala, Elasticsearch, Hadoop ecosystem, OpenShift Container Platform (OCP), Kubernetes, CI/CD pipelines, Data governance, Distributed computing
Preferred skills
Quantexa software proficiency, Graphana, Prometheus, Splunk, Cloud platforms (AWS/Azure/GCP)
Technologies
Apache Spark, Scala, Elasticsearch, Hadoop, HDFS, Hive, Pig, OpenShift, Kubernetes, Docker, Jenkins, Ansible, Git, Python, Java
Responsibilities
Design and implement Spark Scala applications and data processing pipelines; Integrate Elasticsearch with Spark for efficient indexing and querying; Optimize Spark jobs and Elasticsearch queries for performance; Develop fault-tolerant applications adhering to best practices; Deploy solutions on OCP using containerization; Implement data governance, lineage, and metadata management; Collaborate with DevOps on CI/CD and platform stability; Troubleshoot data processing and performance issues.
Seniority
Senior, hands-on IC
