Data / Data Platform Engineer
Core
Design, develop, and optimize high-throughput batch and real-time streaming pipelines for enterprise security scanner findings and vulnerability data.
Role type
Senior Data Platform Engineer (Security Data)
Builds
Modular frameworks for integrating, evaluating, and comparing outputs from new LLM and security scanner models; data pipelines for vulnerability data.
Domain
Cybersecurity / Financial Services
Deliverable
production ML models
Required skills
Python, Go (Golang), PySpark, Apache Hadoop, Apache Hive, Apache Kafka, BigQuery, GCP Data Lake, REST API integration, data quality validation, cross-source deduplication, data enrichment, schema standardization
Preferred skills
Vulnerability management, application security, cybersecurity data analytics, code-scanning (SAST/DAST), enterprise code repository metadata, DevSecOps tools, financial services experience, large enterprise banking environments
Technologies
BigQuery, GCP, GitHub, GitLab, Jenkins, Kafka, PySpark, Hadoop, Hive, ServiceNow, LLM
Responsibilities
Design and optimize batch and real-time streaming pipelines for enterprise security scanner findings; Implement data quality, enrichment, severity/criticality scoring, and deduplication across multiple scanning feeds; Ingest and process data using GCP services while supporting migration from on-premises pipelines; Build a modular framework for security team integration of new LLM and scanner models; Format and validate data for downstream systems and remediation workflows.