Senior Foundry Data Engineer
Core
Develop and optimize complex data processing pipelines and scalable data architectures for global data solutions.
Role type
Senior IC data engineer (Palantir Foundry)
Builds
Scalable data pipelines and architectures
Domain
Enterprise data engineering / Cloud data platforms
Deliverable
production ML models
Required skills
Palantir Foundry, PySpark, Python, SQL, AWS, Data Pipelines, ACE, EBard, NBA, Veeva, OneView, Product MDM
Preferred skills
Research and Development experience
Technologies
Palantir Foundry, PySpark, Python, SQL, AWS, ACE, EBard, NBA, Veeva, OneView, Product MDM
Responsibilities
Develop and optimize complex data processing pipelines using SQL Developer, Spark, PySpark, and Python. Collaborate with cross-functional teams to gather and analyse requirements for data solutions. Implement best practices for data management, ensuring data integrity and security. Provide technical expertise in the design and implementation of scalable data architectures. Troubleshoot and resolve performance issues in data processing workflows. Conduct code reviews to ensure code quality and adherence to coding standards. Mentor junior developers, providing guidance and support in their professional growth. System monitoring: Monitoring of global jobs and data health checks, issue communications, root cause identification and remediation and performance optimizations. Issue management: Attend triage calls, analysis and resolution of the issues, ticket updates and interaction with users. Demand management: Work on minor enhancements, bug fixes to the existing code. Clarify requirements with the team. Remediate upcoming impacts on existing global pipelines as per release change process. Operational reporting: Providing statistics about pipeline monitoring, issues, health checks and system usage. Documentation: Create team knowledge documents and keep it updated. Ensure adherence to standards, policies and quality compliance, enterprise metadata definition. Support user acceptance testing. Interaction with business stakeholders for seeking or providing clarifications.