[Job-32448] Master Data Developer (Databricks), Brazil
Core
Design, develop, and maintain scalable data infrastructure and pipelines, specifically migrating data architectures from Databricks to AWS Glue and optimizing data quality and performance.
Role type
Senior Data Developer (Data Engineering)
Builds
Scalable data pipelines, ETL/ELT processes, data lakes, and warehouses
Domain
Data Engineering, Cloud Platforms (AWS), Big Data
Required skills
Databricks migration, Unity Catalog to Glue Catalog, Deltalake to Iceberg format, Spark SQL, PySpark, Python, AWS S3, AWS Glue, AWS Secrets Manager, AWS IAM, AWS Athena, AWS LakeFormation, AWS EMR, Auto Loader, DLT pipelines, Data quality validation, API development
Preferred skills
Databricks Certified Consultant, DBR to EMR on AWS migration experience
Technologies
Databricks, AWS, Spark, Python, Java, Scala, Iceberg, Delta Lake, S3, Glue, EMR, Athena, LakeFormation
Responsibilities
Build and optimize scalable data pipelines and ETL/ELT processes; Implement data architectures including data lakes and warehouses; Ensure data quality and governance through validation and monitoring; Develop integrations between data sources and analytics platforms; Collaborate with stakeholders to deliver data-driven solutions; Monitor and optimize pipeline performance and storage costs; Create technical documentation; Implement data security best practices; Leverage AI tools to enhance workflows.
Seniority
Senior, hands-on IC