Data Engineer
Core
Design, develop, and optimize big data pipelines and architectures on the Databricks Unified Analytics Platform to enable scalable data solutions for Fortune 500 clients.
Role type
Senior Data Engineer (Big Data & Cloud)
Builds
Scalable ETL workflows, data lakehouse architectures, and AI/ML data pipelines
Domain
Cloud Data Engineering (AWS/Azure/GCP)
Deliverable
production ML models | infrastructure
Required skills
Apache Spark, SQL, Python, Databricks, ETL frameworks, CI/CD, Infrastructure-as-Code
Preferred skills
MLflow, Feature Store, Kubernetes, Docker, Terraform, Streaming data architectures (Kafka, Kinesis)
Technologies
Databricks, Apache Spark, AWS, Azure, GCP, Delta Lake, Kafka, Kinesis, MLflow, Terraform
Responsibilities
Design and implement big data pipelines in Databricks; Develop scalable ETL workflows; Leverage Apache Spark for distributed data processing; Implement data governance and security policies; Optimize data lakehouse architectures; Collaborate with data scientists to enable AI/ML workflows; Monitor and troubleshoot Databricks clusters; Automate workflows using CI/CD pipelines
Seniority
Senior, hands-on IC
