Data Engineer / Data Scientist
Core
Build scalable batch and streaming data pipelines, optimize Spark applications, and support machine learning model deployment on Azure.
Role type
Senior IC data engineer (MLOps)
Builds
Production data pipelines and ML model deployments (via careerplan.io/jobs/R1767376-data-engineer-data-scientist-at-cnx)
Domain
Cloud data engineering and machine learning
Deliverable
production ML models
Required skills
Python, PySpark, Azure Databricks, MLOps, Spark DataFrame optimization, API development, Delta Lake, Kafka, Azure Event Hubs, Azure Data Lake Storage
Preferred skills
MLflow, CI/CD for data workloads, Spark performance tuning, data quality monitoring, managed identity integration
Technologies
Azure, Databricks, PySpark, Kafka, Delta Lake, MLflow, GitHub
Responsibilities
Develop scalable data processing solutions using Python and PySpark; Build and maintain batch and streaming data pipelines; Debug and troubleshoot Spark jobs in Azure Databricks; Implement Delta Lake solutions for data reliability; Develop APIs using Python or Scala for data and ML applications; Support machine learning projects and MLOps workflows.
Seniority
Senior, hands-on IC