Data Engineer
Core
Design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data to produce a cloud cost chargeback dataset.
Role type
Senior Data Engineer (FinOps)
Builds
FinOps Data Platform, ETL processes, data models, and data infrastructure components
Domain
Cloud infrastructure, Big Data, FinOps
Deliverable
production ML models | product features | dashboards & analysis | infrastructure
Required skills
Python, Java, Scala, SQL, ETL tools (Databricks, Azure Data Factory), Big Data frameworks (Hadoop, Spark, Kafka), Cloud platforms (AWS, Azure, GCP, Snowflake), Data modeling, API development, Data validation
Preferred skills
TBM Studio, Cloudability Data Platform
Technologies
Azure Data Factory, Synapse, AWS S3, BigQuery, Hadoop, Spark, Kafka, Snowflake
Responsibilities
Design and implement end-to-end data pipelines for cloud cost chargeback; Develop robust ETL processes to integrate data from diverse sources; Design and maintain data models, schemas, and database structures; Optimize data storage and retrieval mechanisms for performance and scalability; Build and maintain integrations with internal and external data sources and APIs; Configure and manage data infrastructure components including databases, data warehouses, and distributed computing frameworks; Monitor system performance, troubleshoot issues, and implement optimizations; Implement data security controls and access management policies; Collaborate with data analytics and other stakeholders to understand data requirements; Document technical designs, workflows, and best practices.
Seniority
Senior, hands-on IC
