Senior Data Engineer

## About the Role We are building a modern data platform that will allow Willow Innovations the ability to make the most optimal business decisions. We are a growing organization and the Senior Data Engineer will have a huge influence in shaping the data strategy. If you feel a deep sense of ownership for your work, are passionate about data, desire to solve problems, are curious and love learning new things, we'd love to talk to you! ## Roles & Responsibilities - Own and drive data integration architecture involving streaming, micro-batch and batch - Work closely with various business groups to collect business requirements - Design, develop and deploy high performance data pipelines both inbound and outbound - Build aggregate and enriched data sets needed to effectively analyze / visualize data - Document, maintain and support data pipelines - Measure and monitor quality and availability of data - Implement data policies - retention, privacy, security, and compliance - Responsible for maintaining the Data Pipeline framework - Mentor and guide junior data engineers ## Experience ### Must Have - 5+ years of experience in building data lakes and data warehouses - Experience working with Google Cloud, BigQuery, GCS, Pub Sub, Data Flow etc - Expert knowledge on SQL and Python programming - Good understanding of data formats such as JSON, Parquet, ORC, XML, etc - Experience in tuning queries for performance and scalability - Experience working with any workflow management tools (Airflow, Oozie, etc) - Excellent organizational, prioritization and analytical abilities - Strong communication and interpersonal skills ### Nice to Have - Experience in Real Time Data Ingestion using Kafka, Spark, Storm etc - Knowledge of NoSQL - Experience with Data visualization tools such as Looker, Tableau, PowerBI