Data Engineer

## Responsibilities - Display technical expertise in data analytics focusing on a team of diversified technical competencies. - Build and maintain accurate and scalable data pipeline and infrastructure such as SQL Warehouse, Data Lakes, etc. using Cloud platforms (e.g.: MS Azure, Databricks). - Proactively work with business stakeholders to understand data lineage, definitions, and methods of data extraction. - Write production-grade SQL and PySpark code to create data architecture. - Consolidate SQL databases from multiple sources, data cleaning, and manipulation in preparation for analytics and machine learning. - Use data visualization tools such as Power BI to create professional quality dashboards and reports. - Write good quality documentation for data processing for different projects to ensure reproducibility. - Living Hitachi Energy’s core values safety and integrity, which means taking responsibility for your own actions while caring for your colleagues and the business. ## Requirements - BE / B.Tech in Computer Science, Data Science, or related discipline and at least 5 years of related working experience. - 5 years of data engineering experience, with understanding lake house architecture, data integration framework, ETL/ELT pipeline, orchestration/monitoring, star schema data modeling. - 5 years of experience with Python/PySpark and SQL (Proficient in PySpark, Python, and Spark SQL) - 2-3 years of hands-on data engineering experience using Databricks as the main tool (meaning >60% of their time is using Databricks instead of just occasionally). - 2-3 years of hands-on experience with different Databricks components (DLT, workflow, Unity catalog, SQL warehouse, CI/CD) in addition to using notebooks. - Experience in Microsoft Power BI. ## Nice to Have - Basic understanding of Machine Learning algorithms. - Enhanced Data Processing: AI/ML can automate and improve data processing tasks, making them more efficient and accurate. - Predictive Analytics: Integrating AI/ML can help in building predictive models that can forecast trends and outcomes, providing valuable insights for decision-making. - Personalization: AI/ML can be used to create personalized experiences for users by analyzing their behavior and preferences. - Automation: AI/ML can automate repetitive tasks, freeing up time for more complex and creative work. - Ability to quickly grasp concepts from a field that is not one's core competency; A fast learning generalist capable of solving problems independently and resourceful in a matrixed corporate environment. ## Benefits - Proficiency in both spoken & written English language is required.