Lead Data Engineer
Job Description:
Title: Lead data engineer
Experience - 7-10 Years
About the Role
We are seeking a highly skilled and delivery-focused Lead GCP Data Engineer to support the design, development, and implementation of next-generation enterprise data and AI platforms on Google Cloud Platform (GCP).
This role will work closely with Enterprise Architects, platform leaders, and cross-functional engineering teams to build scalable, reusable, and AI-ready data foundations that enable advanced analytics, intelligent automation, and enterprise AI adoption.
The ideal candidate combines strong hands-on expertise in cloud-native data engineering, modern data platform development, semantic data enablement, and scalable pipeline engineering with the ability to lead engineering teams and drive high-quality delivery across multiple initiatives.
This role is expected to play a critical leadership position within the engineering organization by driving implementation excellence, mentoring teams, and operationalizing modern data architecture patterns.
Key Responsibilities
1. Enterprise Data Platform Engineering
• Design, develop, and optimize scalable cloud-native data platforms and pipelines on GCP.
• Implement robust batch, streaming, and event-driven data processing solutions supporting enterprise analytics and AI use cases.
• Collaborate with Enterprise Architects to translate target-state architecture into scalable engineering implementations.
• Contribute to modernization of legacy data ecosystems into reusable, governed, and AI-ready cloud platforms.
• Support implementation of scalable ingestion, transformation, serving, and orchestration frameworks.
2. Data Product Engineering
• Develop reusable and domain-oriented data products aligned with data mesh and data-as-a-product principles.
• Implement scalable and modular data pipelines supporting multiple downstream consumers including analytics, AI/ML, and operational applications.
• Contribute to implementation of:
• Data contracts
• Schema management
• Metadata enrichment
• Data quality frameworks
• Reusable transformation patterns
• Enable discoverability, trust, and operational reliability of enterprise data assets.
3. Semantic Layer & Consumption Enablement
• Support implementation of semantic and business-consumption layers that simplify enterprise data access.
• Collaborate with analytics and BI teams to enable standardized business metrics, reusable dimensions, and governed KPI definitions.
• Contribute to semantic modeling and metadata integration initiatives supporting self-service analytics and AI consumption.
• Assist in improving enterprise data usability, consistency, and discoverability across platforms.
4. GCP-Native Engineering & Development
• Develop and optimize solutions leveraging GCP-native services including:
• BigQuery
• Dataflow
• Dataproc
• DBT
• Pub/Sub
• Cloud Storage
• Cloud Composer (Airflow)
• Cloud SQL
• Build scalable ETL/ELT frameworks and real-time streaming pipelines.
• Optimize data processing performance, reliability, scalability, and cost efficiency.
• Implement CI/CD pipelines and engineering automation for data platform delivery.
5. AI/ML & GenAI Data Enablement
• Build AI-ready data pipelines and scalable feature engineering workflows supporting enterprise AI initiatives.
• Support integration with:
• Vertex AI
• BigQuery ML
• Vector databases
• LangChain
• Generative AI Studio
• Contribute to implementation of RAG architectures, semantic search, and AI-assisted data interaction patterns.
• Partner with AI/ML teams to operationalize scalable ML and GenAI workflows.
6. Engineering Leadership & Delivery Excellence
• Lead day-to-day engineering activities across multiple data engineering workstreams.
• Guide and mentor junior and mid-level data engineers on modern engineering best practices.
• Ensure adherence to coding standards, architecture guidelines, and operational best practices.
• Drive engineering quality through automated testing, observability, monitoring, and performance optimization.
• Collaborate with architects, product owners, analysts, and client stakeholders to ensure successful delivery outcomes.
7. Governance, Reliability & Observability
• Implement data governance, lineage, monitoring, and observability frameworks.
• Support enforcement of enterprise standards around security, reliability, scalability, and operational readiness.
• Contribute to platform monitoring, incident management, and continuous improvement initiatives.
• Ensure production readiness of pipelines and data services through robust testing and validation processes.
Technical Expertise Required
Area
Skills / Technologies
Cloud Data Engineering
GCP, BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage, Cloud SQL
Data Transformation
DBT, PySpark, SQL, ETL/ELT frameworks
Streaming & Pipelines
Apache Beam, real-time processing, event-driven architectures
Semantic Layer & Modeling
Semantic modeling concepts, Looker modeling, business metrics standardization
AI/ML Enablement
Vertex AI, BigQuery ML, LangChain, Vector Databases, GenAI integration
Orchestration & Automation
Cloud Composer (Airflow), CI/CD, Workflows
Metadata & Governance
Data Catalog, lineage, metadata management, observability frameworks
Programming
Python, SQL, PySpark
Qualifications
• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, or related field.
• 7+ years of experience in data engineering and cloud-native data platform development.
• Minimum 4+ years of hands-on experience delivering enterprise-scale solutions on GCP.
• Strong expertise in building scalable batch and streaming data pipelines.
• Experience working on modern enterprise data platforms supporting analytics, AI/ML, and GenAI use cases.
• Good understanding of semantic layer concepts, reusable data models, and governed data consumption patterns.
• Experience working within large-scale data modernization and cloud transformation initiatives.
• Strong problem-solving, debugging, and performance optimization skills.
• Proven ability to lead engineering teams and collaborate across architecture, product, and business functions.
• Excellent communication and stakeholder management skills.
• GCP certifications such as Professional Data Engineer preferred.
Location:
DGS India - Mumbai - Thane Ashar IT Park
Brand:
Merkle
Time Type:
Full time
Contract Type:
Permanent
