Lead Data Engineer
Core
Design and implement a modern lakehouse data platform for a federal permitting modernization initiative, leading engineering patterns for ingestion, harmonization, and serving data across multiple source systems.
Role type
Lead Data Engineer (hands-on technical leadership)
Builds
Core analytical data platform (DuckDB/Apache Iceberg on AWS), certified data products, and business metrics
Domain
Federal Government / Data Engineering / Cloud Infrastructure
Required skills
DuckDB, Apache Iceberg, AWS (S3, Glue), Python, Advanced SQL, Data Modeling (Bronze/Silver/Gold), Metadata-driven engineering, AI/Agentic workflows (LangChain/LangGraph), Technical Leadership
Preferred skills
Kafka/MSK, GeoParquet/PostGIS, OpenSearch, Enterprise SaaS integration, U.S. Federal Government experience
Technologies
DuckDB, Apache Iceberg, Amazon S3, AWS Glue, Python, LangChain, LangGraph
Responsibilities
Design and implement the core DuckDB / Apache Iceberg data platform on AWS; Define physical data organization, schema, partitioning, file layout, snapshot, compaction, and optimization strategies for Iceberg tables; Integrate DuckDB with Amazon S3, Apache Iceberg, and AWS Glue Catalog; Establish bronze, silver, and gold data-layer conventions and reusable engineering patterns; Lead and mentor Data Engineers, providing technical direction, code reviews, and engineering guidance; Build and maintain analytical datasets supporting application services, reporting, analytics, and data science; Develop reusable patterns for incremental loads, CDC, schema evolution, late-arriving data, backfills, and historical reconstruction; Work with the Data Architect to implement the common canonical data model and source-to-canonical mappings; Lead migration and harmonization of datasets from multiple source systems; Implement reconciliation processes that demonstrate completeness and accuracy between source and target systems; Establish automated data-quality monitoring, operational observability, and testing; Implement agentic and AI-enabled data engineering patterns supporting activities such as source analysis, mapping, pipeline generation, troubleshooting, reconciliation, and quality assurance; Support integration of structured data with associated document and geospatial datasets; Lead development of certified data products, analytical datasets, and business metrics; Optimize DuckDB queries and Iceberg datasets for performance and efficiency; Develop engineering documentation, operational procedures, and technical handoff materials.
Seniority
Lead, hands-on technical leadership