Lead Software Engineer
Core
Lead the design, development, and maintenance of data engineering infrastructure, architecting scalable pipelines for batch and real-time processing while migrating on-prem applications to cloud-based PaaS.
Role type
Senior IC data engineering lead (cloud migration & architecture)
Builds
Scalable data pipelines, data warehouses, and cloud-based data solutions for multi-platform consumption
Domain
Data engineering, cloud infrastructure, big data
Deliverable
production ML models | product features | infrastructure
Required skills
Cloud architecture, data modeling, ETL/ELT design, database optimization, data pipeline architecture, data governance, technical leadership, agile development, problem-solving
Preferred skills
Big data technologies, data orchestration, AWS migration services, microservices architecture
Technologies
Python, SQL, Java, Scala, Apache Spark, Hadoop, Kafka, Hive, AWS (S3, EC2, RDS, Lambda, DynamoDB, Glue), Azure (Data Factory, Synapse, Databricks), GCP (Big Query, Dataflow, Dataproc), Snowflake, Redshift, Fabric Data Warehouse, Airflow, dbt, Informatica, Talend, Glue, PostgreSQL, MySQL, SQL Server, Oracle, MongoDB, Cassandra, DynamoDB, Influx DB, Timescale DB, Docker, Kubernetes, Terraform, CloudFormation, Git, GitLab/GitHub, Jenkins, GitHub Actions, GitLab CI, Datadog, Grafana
Responsibilities
Architect end-to-end data solutions including collection, storage, modeling, and consumption; Lead migration of multi-terabyte databases from on-prem to cloud; Design data-intensive applications with APIs and streaming pipelines; Establish coding standards and technical documentation; Mentor junior data engineers; Optimize data models and database designs for performance; Implement data quality controls and monitoring systems; Deploy and manage cloud-based data solutions; Evaluate and recommend new data technologies
Seniority
Senior, hands-on IC with leadership responsibilities
