Senior Data Engineer
Required skills
3-5+ years of professional work experience in data engineering, Strong expertise in Python and common data frameworks (Pandas, PySpark), Experience with modern data storage solutions: Data lakes (Delta tables), and data warehouses, Experience with data orchestration tools (Dagster, Airflow), Experience with streaming technologies (Kafka, Spark structured streaming), Experience with cloud platforms (AWS S3, EC2 etc.), OLAP Databases: Hands-on experience with ClickHouse and/or Apache Druid for real-time analytics and low-latency data serving, Strong problem-solving skills and ability to excel in a fast-paced startup environment, Based in Barcelona or ready to relocate to Barcelona
Preferred skills
Experience with Databricks platform, Experience in building ML data platforms
Technologies
Databricks platform (Apache Spark, Delta Lake, Delta Live Tables), ClickHouse, Druid, Kafka, Spark structured streaming, AWS S3, EC2, Pandas, PySpark, Dagster, Airflow
Responsibilities
Design and build scalable data pipelines: Architect, construct, and maintain robust ETL/ELT pipelines using the Databricks platform (Apache Spark, Delta Lake, Delta Live Tables) for batch and streaming data ingestion from various sources, Manage data architecture: Define and maintain a scalable and secure data platform architecture that integrates Databricks (for data lakehouse management and complex transformations) with specialized OLAP engines (ClickHouse/Druid for fast querying), Design, develop, test, and orchestrate data workflows to streamline pipelines, Optimize performance and reliability: Monitor, troubleshoot, and fine-tune data workflows and database performance (e.g., Spark job optimization, ClickHouse schema tuning, Druid data partitioning) to ensure maximum efficiency and reliability, Contribute to the development of internal tools for automating data processes and business workflows, enhancing efficiency and scalability, Collaborate closely with product, MLOps and data science teams to ensure an optimal data experience for all ML practitioners and data consumers
Seniority
Senior
Domain
Mobile app monetization
