Principal Engineering Lead — Content & Data Pipeline
Core
Leads the end-to-end architecture and delivery of Wiley's Content & Data Pipeline, transforming scholarly content (journals, articles, books) from various sources into structured data for internal AI/RAG systems and external customers.
Role type
Principal Engineering Lead (Content & Data Pipeline)
Builds
Scalable data ingestion, processing, and delivery platforms for scholarly content
Domain
Scholarly publishing, data engineering, AI infrastructure
Deliverable
production ML models
Required skills
AWS (S3, Lambda, ECS/EKS, SQS/SNS, EventBridge, Step Functions, RDS, DynamoDB), Python, SQL, Workflow orchestration (Airflow, Step Functions, Dagster), Content transformation (XML, schema-driven), Data quality practices, Distributed systems, API/bulk ingestion, CI/CD, Observability, AI coding agents
Preferred skills
Scholarly publishing domain knowledge (JATS, DOIs, Crossref, retraction handling), BigQuery/GCP, Specialized publishing tooling (XSLT, CMS), Streaming ingestion (Kafka)
Responsibilities
Own end-to-end architecture for content pipeline; Manage and coach a team of engineers; Partner with product, business, and licensing partners; Define technical roadmap and standards; Operate production pipelines and ensure data quality; Translate stakeholder needs into platform capabilities
Seniority
Principal, hands-on IC with team leadership
