Senior Python Backend Engineer Async Systems Pipeline Infrastructure

🌐 Remote💼 Full-time💰 $31,200–$31,200🗓 2026-07-26

Core

Design and build long-running, multi-step async pipeline systems handling complex workflows, external API interactions, and LLM orchestration with robust failure recovery.

Role type

Senior Python Backend Engineer (Async Systems & Pipeline Infrastructure)

Builds

Production-grade async, queue-driven pipeline engines and background processing infrastructure

Domain

Distributed systems, backend infrastructure, LLM workflow orchestration

Deliverable

production ML models | product features | infrastructure

Required skills

Python, async programming, distributed systems patterns, FastAPI, Celery/Dramatiq, PostgreSQL, SQLAlchemy, database design, idempotency, retry logic, observability

Preferred skills

LLM workflow orchestration, multi-agent system design, incident response tooling

Technologies

FastAPI, Celery, Dramatiq, Redis, PostgreSQL, SQLAlchemy 2.0, Pydantic v2

Responsibilities

Design long-running multi-step async pipeline systems; Develop background processing infrastructure; Implement distributed systems patterns (idempotency, retries, dead-letter queues); Build orchestration layers for LLM and multi-agent workflows; Develop backend services with async database access; Build monitoring and incident response tooling

Seniority

Senior, hands-on IC

Rewrite

## About the Role We're hiring a Senior Python Backend Engineer to build and own highly reliable, long-running async pipeline systems. This is a backend-heavy role where the challenge is reliability, orchestration, and distributed systems, not CRUD applications. Our core platform is a pipeline engine where jobs move through complex multi-step workflows that can run for hours. Some steps interact with external APIs, some require human input, and others orchestrate LLM workflows. The system must handle failures, retries, duplicate deliveries, and partial execution gracefully. We're looking for someone who has built and operated production-grade async, queue-driven systems and can independently own critical backend infrastructure. ## What You'll Work On - Design and build long-running, multi-step async pipeline systems. - Develop background processing infrastructure using Celery, Dramatiq, Redis, or similar technologies. - Implement robust distributed systems patterns, including: - Idempotency - Retries - At-least-once delivery - Dead-letter queues - Failure recovery - Build orchestration layers for LLM and multi-agent workflows, including timeout and cost controls. - Develop backend services using: - FastAPI - SQLAlchemy 2.0 (async) - PostgreSQL - Pydantic v2 - Design pipeline state models that are: - Observable - Resumable - Fault-tolerant - Build monitoring, observability, and incident response tooling. ## Requirements - 5+ years of experience building backend systems with Python. - Production experience with FastAPI. - Hands-on experience with Celery, Dramatiq, or similar task queue systems. - Strong experience with PostgreSQL, SQLAlchemy, and database design. - Comfortable independently owning large areas of backend infrastructure. ## Nice to Have - If you enjoy building resilient backend systems, distributed workflows, and long-running asynchronous infrastructure, we'd love to hear from you. ## How to Apply To apply, please share: - Your resume or LinkedIn profile - Relevant distributed systems or async infrastructure projects - Your availability and expected hourly rate

Sourced via wellfound · Listed on CareerPlan, which tracks 70,000+ jobs from 20+ sources.