Highly Scalable, Fault-tolerant and Event-driven Multi-stage Distributed Execution Pipeline

The Challenge

Execution requests arrived from multiple trigger sources with different payload shapes, while the platform had to fetch execution data, validate subscriptions and usage limits, select the right infrastructure, and keep jobs moving even when services or orchestration steps failed.

Engineering Approach

• Normalized GitHub, CircleCI, scheduled, and manual triggers into a shared execution contract so downstream stages could remain independent of the source format.
• Split the pipeline into isolated stages for resource lookup, subscription and quota validation, test enrichment, infrastructure selection, and execution dispatch so each step could scale and fail independently.
• Routed API tests to a dedicated processor, web tests to AWS ECS-backed Playwright environments with browser selection based on requirements, and mobile tests to AWS Device Farm when devices were available.
• Introduced queue-based waiting with priority-aware scheduling so paid customers could receive shorter wait times than free-trial users when capacity was constrained.
• Added retry-aware failure handling and dead-letter processing so transient infrastructure issues could be retried instead of being marked failed immediately.
• Used separate queues and processors with explicit inputs and outputs to keep each stage independently scalable and prevent failures in one path from blocking the rest of the system.

Impact

• Handled large volumes of execution requests without overloading limited AWS capacity by staging work through independent queues and processors.
• Improved reliability and client visibility by recovering from transient infrastructure issues, preserving failed jobs, classifying failure causes, and surfacing accurate retry or failure status instead of dropping requests.