A scalable, high-performance, and locality-aware serverless DAG engine
Scroll to explore
The Problem
Serverless computing meets parallel computing
Serverless computing is increasingly being used for parallel computing, which have traditionally been implemented as stateful applications. But executing complex, burst-parallel workloads on serverless platforms introduces fundamental challenges.
The Challenge
Executing complex DAG jobs at massive scale
Rapid Scaling
Serverless frameworks must rapidly scale to handle burst-parallel workloads with thousands of concurrent tasks.
High-Throughput Scheduling
Tasks must be scheduled at high throughput to keep pace with the parallelism of the computation.
Data Movement
Minimizing data movement across tasks is critical for performance — shuffling data between executors is expensive.
The Insight
Decentralized
Scheduling
Rather than relying on a single centralized scheduler, Wukong distributes scheduling across Lambda executors themselves — enabling tasks to be scheduled in parallel, reducing resource contention, and making task scheduling locality-aware.
Key Benefits
Why decentralized scheduling wins
Enhanced Data Locality
Tasks are scheduled on executors that already hold the required data, eliminating costly data transfers.
Reduced Network I/O
Network I/O is reduced by multiple orders of magnitude compared to centralized approaches.
Automatic Elasticity
Resources scale automatically with workload demands — no manual provisioning required.
Cost Effective
Pay only for the compute you use. Wukong achieves 92.96% tenant-side cost savings.
Results
Performance that speaks for itself
Deployed on AWS Lambda, Wukong delivers transformative improvements across every dimension.
Faster execution
Less network I/O
Cost savings
Scalability
Implementation
Built on AWS Lambda
Wukong is implemented and deployed on AWS Lambda, leveraging serverless functions as distributed executors. Each Lambda function acts as both a worker and a scheduler, enabling fully decentralized task coordination with no dedicated scheduling infrastructure.
Explore Wukong
Read the paper, explore the code, or watch the presentation.