λFS: Elastically Scaling Distributed File System Metadata Service using Serverless Functions
Published in ASPLOS 2023, 2024
The metadata service (MDS) sits on the critical path for file system operations, and as such it is key to the overall performance of a large-scale distributed file system (DFS). Common “serverful” MDS architectures, such as a single server or cluster of servers, have a significant shortcoming: either they are not scalable, or they make it difficult to achieve an optimal balance of performance, resource utilization, and cost. A modern MDS requires a novel architecture that addresses this shortcoming. To this end, we design and implement λFS, an elastic, high-performance metadata service for large-scale distributed file systems. λFS scales a DFS metadata cache on a FaaS (Function-as-a-Service) platform and synthesizes a series of techniques to overcome the obstacles that are encountered when building large stateful applications on FaaS platforms. λFS takes full advantage of the unique benefits offered by FaaS–elastic scaling and massive parallelism–to realize a highly-optimized metadata service capable of sustaining up to 4.13× higher throughput, 90.40% lower latency, 85.99% lower cost, and better resource utilization and efficiency than a state-of-the-art DFS for an industrial workload.
Download here