An elastic, high-performance metadata service for large-scale distributed file systems
Scroll to explore
The Problem
Metadata is the bottleneck
The metadata service (MDS) sits on the critical path for every distributed file system operation. Common “serverful” MDS architectures — a single server or a cluster of servers — are either not scalable or make it difficult to achieve an optimal balance of performance, resource utilization, and cost.
The Serverful Trade-off
Scalability, performance, or cost — pick two
Single-Server MDS
Simple and low-cost, but becomes a scalability bottleneck as the file system grows to billions of files and thousands of clients.
Clustered MDS
More scalable, but achieving optimal performance with good resource utilization at reasonable cost requires careful and complex tuning.
The Missing Architecture
A modern MDS needs elastic scaling, massive parallelism, and cost efficiency — without sacrificing performance or forcing operational complexity.
The Insight
Serverless
Metadata
λFS scales a DFS metadata cache elastically on a FaaS platform, synthesizing a series of techniques to overcome the obstacles encountered when building large, stateful, and performance-sensitive applications on serverless infrastructure.
How It Works
Serverless functions as metadata infrastructure
FaaS Metadata Cache
λFS deploys metadata caching logic as serverless functions, scaling cache capacity elastically with demand instead of over-provisioning servers.
Elastic Scaling
Takes full advantage of FaaS elastic scaling — the metadata layer grows and shrinks automatically to match workload intensity.
Massive Parallelism
FaaS enables massive parallelism for metadata operations, processing thousands of concurrent requests without coordination bottlenecks.
Stateful on Serverless
Synthesizes novel techniques to overcome the challenges of building large, stateful, performance-sensitive applications on inherently stateless FaaS platforms.
Results
Outperforming the state of the art
Evaluated against state-of-the-art distributed file systems using an industrial workload, λFS delivers transformative improvements.
Higher throughput
Lower latency
Lower cost
Better perf-per-cost
Implementation
Multi-cloud, battle-tested
λFS is built on top of HopsFS/HDFS and has been deployed and evaluated across AWS, Google Cloud, and IBM Cloud using serverless platforms including Nuclio and OpenWhisk. The metadata cache layer is implemented in Java and C++ for maximum throughput, backed by MySQL Cluster NDB for persistent storage.
Explore λFS
Read the paper, explore the code, or watch the presentation.