NoKV  by feichai0017

Namespace metadata substrate for distributed storage and AI workloads

Created 1 year ago
379 stars

Top 74.9% on SourcePulse

GitHubView on GitHub
Project Summary

NoKV provides an open-source namespace metadata substrate designed for distributed filesystems, object storage, and AI dataset workloads. It implements a "stateless schema layer + transactional KV" pattern, offering native metadata primitives as first-class server-side operations. This approach benefits developers building DFS frontends, S3 gateways, or AI dataset services by delivering significantly higher performance and a unified source of truth for namespace metadata, abstracting away the complexities of underlying storage.

How It Works

NoKV features a vertically integrated architecture, encompassing its own LSM-tree storage engine (with an ART memtable), Raft consensus, Percolator MVCC, and a coordinator. This deep integration allows for the implementation of namespace-specific operations like ReadDirPlus, WatchSubtree, SnapshotSubtree, and RenameSubtree as native server-side primitives, rather than client-side compositions over generic Get/Put/Scan calls. This design yields substantial performance advantages and ensures a single, replicated event log for critical metadata states such as mount lifecycle, subtree authority, snapshot epochs, and quota fences.

Quick Start & Requirements

Highlighted Details

  • ReadDirPlus operations are up to 42.5x faster compared to generic KV calls.
  • WatchSubtree provides sub-second end-to-end change feed latency for prefix-scoped metadata watches.
  • The underlying KV layer demonstrates superior performance over Badger and Pebble in YCSB benchmarks.
  • Key metadata primitives like ReadDirPlus, WatchSubtree, SnapshotSubtree, and RenameSubtree are first-class server-side operations.

Maintenance & Community

The project appears actively developed, with detailed documentation and recent benchmarks. No specific community channels (like Discord or Slack) or notable external contributors/sponsorships are listed in the provided README.

Licensing & Compatibility

NoKV is licensed under the Apache-2.0 license. This license is permissive and generally compatible with commercial use and linking within closed-source projects.

Limitations & Caveats

NoKV is explicitly positioned as a metadata "substrate," not a complete filesystem server; it does not ship with FUSE drivers or S3 gateways out-of-the-box. Its internal KV layer is optimized for its metadata service and is not intended as a direct replacement for general-purpose distributed KV stores like TiKV or FoundationDB.

Health Check
Last Commit

13 hours ago

Responsiveness

Inactive

Pull Requests (30d)
34
Issues (30d)
17
Star History
154 stars in the last 30 days

Explore Similar Projects

Starred by Chip Huyen Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"), Joe Walnes Joe Walnes(Head of Experimental Projects at Stripe), and
9 more.

3FS by deepseek-ai

0.2%
10k
Distributed file system for AI training/inference workloads
Created 1 year ago
Updated 4 weeks ago
Feedback? Help us improve.