Self-host Hugging Face Xet storage without changing your tools
OpenXet lets you run a chunk-deduplicated Xet server so official hf and Git clients upload only changed file chunks to your own infrastructure.
- LinkLoot access
- Free
- Provider costs
- Unknown
What you get from it
What it does
OpenXet provides an open-source server and client implementation for the Xet protocol, which is Hugging Face's method for storing large, versioned files using content-addressable storage (CAS) with global deduplication. Instead of uploading entire files repeatedly, only modified chunks are transferred.
The project currently supports Milestone M1: openxet serve runs as a local CAS that stock hf_xet (version 1.7.0+) uploads to and downloads from directly. It includes verified uploads, scoped tokens, and signed download URLs. Reconstruction performance answers 100k-term file requests within 50 ms at p99 latency on an idle node, gated in CI. The system runs on local disk with a SQLite index.
Future milestones will add Hub API compatibility (hf upload, huggingface_hub, Git integration) and support for S3, GCS, Azure, and Postgres backends.
Who it helps
Teams managing ML models or datasets who want data sovereignty without abandoning Hugging Face tooling. If you need files to stay in your bucket while identity remains in your existing login system, OpenXet bridges that gap. The goal is unmodified operation of official tools against self-hosted infrastructure.
Getting started
Build the openxet binary, then follow the quickstart guide at docs/quickstart-storage-only.md. Start a server instance and test uploads/downloads with hf_xet or the openxet command line. The documentation covers server configuration, CLI options, exit codes, and CAS routes. Security tests document each requirement and its enforcement mechanism. Benchmarks detail the latency gate and reconstruction metrics.
Note: There is no release yet. This is development-stage software.
Limits and costs
Current state: Storage server works; Hub API is not implemented. Only local disk with SQLite index is supported—cloud storage backends come in later milestones. No hosted service exists yet despite the openxet.com homepage reference. Apache-2.0 licensing permits commercial use but does not imply free hosting. Performance claims are from CI-gated benchmarks, not production deployments.
Source links
- Official repository: https://github.com/stubbi/openxet
- Project documentation: https://github.com/stubbi/openxet/blob/main/README.md
- Project license: https://github.com/stubbi/openxet/blob/main/LICENSE
Discussion
Share practical experience, questions, or warnings with the community.
Sign in to join the discussion and vote on comments.
Sign in