Self-host Hugging Face's Xet storage layer and keep model files servable offline

What it does

xet-server is a Go implementation of Hugging Face's Xet storage protocol, the content-defined-chunking, dedup-first layer that replaced Git LFS for large model files on the Hub. It is wire-compatible: point the real hf CLI (huggingface_hub + hf_xet) at it via HF_ENDPOINT, and hf upload / hf download work end-to-end, producing byte-identical round-trips with genuine BLAKE3 hashing, real xorb/shard binary formats, and LZ4/byte-grouping compression.

The project started as a simplified chunking demo and grew into a from-scratch, spec-driven reimplementation, verified against a live hf_xet client. Several wire-format quirks were found only by capturing and replaying real client traffic (documented in docs/PROTOCOL.md).

It ships two main pieces:

• The wire-compatible server — a CAS HTTP API (xorb upload/fetch, shard upload, file reconstruction with Range paging) plus a Hub API shim covering repo create, preupload, read/write tokens, commit, and resolve/HEAD, including independent revisions and branches per repo.
• xet-proxyd — a caching pull-through proxy for the real huggingface.co. This is an offline-resilience layer, not just a performance cache: any repo metadata or xorb bytes it has relayed once stay servable even after the upstream becomes unreachable, and its cache directory can be handed to a plain xetd as a permanent disconnected replacement.

Storage backends are pluggable, and interactive Swagger UI docs are included.

Who it helps

Teams running air-gapped or bandwidth-constrained ML infrastructure who want to serve model weights through the standard toolchain without depending on Hugging Face's hosted endpoint. Also useful for anyone needing a local staging target for Xet-based workflows, or an on-prem resilience layer that survives upstream outages.

Getting started

Build and run the server, then set HF_ENDPOINT to its address and use the normal hf commands. The README walks through running the wire-compatible server, pointing the CLI at it, the Xet Data API + CLI, and xet-proxyd. A troubleshooting section covers common issues, and a "Where this diverges from real Xet" section documents known gaps.

Limits and costs

MIT-licensed, so source use is unrestricted, but hosting is on you — hardware, storage, and operational overhead are not covered by the license. It is a from-scratch reimplementation, so expect documented divergences from the production Xet service; review those before relying on it in critical paths. No pricing is published because there is none beyond self-hosting.

Source links

• Official repository: https://github.com/guilt/xet-server
• Project documentation: https://github.com/guilt/xet-server/blob/main/README.md
• Project license: https://github.com/guilt/xet-server/blob/main/LICENSE.md