A file that tells crawlers which pages changed

September 17 correction: PageDigest v1.0 is final. This July announcement has been corrected to describe revisions as publisher declarations and to account for incomplete caches, processing, and audits. The earlier wording overstated what an unchanged revision proves. Release-candidate details below describe July 4; see the project page for current releases and measurements.

PageDigest v1 is now in release-candidate status.

PageDigest is a one-file change-detection protocol for public websites. A publisher serves a manifest at /.well-known/pagedigest.json. Consumers compare it with saved state and can reuse cached pages that the publisher reports as unchanged, subject to cache validity, pending processing, and audit requirements.

For a 10,000-page site that changes 20 pages in a week, a consumer with a complete, valid cache and no pending processing or audits could make one manifest request plus 20 page fetches instead of 10,000 per-URL checks. Missing cached pages and scheduled audits add work to that example.

Why this exists

Publishers see automated clients fetching the same pages again and again. Crawlers still need to know what changed. A shared revision manifest gives them a way to coordinate those checks:

{
  "version": 1,
  "generated": "2026-07-04T00:00:00Z",
  "site_rev": 18294,
  "entries": {
    "/": { "rev": 47 },
    "/blog/hello-world": {
      "rev": 4,
      "digest": "sha256:..."
    }
  }
}

site_rev is the fast path. If it did not change, a consumer can stop after one manifest request only when its cache is complete and valid and no processing or audits remain due. Per-URL rev values identify the changes declared by the publisher when the site revision advances. Optional digest values let consumers audit those declarations; sampled checks do not prove that every skipped page is unchanged.

What ships in the RC

The v1 release candidate includes:

• a stable wire format with site_rev, per-URL rev, optional digest, and optional modified fields
• digest audit guidance for consumers that want to verify publisher claims
• optional PageDigest-State request headers so cooperative consumers can make manifest observations visible in ordinary publisher logs
• reference publisher and consumer implementations
• distribution through GitHub Releases, Cargo, npx, PyPI, and an Astro integration
• production dogfood on pagedigest.org and a measured producer case study for dotrepo.org

Install paths:

npx pagedigest ./site-dist
cargo install pagedigest
python -m pip install pagedigest
npm install @pagedigest/astro

What would help before v1.0

I’d like to hear from people trying it:

• publisher integration reports from static-site generators, custom build systems, CMS exports, and CDN or edge deployments
• consumer integration reports from crawlers, caches, monitors, archive systems, agents, and search/indexing pipelines
• real cases where URL keys, redirects, digest audits, deployment races, or partial coverage semantics are surprising

The project has a public v1 RC feedback issue. The canonical announcement lives on pagedigest.org, and the implementation lives in the pagedigest repository.

The main remaining v1.0 dependency outside the repo is registry review for the well-known URI and link relation requests. The v1 field semantics are expected to remain stable through final v1.0.