PageDigest

PageDigest is a one-manifest protocol for website change detection.

A publisher serves /.well-known/pagedigest.json. A stateful crawler, monitor, indexer, or automated client fetches that one file first, compares it with the state it has already seen, and skips page fetches for URLs whose revisions have not changed.

It is the sibling layer to dotrepo. dotrepo makes repository interpretation reusable; PageDigest makes website fetching reusable. Together they are a cooperative automation pair: fetch less, infer less, and keep the evidence visible.

What it is

The manifest is intentionally small:

• site_rev gives consumers a fast whole-site unchanged check.
• Per-URL rev values identify the pages that changed when the site revision moves.
• Optional SHA-256 digests let consumers spot-audit publisher claims.
• Optional modified timestamps help clients sort and report changes without making timestamps the source of truth.

The protocol is meant for public websites and cooperative automated clients. It does not ask crawlers to parse every page before learning that nothing changed, and it does not ask publishers to invent a custom change API for each site.

What ships today

The v1 release candidate is a real implementation surface, not just a document.

It includes:

• a stable v1 manifest format and specification
• Rust publisher tooling and release binaries
• npm and Cargo publisher paths
• a Python consumer package
• an Astro integration
• conformance tests and fixtures
• a live dogfood site at pagedigest.org

The important part is the coordination contract: one cheap manifest request can stand in front of thousands of redundant page checks when the publisher and consumer both want to avoid waste.

Why it matters

Automated clients often do expensive inference because websites do not expose enough declared state. PageDigest makes one narrow fact machine-readable: what changed since last time?

That is deliberately modest, but the payoff is large. A crawler can avoid unchanged pages before fetching them, a publisher can reduce repeated bot traffic, and both sides can preserve an auditable record of what was declared and when.

Where to start

Start with the public site for the plain-language explanation, then read the spec if you want the wire format and client behavior details.

• PageDigest site
• Specification
• Releases
• Repository