Set AI crawler rules at the origin: Cloudflare Markdown for Agents now preserves Content-Signal headers
Cloudflare's July 13 update makes origin-set Content-Signal, security, and cache headers survive Markdown for Agents conversion, giving publishers a clearer control point for AI-readable web pages.
Confirmed: Cloudflare updated Markdown for Agents on July 13, 2026 so converted Markdown responses preserve origin-set security, cache, and Content-Signal headers. The important change is control placement: if your origin already sends a content-signal policy, Cloudflare now keeps that policy instead of only applying the service default.
Image: Cloudflare's official Markdown for Agents visual.
What changed
Cloudflare's Markdown for Agents converts HTML pages into text/markdown when a client requests Markdown through HTTP content negotiation. The July 13 update says converted responses now preserve headers that matter for security and caching, including HSTS, CSP, X-Frame-Options, Set-Cookie, CORS headers, Cache-Control, Expires, and Age.
The policy detail matters more than the format detail. Cloudflare says an origin-supplied content-signal header is now authoritative. If the origin does not send one, Cloudflare adds the default Content-Signal: ai-train=yes, search=yes, ai-input=yes.
| Area | Before this update | After this update | What to check |
|---|---|---|---|
| AI use policy | Default Content-Signal behavior was the main visible path | Origin content-signal can carry through conversion | Confirm your intended ai-train, search, and ai-input values |
| Security headers | Conversion could create uncertainty about header survival | Security headers are documented as preserved | Test CSP, HSTS, cookies, and CORS on Markdown responses |
| Cache behavior | Markdown and HTML variants still needed care | Cache headers continue to pass through | Verify Vary: Accept and cache variant handling |
| Links | Directory-style relative links could resolve incorrectly | Cloudflare says relative link resolution now follows RFC 3986 | Test nested docs and blog paths |
Why this is early
The change is confirmed by Cloudflare's changelog and reflected in the current Cloudflare Markdown for Agents documentation. Independent coverage from Search Engine Journal covered the original launch mechanics and warned publishers to review the permissive default Content-Signal values before enabling the feature.
Read the Docs also documents Markdown responses for AI agents and says the feature is powered by Cloudflare. That does not independently confirm the July 13 header-preservation release, but it shows the mechanism is already visible on a real documentation platform.
Key takeaways
- If you use Markdown for Agents, set
content-signalat the origin when you do not want Cloudflare's default AI-use policy. - Treat the Markdown response as a first-class representation of the same page, not a side file for bots.
- Test security headers on Markdown responses, especially if your pages rely on CSP, cookies, CORS, or framing controls.
- Check SEO and AI-search risk separately: content negotiation is cleaner than user-agent sniffing, but publishers still need parity between HTML and Markdown content.
- Use Cloudflare's token headers only for planning. They are useful for agents, but they are not a privacy or licensing control.
Availability and access
Cloudflare's documentation says Markdown for Agents is available on Pro, Business, and Enterprise plans, plus SSL for SaaS customers. Site owners can enable it for a zone, selected subdomains, selected paths, or custom hostnames depending on plan and configuration.
Users can try the feature now where a Cloudflare zone has it enabled by sending an Accept: text/markdown request. For publishers, the practical work is to decide whether the default ai-train=yes, search=yes, ai-input=yes matches the site's content policy.
Practical LinkLoot angle
This is a useful update for publishers, docs teams, SaaS marketers, and developers building AI-readable sites. Markdown for Agents can reduce token waste and make pages easier for agents to parse, but the Content-Signal policy is where the business decision sits.
For teams building AI workflows, pair this with LinkLoot's guide to AI workflow automation. The workflow is simple: publish useful pages, expose clean machine-readable versions, then keep licensing, training, and search permissions explicit at the origin.
What to verify before you act
- Confirm whether Markdown for Agents is enabled for the exact host, path, or custom hostname you care about.
- Send a Markdown request and inspect
content-type,content-signal,Vary, cache headers, CSP, cookies, CORS, and HSTS. - Compare the Markdown body with the HTML page to make sure important disclosures, pricing notes, legal text, and links are still present.
- Decide whether your origin should send
ai-train=no,search=yes,ai-input=yes, or another policy mix. - Watch for Google or other search guidance if your implementation starts serving materially different content to agents than to human users.
Source check
Confirmed by:
- Cloudflare changelog: July 13, 2026 header-preservation update for Markdown for Agents.
- Cloudflare documentation: current behavior for
content-signal, token headers, preserved headers, feature access, and enablement.
Early signal / context:
- Search Engine Journal: independent context on the original Cloudflare launch, default Content-Signal values, and SEO concerns.
- Read the Docs: external adoption context showing Markdown for AI agents on a production documentation platform.
LinkLoot will treat a later Cloudflare policy change, pricing change, or search-engine guidance update as a follow-up trigger, not as proof that today's origin-header behavior should be rewritten.
Cloudflare says converted Markdown responses now preserve origin security, cache, and Content-Signal headers, and relative links resolve more correctly for directory-style URLs.