Use Cloudflare's AI traffic controls before crawler defaults change
Cloudflare now lets all customers manage AI crawler traffic by Search, Agent, and Training behavior, with new onboarding defaults scheduled for September 15, 2026.
Cloudflare has shipped finer AI crawler controls for all customers. Confidence level: confirmed for Cloudflare customers and confirmed for the September 15, 2026 default-change date stated by Cloudflare. The practical change is that site owners can manage AI traffic by behavior: Search, Agent, and Training.
What changed
Cloudflare moved beyond a single AI-bot blocking choice. Customers can now decide how to handle three AI traffic classes: search crawlers, real-time agent activity, and training crawlers. Cloudflare says the controls are available to all customers, including Free plans.
The September 15, 2026 change matters for new domains onboarding to Cloudflare. Cloudflare says new defaults will block Training and Agent bots on pages that display ads, while Search remains allowed. Existing customers can configure the controls now and can opt out of the future defaults.
| AI traffic class | Typical intent | Default direction for new ad pages after Sept. 15 | What to decide |
|---|---|---|---|
| Search | Discovery and answer indexing | Allowed | Whether AI search visibility is worth the crawl cost |
| Agent | Real-time tasks on a user's behalf | Blocked on ad pages | Whether agents create revenue, support load, or abuse risk |
| Training | Model training or fine-tuning | Blocked on ad pages | Whether licensing, consent, or compensation is required |
Key takeaways
- Cloudflare now separates AI traffic into Search, Agent, and Training controls.
- The controls are available to all customers, including Free plans.
- New domains get changed defaults on September 15, 2026 for ad-supported pages.
- Cloudflare is also adding BotBase and Attribution Business Insights for deeper bot visibility.
- Site owners should decide crawler policy by business value, not by bot label alone.
Availability and access
Cloudflare says existing customers can configure the new controls in zone settings now. New domains onboarded after September 15 will receive the changed defaults unless the customer opts out before then.
The broader visibility layer is more segmented. BotBase and Attribution Business Insights are aimed at Bot Management customers, with the Attribution dashboard focused on crawl-to-referral value and operator-level behavior. That means Free customers get the controls, while deeper reporting may depend on plan and product access.
Practical LinkLoot angle
This is a useful update for publishers, affiliates, SaaS content teams, and ecommerce sites that rely on organic discovery but do not want every crawler treated the same way. A search bot that sends customers back is different from a training crawler that consumes pages without referral traffic.
Do a short policy pass before changing anything: identify ad-supported pages, high-value guides, API docs, product pages, and gated resources. Then set AI traffic rules by page value. For example, keep Search open for public buying guides, block Training on ad pages, and watch Agent traffic before making a hard allow or block decision.
For broader workflow planning, LinkLoot's AI agent tools guide is a useful companion because crawler rules now affect how agents discover, fetch, and act on public web content.
What to verify before you act
- Whether your domain is already on Cloudflare or will onboard after September 15, 2026.
- Which pages display ads and will fall under the new default behavior.
- Whether your traffic reports separate AI search, agent fetches, and training crawls.
- Whether blocking a crawler could reduce AI search visibility or useful referral traffic.
- Whether commercial content needs licensing or pay-per-crawl strategy before broad access.
Source check
Confirmed by: Cloudflare's July 1, 2026 blog post confirms the Search, Agent, and Training controls, Free-plan availability, and the product direction. Cloudflare's changelog confirms the September 15 default-change details and the ad-page behavior.
Context: Cloudflare's Attribution Business Insights post explains the crawl-to-referral reporting layer for Bot Management customers. Digital Applied provides independent crawler-traffic context based on Cloudflare Radar data, useful for understanding why the controls matter beyond one product setting.
They let site owners manage AI crawler behavior separately for Search, Agent, and Training traffic.