Cloudflare Introduces Tool to Block AI Bots from Scraping Web Content

Cloudflare Inc (NYSE: NET) on Wednesday announced a new feature aimed at protecting website owners from unauthorized content scraping by AI bots. This comes in response to growing concerns among content creators about the unchecked harvesting of online data for AI training purposes.

The new tool, available to all Cloudflare customers including those on the free tier, offers a simple “one-click” option to block AI bots from accessing websites. This enhanced protection uses machine learning and digital fingerprinting to identify and prevent bot activity, providing a more robust defense compared to traditional methods like robots.txt files.

Cloudflare’s decision to implement this feature stems from widespread customer demand and significant AI bot activity observed across its network. The company reported that in June, AI bots accessed 39% of the top one million sites served by Cloudflare, despite previous blocking attempts.

“We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly,” the company wrote. “To help, we’ve added a brand new one-click to block all AI bots. It’s available for all customers, including those on the free tier. To enable it, simply navigate to the Security > Bots section of the Cloudflare dashboard, and click the toggle labeled AI Scrapers and Crawlers.”

Among the most active AI crawlers identified were Bytespider, GPTBot, and ClaudeBot. These bots are associated with various AI companies and are used for tasks such as training large language models and supporting chatbot services.


Information for this story was found via the sources and companies mentioned. The author has no securities or affiliations related to the organizations discussed. Not a recommendation to buy or sell. Always do additional research and consult a professional before purchasing a security. The author holds no licenses.

Video Articles

2026 Could Be Gold’s Biggest Year Yet!? | Ryan King – Equinox Gold

Gold Is Screaming Higher While Currencies Burn | Simon Ridgway – Rackla Metals

We Have the Highest-Grade Antimony Deposit in North America!? | Jim Atkinson -Antimony Resources

Recommended

Canadian Copper Secures $8 Million Lead Order From Ocean Partners As Part Of Larger Funding Round

Northern Superior Expands Philibert With 350 Metre Step Out Testing 1.10 g/t Gold Over 25.5 Metres

Related News

Cloudflare Accuses AI Startup Perplexity of Using ‘Stealth Crawlers’ to Bypass Website Blocks

Internet infrastructure giant Cloudflare accused AI search startup Perplexity on Monday of deploying undeclared “stealth...

Tuesday, August 5, 2025, 09:31:00 AM