Cloudflare Inc (NYSE: NET) on Wednesday announced a new feature aimed at protecting website owners from unauthorized content scraping by AI bots. This comes in response to growing concerns among content creators about the unchecked harvesting of online data for AI training purposes.
The new tool, available to all Cloudflare customers including those on the free tier, offers a simple “one-click” option to block AI bots from accessing websites. This enhanced protection uses machine learning and digital fingerprinting to identify and prevent bot activity, providing a more robust defense compared to traditional methods like robots.txt files.
Cloudflare’s decision to implement this feature stems from widespread customer demand and significant AI bot activity observed across its network. The company reported that in June, AI bots accessed 39% of the top one million sites served by Cloudflare, despite previous blocking attempts.
“We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly,” the company wrote. “To help, we’ve added a brand new one-click to block all AI bots. It’s available for all customers, including those on the free tier. To enable it, simply navigate to the Security > Bots section of the Cloudflare dashboard, and click the toggle labeled AI Scrapers and Crawlers.”
Among the most active AI crawlers identified were Bytespider, GPTBot, and ClaudeBot. These bots are associated with various AI companies and are used for tasks such as training large language models and supporting chatbot services.
Information for this story was found via the sources and companies mentioned. The author has no securities or affiliations related to the organizations discussed. Not a recommendation to buy or sell. Always do additional research and consult a professional before purchasing a security. The author holds no licenses.
