7. 07. 2025 | Natalie Bezděková

0

• Share with others    

Artificial intelligence needs huge amounts of data to function. To collect this data, AI bots, or crawlers, are programs that automatically crawl web pages and save text, images or other content from them. It is these bots that Cloudflare is now taking a strong stand against, protecting more than 16% of global internet traffic.

Since July, every website using Cloudflare’s services has been given the option to decide whether to allow technology companies to access their content. If the site owner does not wish to do so, Cloudflare will automatically block AI bots from accessing it – and on tens of millions of sites.

Cloudflare thus brings a tool that gives content creators back control over who uses their material. Companies like OpenAI and Google, which train their language models on this data, will have to respect the new rules.

Creators can charge a fee for content

Another new feature is the Pay Per Crawl program. This allows website operators to set the price at which they allow AI systems to access their content. Interested technology companies can review these terms and decide whether to agree to them or give up access. For now, this is a pilot project available to select large publishers and creators.

“We want AI companies to have access to quality content, but also to use it fairly – with the permission of the authors and appropriate remuneration,” Cloudflare said in a statement. Previously, the internet drove visitors directly to the original sources – so creators could gain readers and advertising revenue. Today, however, AI bots often download content without visiting the site, depriving publishers of traffic and earnings.

Roger Lynch, CEO of Condé Nast (publisher of titles such as Vogue and The New Yorker, among others), called Cloudflare’s new move “groundbreaking. “This is an important milestone in restoring a fair environment on the Internet. We are protecting quality journalism and forcing AI companies to pay for the use of content,” he said.

Another step in protecting against AI crawlers

Cloudflare has previously allowed sites to block bots using so-called robots.txt – but this was a voluntary solution that many crawlers did not respect. Starting in 2023, the company has also offered a one-click option to block AI bots across the board. Now it goes even further: this protection is now automatically enabled for all sites using its services.

Cloudflare is setting a new standard for content protection in the AI boom, giving publishers and creators more options to protect their work and potentially monetize it.

Photo source: www.pexels.com

Natalie Bezděková

I am a student of Master's degree in Political Science. I am interested in marketing, especially copywriting and social media. I also focus on political and social events at home and abroad and technological innovations. My free time is filled with sports, reading and a passion for travel.