Several artificial intelligence chatbots — including OpenAI’s ChatGPT, Google’s Gemini, and Elon Musk’s Grok — have successfully established themselves in the market.
The data scrapers of these AI companies collect data from millions of websites without permission and without payment, feeding it into their algorithms so they can instantly provide users with the information they seek.
This activity by AI chatbots developed by major tech companies has shaken the economy of online websites.
Before the rise of AI chatbots, websites allowed search engines like Google or Bing to access their content so that they could gain more traffic and earn revenue through advertisements.
But the rapid development of generative AI has enabled companies like Google Gemini and OpenAI ChatGPT to provide information to users directly on their own pages — in their own words, instead of sending users to the original website.
They do this using their web crawlers, which visit millions of websites per second to extract answers.
As a result, website publishers and traditional content creators (like media organizations) are now being stripped of their value by AI crawlers. Human users no longer visit their websites — only chatbots do, and chatbots cannot be shown ads.
Declining website traffic is real
Wikipedia reported last month that human traffic dropped by 8% between 2024 and 2025, due to the rise of AI-generated summaries by search engines.
Cloudflare’s emergency step for website publishers
Matthew Prince, CEO of the American internet service provider Cloudflare, has expressed concern that the new AI-based business model of the internet “does not generate human traffic for websites” at all.
Cloudflare, which processes more than 20% of global internet traffic, introduced a new initiative this summer for website publishers.
The purpose of this initiative is to prevent AI crawlers from accessing website content without permission and without payment.
The company says that over time, they will tighten these controls further to ensure that AI companies cannot exploit website publishers.
This new measure, which applies to more than 10 million websites, has already “caught the attention of major artificial intelligence firms.”
US startup TollBit steps forward for website publishers
On a smaller scale, the American startup TollBit provides tools to online news publishers so they can block, monitor, or even earn revenue from AI crawler traffic.
TollBit is currently working with more than 5,600 websites, including USA Today, Time Magazine, and the Associated Press. The platform allows media organizations to set their own fees for AI companies accessing their content.
Analytics are free for publishers, but AI companies are charged a transaction fee for each content access.
Conclusion
The takeover of the internet by AI crawlers cannot be solved through “isolated measures or by any one company.”
Chatbots have brought a transformation to the online economy whose boundaries will take years to define.
If these bots continue to roam freely across the web, all incentives for creating online content will eventually disappear.
This loss will not only harm humans who want to read this content, but it will also be disastrous for the AI companies themselves — because these companies rely on the original and latest content produced by these very websites.