I watched the State of the Union, but found the State of the Bots more engaging.

The State of the Bots is a report published by Tollbit CEO Toshit Panigrahi that assesses the volume and impact of web scraping on publishers.

It illustrates the potentially devastating impact of the combination of extracting information from web sites and reformulating it into easy-to-digest answers for generative AI. 

According to the report:

–The volume of web scraping doubled between the third quarter and fourth quarter of 20024.

–During that same time, the amount of unauthorized web scraping increased by more than 40%.

–Even when blocked, apps like Perplexity used unidentified agents to scrape sites.

All that adds up to one devastating conclusion: 

“AI chat bots on average drive referral traffic at a rate that is 96% lower than traditional Google search.”

In plain English that means online publishers can expect to receive 96% fewer clicks on their web sites in a new world where most internet queries are answered using ChatGPT.

A friend of mine compared this to the impact of Uber on taxis in New York. Before Uber, taxis were plentiful and cheap. Afterwards they largely disappeared. Uber may be convenient, but its more expensive and it wiped out the value of owning a taxi license.

In the case of publishers, it’s not clear how they will pay writers since the current online ecosystem for advertising-supported content depends on traffic and the bulk of visits come from Google .

Tech companies have generally balked at compensating publishers for content – either to be used in training models or to be accessed as a reference to answer questions. 

Analyst Matthew Scott Goldstein calls it “The Greatest Heist on Earth.”  

He wrote: “Powerful tech companies, from social media giants to artificial intelligence startups, are quietly siphoning off the value from content creators across all industries.”

Goldstein says safeguarding copyright is essential and steps need to be taken now. 

“Once AI models are trained on stolen content, the damage is irreversible. The longer this goes unchecked, the harder it will be to claw back rights.”

It’s possible some of the damage will be mitigated by new search designs Google is unveiling which it says will make it more attractive for people to click through to web sites.

What the State of the Bot report makes clear is that that future is coming fast.