
I feel like the owners/operators of traditional websites like this one are now faced with the task of disallowing incoming requests from AI sources. I realize its nearly impossible to do, much like handling the traffic in a denial of service attack was years ago - not to mention that anyone can stand up a standalone LLM and such on their hobby machine. So I wish you luck.
I do rely on this site (forums and lost ski areas) often and appreciate the resource.
I am not sure how well the new terrain bodes for traditional websites when the AI result in google and such is the first thing that gets returned to people (especially younger people)
needawax wrote:
I feel like the owners/operators of traditional websites like this one are now faced with the task of disallowing incoming requests from AI sources. I realize its nearly impossible to do, much like handling the traffic in a denial of service attack was years ago - not to mention that anyone can stand up a standalone LLM and such on their hobby machine. So I wish you luck.
I do rely on this site (forums and lost ski areas) often and appreciate the resource.
I am not sure how well the new terrain bodes for traditional websites when the AI result in google and such is the first thing that gets returned to people (especially younger people)
It's a challenge for sure. I've spent some time trying to figure out how to block AI requests, but unfortunately, as you note, it's virtually impossible to block the ill-behaved ones as they're implemented as botnets and designed to elude such measures.
Here's an example of my page views over the past 18 months:

This would normally look like a very healthy graph, with a rapidly growing audience. But a lot of that growth (and certainly the "spikes" you see - some exceeding 100,000 page view requests per day) are simply AI bots scraping every character of text off every page.
And here is the unique visitors per day graph, which is even more dramatic:
I mean, that's just crazy. Despite that sudden and continuous surge, I've seen traditional advertising revenue (e.g., from Google AdSense) go down, because AI bots don't view ads.
It is possible to tell AI bots not to scrape the site, but they ignore that request. A majority of the "firehose" requests I'm seeing are actually coming from distributed I.P. addresses in China.
This isn't the first challenge smaller, independent web sites have faced over the years, of course.
As you note, Google is trying to provide AI-produced "answers" right at the top of search results, so users never end up clicking on to other web sites. (This is the "no click" problem publishers are fretting about right now.)
But ultimately, it will be a problem if/when independent sites and human journalists disappear. Because AI can only provide answers for what it's trained on. Some of those answers might be "evergreen" (how many ounces are in a gallon?), but for anything that's evolving and changing, they need to keep scraping and training on new data. If there's no motivation for humans to produce that new data, AI results will become stale and outdated and incomplete.
It does seem like a "patron" type model is increasingly becoming the solution for sites like DCSki continue on. And I really appreciate any contributions users can make.
We don't want human reviewers, journalists, and people making comments to disappear. I've made a donation, and thanks for all the work over the years.


Join the conversation by logging in.
Don't have an account? Create one here.