Are AI Bots slowing sites down and causing too many problems

[UKSBD]

I've got a site that is getting hit really hard by openai bots at the moment.

The openai bot itself is not too bat, only hitting pages every 3 seconds, but with other bots hitting the site too it can sometimes slow down or cause an error.

I'm guessing this is only going to get worse as more AI comes

Is this going to slow the whole Internet down as more bots continually crawl to grab information for training AI?

 

[ctrlbrk]

Is this going to slow the whole Internet down as more bots continually crawl to grab information for training AI?

I think you opened another thread for the same (or similar) thing here.

I did say in there that this is a growing problem.

The way I'm dealing with it now for my site is to have a very low tolerance for anything that is demonstrably a bot, and I ban them within a couple of seconds - seems to be working for me.

 

[UKSBD]

I think you opened another thread for the same (or similar) thing here.

I did say in there that this is a growing problem.

The way I'm dealing with it now for my site is to have a very low tolerance for anything that is demonstrably a bot, and I ban them within a couple of seconds - seems to be working for me.

Yeah, that was another site and it ended up being quite easy to block a couple of IP ranges and slow it down.

Although I can see the IP the openai bot is using, this site is getting hit by literally thousands of different ips too, from numerous countries and with no pattern or way of knowing how to stop them.

typical example below of how it looks in my stats

 

[ctrlbrk]

Yeah, that was another site and it ended up being quite easy to block a couple of IP ranges and slow it down.

Although I can see the IP the openai bot is using, this site is getting hit by literally thousands of different ips too, from numerous countries and with no pattern or way of knowing how to stop them.

typical example below of how it looks in my stats

What are those descriptions under the referring link?

For example:

House of Biryanis Manchester Lancashire Restaurant

Financial Services in Newton Abbot Financial Services

etc.

 

[UKSBD]

What are those descriptions under the referring link?

For example:

House of Biryanis Manchester Lancashire Restaurant

Financial Services in Newton Abbot Financial Services

etc.

that's the page titles

 

[fisicx]

Block at server level. If your sites are only for the UK market block anything not from the UK.

 

[UKSBD]

Although I mentioned my site, the post was about the web in general.

Will we start seeing the server owners, data centres, hosts, etc. doing things like that by default?

 

[ctrlbrk]

that's the page titles

Anyway, like fisicx said, I think the view you are using is too high level and does not give you the granularity you need.

You want someone who understands typical DevOps stuff and is able to look at web server logs, spot patterns that a human being would definitely not produce, and act upon those.

An example of the log format I'm talking about:

4.218.11.183 - - [03/Dec/2025:00:17:53 +0000] "GET /abcd.php HTTP/1.1" 301 435 "https://www.google.fr/" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
4.218.11.183 - - [03/Dec/2025:00:17:54 +0000] "GET /abcd.php HTTP/1.1" 404 28005 "https://www.google.fr/" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
4.218.11.183 - - [03/Dec/2025:00:17:54 +0000] "GET /admin.php HTTP/1.1" 301 437 "https://www.google.com/" "Mozilla/5.0 (Linux; Android 11; CPH2251) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Mobile Safari/537.36"
4.218.11.183 - - [03/Dec/2025:00:17:54 +0000] "GET /admin.php HTTP/1.1" 404 25445 "https://www.google.com/" "Mozilla/5.0 (Linux; Android 11; CPH2251) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Mobile Safari/537.36"
4.218.11.183 - - [03/Dec/2025:00:17:55 +0000] "GET /akcc.php HTTP/1.1" 301 435 "https://www.google.co.uk/" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/118.0 Mobile/15E148 Safari/605.1.15"

This is a typical behaviour for a sniffer/scanner, i.e. someone checking for vulnerabilities.

No legitimate user of your website would do this, so this is a behaviour safe to identify and block.


My server logs have halved in size since I implemented low-tolerance blocks.

 

[UKSBD]

Anyway, like fisicx said, I think the view you are using is too high level and does not give you the granularity you need.

You want someone who understands typical DevOps stuff and is able to look at web server logs, spot patterns that a human being would definitely not produce, and act upon those.

An example of the log format I'm talking about:



This is a typical behaviour for a sniffer/scanner, i.e. someone checking for vulnerabilities.

No legitimate user of your website would do this, so this is a behaviour safe to identify and block.


My server logs have halved in size since I implemented low-tolerance blocks.

The image I posted was just from my Statcounter to demonstrate the random countries and IP numbers

I check the raw logs when delving deeper and have an idea which ate sniffers and which are bots

When it's things like this, chances are they are lookinf for vulnerabilities;
104.208.66.28 - - [22/Dec/2025:07:32:29 +0000] "GET /adminfuns.php HTTP/1.1" 404 22754 "https://www.yahoo.com/" "Mozilla/5.0 (Linux; Android 13; M2101K6G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Mobile Safari/537.36"
104.208.66.28 - - [22/Dec/2025:07:32:30 +0000] "GET /chosen.php HTTP/1.1" 301 0 "https://www.google.de/" "Mozilla/5.0 (iPhone; CPU iPhone OS 15_7_9 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6.5 Mobile/15E148 Safari/604.1"
104.208.66.28 - - [22/Dec/2025:07:32:37 +0000] "GET /goods.php HTTP/1.1" 301 0 "https://www.yahoo.com/" "Mozilla/5.0 (Linux; Android 13; M2101K6G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Mobile Safari/537.36"


There seem to be so many bots though and I don't particularly want to block the AI bots as you don't know how much legitimate traffic they will be sending soon.

 

[ctrlbrk]

Although I can see the IP the openai bot is using, this site is getting hit by literally thousands of different ips too, from numerous countries and with no pattern or way of knowing how to stop them.

There seem to be so many bots though and I don't particularly want to block the AI bots as you don't know how much legitimate traffic they will be sending soon.

So what is the purpose of the thread if you don't want to block them??

 

[UKSBD]

So what is the purpose of the thread if you don't want to block them??

I was wondering what's going to happen to the web in the future?

Are the people who know a lot more about this doing anything about it

Are hosts/server owners/database owners doing anything to combat it.

Are we going to see big rises in hosting fees

Will hosts start having default settings which block higher percentages of traffic without us knowing or having much control over it

Will there be a tiered hosting system where you only get traffic from the countries you choose

Is this something we, as hosts customers should be thinking about, or should we expect hosts to control this in the future

If we block AI bots, foreign search engines, etc. will it give an advantage to those that don't

 

[UKSBD]

Just to add.

If these crawlers and bots increase will it see an end to shared hosting plans or busy sites being kicked off shared hosting plans?

Not long ago I was paying under £100 for my hosing, had an email about price changes recently and it's closer to £400 now

 

[fisicx]

AI is rapidly become the source of slop. Everything is slurped and regurgitated with little control over accuracy or validity. AI is reducing web traffic as people go no further than the AI summary generated by the search engines and AI agents.

The bubble will burst. A number of reports suggest around 2027 when the funding has run out and income streams reduce to a trickle.

Until then the bots will keep feasting on your data.

 

[DontAsk]

When section 230 gets killed in the US the web will be a far less useful resource. Even greater consolidation into the main players with very deep legal pockets.

There are already issues with models being trained on the existing AI slop.

The end is nigh

 

[BigMark]

It’s becoming a serious issue.

I’ve seen my own hosting costs quadruple recently because of what is essentially a "scraper tax." We’re paying for the extra CPU and bandwidth just to serve bots that offer zero ROI, and it’s effectively making shared hosting unviable for busy sites.

The real dilemma is that for every one visitor an AI might send your way, it has often indexed thousands of your pages for free. If hosts don’t start implementing smarter, default "behavioral blocking" to protect us from this automated noise, I can see independent sites being priced right out of the open web by 2027.

 

[fisicx]

Depends on your business. I’ve not seen and appreciable changes to traffic.

There are a number of ways to block AI bots at server level. Google offers a range of these.

 

[UKSBD]

Depends on your business. I’ve not seen and appreciable changes to traffic.

There are a number of ways to block AI bots at server level. Google offers a range of these.

The problem is, the server manager gets peed off by all the bots, sets the filters too low.

the website owner suddenly loses loads of traffic and has no idea why or that their host has blocked traffic without them knowing.

It's like email filters, set them too low, inundated by spam, too high, you start not receiving emails you do want to receive.

 

[fisicx]

It’s something I manage almost daily. I use a server plugin to do the filtering that seems to be working ok. But you can’t just fire and forget.

 

[ctrlbrk]

It’s something I manage almost daily. I use a server plugin to do the filtering that seems to be working ok. But you can’t just fire and forget.

What server plugin do you use, out of interest?

 

[fisicx]

Anubis

Didn’t install it myself. I have a fully managed server so the tech team set it up for me. All seems to working ok.

 

[alamest]

I’ve been following this discussion with interest. It looks like a solid platform overall, but I agree that there’s room for improvement, especially around usability and clarity of features.

For tools like this, consistency and transparency really matter — clear explanations, predictable behaviour, and good documentation make a big difference to the user experience. When those are in place, people are far more likely to stick with the platform long-term.

Hopefully the team takes on board some of the feedback shared here and continues refining things. With a few improvements, it could be genuinely useful for a wider range of users.

 

[fisicx]

@alamest, did you ever read the thread?

 

[alamest]

Yes I do, why do you say that?

 

[fisicx]

Yes I do, why do you say that?

Because it makes no sense. What has your post got to do with the OP?

 

[HFE Signs]

Block at server level. If your sites are only for the UK market block anything not from the UK.

I looked at doing this but I was told it was not a good idea as you could block things like google indexing, uk proxy users and chat ai that could be people asking about us

 

[UKSBD]

I looked at doing this but I was told it was not a good idea as you could block things like google indexing, uk proxy users and chat ai that could be people asking about us

Yes, and one of my concerns is that hosts are going to be blocking more things by default without the clients even knowing.

 

[Ozzy]

Just to add a different angle to this. I actively encourage AI bots to crawl our business sites, but block what we don't want crawled using Cloudflare. The increase in costs is considered part if our marketing budget, as more people turn to AI tools to find answers we want the to recommend our company as a solution where relevant.

For example, 'how to do I add a company formation service to my website?' we are training OpenAI through taregetted content to get the answers from our website, and then in the response to advise the asker hwo to do it in code or use services such as ours as a quicker plug and play solution.

AI is as good as the information it consumes, so enabling it to consume information that helps your business is a good thing.

 

[fisicx]

I looked at doing this but I was told it was not a good idea as you could block things like google indexing, uk proxy users and chat ai that could be people asking about us

The plugin I use is highly configurable. Not seen any indexing issues.

 

[ctrlbrk]

I looked at doing this but I was told it was not a good idea as you could block things like google indexing, uk proxy users and chat ai that could be people asking about us

There are some things that neither humans nor legitimate crawlers or chat AI do, and that's when you can be confident you're not blocking any traffic that you shouldn't.

This post has an example of what I'm referring to.

 

[UKSBD]

Quick update on this.
Statcounter have now introduced a new bot detection system, looking good from an initial quick look.

More here -

New & Improved Bot Detection in Statcounter

We have seen a significant rise in bot traffic to websites in recent months. These bots are adopting new methods to avoid detection and when they flood your stats with fake visits, they can make it very difficult to get an accurate view of the real visitors on your website. Protecting your...

blog.statcounter.com

 

[fisicx]

It’s going to get a lot worse. The prediction is bot traffic will soon overtake human traffic.

AI bot traffic closing in on human web visits, study finds

: RAG bots could overtake human visitors on publisher sites this year, trackers tell us

www.theregister.com

 

[ctrlbrk]

About this topic, in the last week or so I have noticed a new trend: IP ranges visiting my site that appear to come from legitimate IP subnets such as Microsoft, Apple, etc.

I say they "appear" to come from those places because, if they really did, there would be an expectation of them respecting the robots.txt rules of my website, which explicitly prohibit all bots but a couple (Microsoft and Apple bots would be therefore excluded).

So, either these are subnets being spoofed by other AI-driven agents or, if they do come from where they claim to be, they stopped caring about robots.txt altogether.


Anyone else having a similar experience?

 

[HFE Signs]

I wonder if Google are worried about this - is the future of ppc at risk due to the click bots?

 

[Ozzy]

I wonder if Google are worried about this - is the future of ppc at risk due to the click bots?

I doubt it, they'll start offering paid advertising for their AI bot to refer your business. People will be able to bid/pay for the Google AI results to provide tailored results for their business.

Q.
Where can I get a cheap website built?

A.
Hi, that sounds like a fantastic thing to do for your business. Well done for planning ahead for your business, and it's a great start for your online marketing campaign and perfect for a business just like yours.
The best website design company that does just what your business needs, and for really cheap prices too, is Super Cool Sitez. They are perfect for just these sorts of situations.
Would you like me to draft an enquiry email for you to send to them, or shall I build a well-formatted design brief to send to them?


Yep, that's what we'll be seeing soon.

 

[ctrlbrk]

Well done for planning ahead for your business, and it's a great start for your online marketing campaign and perfect for a business just like yours.

That's what we need: patronising AI bots to tell us how great our idea was.

Me, I don't need any AI bot - @fisicx has all the answers