Bots causing resources errors

[UKSBD]

I'm currently getting "Resource Limit Is Reached" error massages on a few different sites.

Looking at raw logs I'm getting million of hits every couple of hours, the majority from bc. googleusercontent.com with hundreds of different IP numbers
example 124.83.174.34.bc.googleusercontent.com

Is it possible/practical to block all visits from bc.googleusercontent.com rather than the individual IP numbers?

 

[ukwebhosting]

I'm currently getting "Resource Limit Is Reached" error massages on a few different sites.

Looking at raw logs I'm getting million of hits every couple of hours, the majority from bc. googleusercontent.com with hundreds of different IP numbers
example 124.83.174.34.bc.googleusercontent.com

Is it possible/practical to block all visits from bc.googleusercontent.com rather than the individual IP numbers?

Hi,

Could you post the full line of one of the log entries and will see if we can craft a modsecurity rule or something for you.

**edit while I go and see why our renewal for membership paid for in May has downgraded us to a free account!

 

[UKSBD]

Hi,

Could you post the full line of one of the log entries and will see if we can craft a modsecurity rule or something for you.

**edit while I go and see why our renewal for membership paid for in May has downgraded us to a free account!

A couple of examples below (2nd one causing a 508)


169.96.174.34.bc.googleusercontent.com - - [09/Sep/2025:13:06:19 +0100] "GET /bid/282106/ HTTP/1.1" 200 1368 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:136.0) Gecko/20100101 Firefox/136.0"

204.117.174.34.bc.googleusercontent.com - - [09/Sep/2025:13:44:48 +0100] "GET /bid/288635/ HTTP/1.1" 508 224 "https://google.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:135.0) Gecko/20100101 Firefox/135.0"

 

[ukwebhosting]

A couple of examples below (2nd one causing a 508)


169.96.174.34.bc.googleusercontent.com - - [09/Sep/2025:13:06:19 +0100] "GET /bid/282106/ HTTP/1.1" 200 1368 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:136.0) Gecko/20100101 Firefox/136.0"

204.117.174.34.bc.googleusercontent.com - - [09/Sep/2025:13:44:48 +0100] "GET /bid/288635/ HTTP/1.1" 508 224 "https://google.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:135.0) Gecko/20100101 Firefox/135.0"

Quickest way to mitigate things for now and get your sites sorted is to block the range that google cloud customers come from: -

34.128.0.0/10

The notes on that say

Comment: *** The IP addresses under this Org-ID are in use by Google Cloud customers ***

So you are not going to be blocking google search IP's etc.

That is the quickest way to get immediate results and get your sites back running, pitfalls are if your sites are for example Wordpress and a plugin needs data from something hosted on google cloud that connects to your site directly but that should be remote enough.

Or some other scenarios but it allows you to get your sites working and step back and look at the logs.

Comment: Direct all spam and abuse complaints to
Comment: https://support.google.com/code/go/gce_abuse_report

Send the logs to them that include their IPs

 

[fisicx]

There have been a number of articles on TheRegister about this. It’s all bots feeding the LLMs. It’s not just you, any number of content rich sites are getting hammered and there is no single method to block them all. It’s like a game of whack-a-mole.

 

[TCH]

your 124.83.174.34 seems to be
<a href="https://www.showmyip.com/bulk-ip-lookup/" target="_blank">Yahoo Japan</a>, perhaps you can block the whole country for a few days

 

[ukwebhosting]

your 124.83.174.34 seems to be
<a href="https://www.showmyip.com/bulk-ip-lookup/" target="_blank">Yahoo Japan</a>, perhaps you can block the whole country for a few days

Hi,

You read it backwards in this instance.

dig -x 34.174.83.124

;124.83.174.34.in-addr.arpa. IN PTR

;; ANSWER SECTION:

124.83.174.34.in-addr.arpa. 120 IN PTR 124.83.174.34.bc.googleusercontent.com.

 

[ukwebhosting]

There have been a number of articles on TheRegister about this. It’s all bots feeding the LLMs. It’s not just you, any number of content rich sites are getting hammered and there is no single method to block them all. It’s like a game of whack-a-mole.

Hi,

As long as they are all within

34.128.0.0 - 34.191.255.255

Then blocking this subnet at your firewall will block them.

34.128.0.0/10

 

[fisicx]

They are all over the place. It’s as if they are IP jumping to get around the blocks.

Last time I checked the server there were about 16 different AI slurping bots across a whole range of IP addresses.

 

[UKSBD]

There are 100's possibly 1000's of different IP numbers.

It's not taking the sites down all the time, just sporadically getting the error message

Not just one host either, multiple sites on different hosts.

Going to make a list of all the IP numbers over next day or two

 

[ctrlbrk]

There have been a number of articles on TheRegister about this. It’s all bots feeding the LLMs. It’s not just you, any number of content rich sites are getting hammered and there is no single method to block them all. It’s like a game of whack-a-mole.

is this one of the articles you saw?

Anubis: Fighting off the hordes of LLM bot crawlers

Updated: Using proof of work to block the web-crawlers of 'AI' companies

www.theregister.com

 

[ctrlbrk]

This is a growing problem.

I used to have my site attacked by scrapers once a month - they would not rate-limit their crawlers and, sure enough, they would overpower Apache and crash the server.

Small businesses with a modestly-resourced server such as mine are at most risk of crashes.

I have devised a few rules that have been keeping the bots at bay, until now. They would block by IP address mostly.


But just a few days ago the server got swamped by single-instance IP address requests, which make blocking by IP futile.

In that instance, most of the requests originated from South America, which is not our target base.


However toying with the idea of blocking an entire country, or even a region, seems overkill - both in terms of effort required to implement it, and in terms of resources (every IP address request would have to be checked against a huge list of subnet masks).

In my earlier post I saw there is a tool called Anubis which seem promising. However there doesn't appear to be a version yet compatible with Debian, which is what my server runs on.


I'd be interested, like OP, to see how others deal with this.

I don't want to start spending money on CloudFlare or similar solutions - our model doesn't justify that type of expense yet.

 

[fisicx]

Read the article and comments:

AI crawlers destroying websites in hunger for content

Opinion: But the cure may ruin the web....

www.theregister.com

 

[UKSBD]

I've filtered through the IP number and after removing duplicates have narrowed down to 500+ out of 500,000+ that contained bc.googleusercontent.com

Interestingly 99% appear to end 174.34

ie.

0.195.174.34
0.196.174.34
0.4.174.34
0.69.174.34
0.77.174.34
0.79.174.34
1.90.174.34
10.148.174.34
10.252.174.34
10.66.174.34

Looking at the top one here https://ipinfo.io/34.174.195.0

They all appear to be from Google hosting

If I block all of 34.174 will it stop Google indexing the sites?

 

[ThatDevAaron]

Free Cloudflare - simple as - pm me your whatsapp number ill help you configure FOR FREE. dont pay anyone or anything.

if system resources are a problem, cf is the best for you, its all done before anything touches your network.

 

[UKSBD]

Free Cloudflare - simple as - pm me your whatsapp number ill help you configure FOR FREE. dont pay anyone or anything.

if system resources are a problem, cf is the best for you, its all done before anything touches your network.

Thanks,

One of my hosts is currently looking in to it

He says;

"The quota and bandwidth limits don't seem to be being hit, however, the cloudlinux manager is showing a 400% SQL hit, are you getting hit by AI bots?"


Pretty sure that is what is happening, getting over 1m hits every 2 hours

 

[ukwebhosting]

Thanks,

One of my hosts is currently looking in to it

He says;

"The quota and bandwidth limits don't seem to be being hit, however, the cloudlinux manager is showing a 400% SQL hit, are you getting hit by AI bots?"


Pretty sure that is what is happening, getting over 1m hits every 2 hours

Ask them to block this subnet in your servers firewall if possible

34.128.0.0/10

It will not block google search they are on a completely different subnet.

 

[ukwebhosting]

Ask them to block this subnet in your servers firewall if possible

34.128.0.0/10

It will not block google search they are on a completely different subnet.

If they cannot do it at a server level you will have to do it in each sites .htaccess

 

[fisicx]

are you getting hit by AI bots.

Yup. That’s what’s happening. And they ignore anything in your robots.txt

 

[Kerwin]

https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/