Big rises in hits from the Petal Search Bot

[UKSBD]

Is anyone else seeing PetalBot hitting their sites more and more?

 

[SEODEV#338055]

If PetalBot crawling adversely affects your server performance you can use robots exclusion protocol [robots.txt] to prevent PetalBot accessing your website

User-agent: PetalBot
Disallow: /

 

[Paul Carmen]

If PetalBot crawling adversely affects your server performance you can use robots exclusion protocol [robots.txt] to prevent PetalBot accessing your website

It's a good suggestion, but unfortunately PetalBot seems to completely ignore the robots.txt settings!

We've seen an increase in traffic from this bot, with regular crawls now, and it's a bit of a pain as its particularly intrusive (see above) and more frequent than most.

We did a bit of research on it a while back and its apparently Huawei’s new search engine bot, they are building their own search engine app, and potentially launching as a competitor to Google (they were kicked out of the entire Google App infrastructure, so need their own tools and Apps on their phones). Various people made an effort before to block the AspiegelBot, which was their original bot, so they launched a new one...

 

[ctrlbrk]

We did a bit of research on it a while back and its apparently Huawei’s new search engine bot

So it's a Chinese search engine that's not respecting robots.txt protocols?

What else can be done ?

 

[UKSBD]

So it's a Chinese search engine that's not respecting robots.txt protocols?

What else can be done ?

Just let it do it's thing unless it causes problems.

Not really caused me any problems apart from skewing stats in statcounter

I certainly wouldn't block it, 3rd most popular mobile phone in the UK

 

[Paul Carmen]

So it's a Chinese search engine that's not respecting robots.txt protocols?

What else can be done ?

We made the choice not to do anything, as we have overhead on all our servers and VPS setup, so basically it's not an issue for us.

They also sell a lot of phones in the UK and Europe and their share is rising quickly, so if you block the bot you potentially don't feature in search results if Huawei phone owners use the native Huawei search App!

If you really want to block it, you'll need to do some work and testing, as robots.txt is more like a gentleman's agreement. Baidu and other Chinese bots have never really taken any notice of it, certainly not in the way Google does. You're effectively looking at blocking it by IP address or something along those lines via the server/firewall setup.

 

[SEODEV#338055]

Here's where I sourced the original solution:

webmaster . petalsearch . com / site / petalbot

This is what the site says verbatim:

1.3 How to Block PetalBot from Visiting Your Site

PetalBot complies with the Internet robots protocol. You can use the robots.txt file to completely prevent PetalBot from accessing your website, or to prevent PetalBot from accessing some files on your website.

Note: Banning PetalBot from accessing your site will make the pages on your site and all search engine services provided by Petal unsearchable in the Petal search engine.

setup recommendations

You can set different crawling rules according to different user-agents of each product, and you can directly prevent the crawling of PetalBot. The following robots can prevent Petal crawling or conditional allow:

User-agent: PetalBot

Disallow: /

User-agent: PetalBot

Allow: /w/api/

Disallow: /trap/

 

[ctrlbrk]

Well, the webpage says one thing, Paul says another.

Which is it?

 

[Paul Carmen]

Maybe both, or they might have made changes, you'd need to try it and see if you want to block it (there's noise on sites/forums about it still ignoring robots.txt). We initially reviewed AspiegelBot, which was the original bot crawler from Huawei, this seemed to ignore it completely for weeks when we blocked it.

They then changed to PetalBot and there was much more info from Huawei in the public domain, so we chose to allow it to crawl all sites due to their market share in the UK.