You are viewing a single comment's thread from:
RE: Google's Bot Imposes a Large Impact on the Performance of SteemYY.com
For me, I don't want to ban entirely the Bot, is there a way to rate limit bots in robots.txt?
For me, I don't want to ban entirely the Bot, is there a way to rate limit bots in robots.txt?
I am not a robots.txt expert... but I think so...
at least you tell them which bots are allowed what pages and others are delayed crawling... or something like that
https://stackoverflow.com/questions/7787264/is-it-possible-to-control-the-crawl-speed-by-robots-txt