Googlebot/Nutch-1.7

T
На сайте с 09.03.2016
Offline
14
1257

Заметил бота, который все время по одним урлам ходит. Что это?

Сорри за длинный текст но это лог.

91.121.220.181 site.ru - [27/Sep/2016:00:31:19 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [27/Sep/2016:03:58:37 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.78.198 site.ru - [27/Sep/2016:09:18:38 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.78.198 site.ru - [27/Sep/2016:09:18:38 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.75.9 site.ru - [27/Sep/2016:09:44:29 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.75.9 site.ru - [27/Sep/2016:09:44:29 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

62.210.38.242 site.ru - [27/Sep/2016:13:06:24 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

62.210.38.242 site.ru - [27/Sep/2016:13:06:24 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

62.210.38.242 site.ru - [27/Sep/2016:13:33:34 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.135.78 site.ru - [28/Sep/2016:00:14:19 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.135.78 site.ru - [28/Sep/2016:00:14:19 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.81.33 site.ru - [28/Sep/2016:07:58:41 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [28/Sep/2016:11:55:37 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [28/Sep/2016:11:55:38 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.41.95 site.ru - [28/Sep/2016:12:23:31 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.177 site.ru - [28/Sep/2016:15:45:34 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.40.88 site.ru - [28/Sep/2016:17:06:32 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.88 site.ru - [28/Sep/2016:17:06:33 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [28/Sep/2016:17:31:23 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [28/Sep/2016:17:31:23 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.142.227 site.ru - [28/Sep/2016:21:15:41 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [29/Sep/2016:14:50:21 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.220.121 site.ru - [29/Sep/2016:17:34:16 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.220.121 site.ru - [29/Sep/2016:17:34:16 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.220.121 site.ru - [29/Sep/2016:18:00:33 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.220.121 site.ru - [29/Sep/2016:18:00:33 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.75.46 site.ru - [29/Sep/2016:22:04:26 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.75.46 site.ru - [29/Sep/2016:22:04:26 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.78.198 site.ru - [29/Sep/2016:22:30:15 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [29/Sep/2016:23:40:07 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [29/Sep/2016:23:40:08 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [30/Sep/2016:00:09:03 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [30/Sep/2016:00:09:03 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.136.34 site.ru - [30/Sep/2016:16:06:51 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.136.34 site.ru - [30/Sep/2016:16:06:51 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.135.78 site.ru - [30/Sep/2016:16:32:39 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.135.78 site.ru - [30/Sep/2016:16:32:39 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [30/Sep/2016:17:56:19 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [30/Sep/2016:17:56:19 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [30/Sep/2016:18:19:57 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [30/Sep/2016:18:19:57 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.147.218 site.ru - [30/Sep/2016:22:27:06 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.147.218 site.ru - [30/Sep/2016:22:27:06 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [30/Sep/2016:22:53:28 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [30/Sep/2016:22:53:29 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:05:43:33 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:05:43:34 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.41.177 site.ru - [01/Oct/2016:06:05:55 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.177 site.ru - [01/Oct/2016:06:05:55 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.142.199 site.ru - [01/Oct/2016:09:16:51 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.142.199 site.ru - [01/Oct/2016:09:16:51 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.142.199 site.ru - [01/Oct/2016:09:43:20 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.142.199 site.ru - [01/Oct/2016:09:43:20 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [01/Oct/2016:12:18:24 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [01/Oct/2016:12:18:24 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [01/Oct/2016:12:44:37 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [01/Oct/2016:12:44:37 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:18:28:23 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:18:28:23 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:18:50:11 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:18:50:11 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.81.33 site.ru - [01/Oct/2016:21:39:01 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.81.33 site.ru - [01/Oct/2016:21:39:01 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.81.33 site.ru - [01/Oct/2016:22:07:13 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.81.33 site.ru - [01/Oct/2016:22:07:13 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [01/Oct/2016:23:15:57 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [01/Oct/2016:23:15:57 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.40.88 site.ru - [01/Oct/2016:23:39:35 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.88 site.ru - [01/Oct/2016:23:39:35 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.41.96 site.ru - [02/Oct/2016:04:48:51 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.96 site.ru - [02/Oct/2016:04:48:51 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.41.95 site.ru - [02/Oct/2016:05:10:20 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.95 site.ru - [02/Oct/2016:05:10:20 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

C
На сайте с 21.01.2014
Offline
65
#1

Многие пишут что это зло и надо блокировать

https://devcentral.f5.com/questions/how-to-block-googlebot-nutch-10-bot

Но сам не сталкивался

Mik Foxi
На сайте с 02.03.2011
Offline
905
#2

это не гугл. смотрите whois ip адреса.

Антибот защита сайта от накрутки поведенческих, от спама, взлома и поиска xss, от хит и клик ботов, от парсинга и кражи контента, снижение нагрузки на сервер - https://antibot.cloud/ | Защита яндекс метрики от ботов: https://wmsn.biz/m.php?p=144236 (инструкция по защите от порчи поведенческих факторов).
L
На сайте с 07.12.2007
Offline
351
#3
trillannaa:
Заметил бота, который все время по одним урлам ходит. Что это?

Это какая-то спамилка или парсилка вашего сайта, роботы Google не ходят с хостинга OVH.

Вообще все IP хостингов(кроме вашего) должны быть заблокированы в ipfw

D
На сайте с 28.06.2008
Offline
993
#4
Ladycharm:
Вообще все IP хостингов(кроме вашего) должны быть заблокированы в ipfw

А списком IP сетей не поделитесь?

K
На сайте с 04.10.2016
Offline
7
#5

Где проверить это можно?

Авторизуйтесь или зарегистрируйтесь, чтобы оставить комментарий