Googlebot/Nutch-1.7

T

14

trillannaa

2 октября 2016, 05:22

1282

Заметил бота, который все время по одним урлам ходит. Что это?

Сорри за длинный текст но это лог.

91.121.220.181 site.ru - [27/Sep/2016:00:31:19 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [27/Sep/2016:03:58:37 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.78.198 site.ru - [27/Sep/2016:09:18:38 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.78.198 site.ru - [27/Sep/2016:09:18:38 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.75.9 site.ru - [27/Sep/2016:09:44:29 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.75.9 site.ru - [27/Sep/2016:09:44:29 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

62.210.38.242 site.ru - [27/Sep/2016:13:06:24 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

62.210.38.242 site.ru - [27/Sep/2016:13:06:24 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

62.210.38.242 site.ru - [27/Sep/2016:13:33:34 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.135.78 site.ru - [28/Sep/2016:00:14:19 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.135.78 site.ru - [28/Sep/2016:00:14:19 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.81.33 site.ru - [28/Sep/2016:07:58:41 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [28/Sep/2016:11:55:37 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [28/Sep/2016:11:55:38 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.41.95 site.ru - [28/Sep/2016:12:23:31 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.177 site.ru - [28/Sep/2016:15:45:34 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.40.88 site.ru - [28/Sep/2016:17:06:32 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.88 site.ru - [28/Sep/2016:17:06:33 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [28/Sep/2016:17:31:23 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [28/Sep/2016:17:31:23 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.142.227 site.ru - [28/Sep/2016:21:15:41 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [29/Sep/2016:14:50:21 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.220.121 site.ru - [29/Sep/2016:17:34:16 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.220.121 site.ru - [29/Sep/2016:17:34:16 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.220.121 site.ru - [29/Sep/2016:18:00:33 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.220.121 site.ru - [29/Sep/2016:18:00:33 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.75.46 site.ru - [29/Sep/2016:22:04:26 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.75.46 site.ru - [29/Sep/2016:22:04:26 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.78.198 site.ru - [29/Sep/2016:22:30:15 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [29/Sep/2016:23:40:07 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [29/Sep/2016:23:40:08 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [30/Sep/2016:00:09:03 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [30/Sep/2016:00:09:03 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.136.34 site.ru - [30/Sep/2016:16:06:51 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.136.34 site.ru - [30/Sep/2016:16:06:51 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.135.78 site.ru - [30/Sep/2016:16:32:39 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.135.78 site.ru - [30/Sep/2016:16:32:39 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [30/Sep/2016:17:56:19 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [30/Sep/2016:17:56:19 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [30/Sep/2016:18:19:57 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [30/Sep/2016:18:19:57 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.147.218 site.ru - [30/Sep/2016:22:27:06 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.147.218 site.ru - [30/Sep/2016:22:27:06 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [30/Sep/2016:22:53:28 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.143.113 site.ru - [30/Sep/2016:22:53:29 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:05:43:33 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:05:43:34 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.41.177 site.ru - [01/Oct/2016:06:05:55 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.177 site.ru - [01/Oct/2016:06:05:55 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.142.199 site.ru - [01/Oct/2016:09:16:51 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.142.199 site.ru - [01/Oct/2016:09:16:51 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.142.199 site.ru - [01/Oct/2016:09:43:20 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.142.199 site.ru - [01/Oct/2016:09:43:20 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [01/Oct/2016:12:18:24 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [01/Oct/2016:12:18:24 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [01/Oct/2016:12:44:37 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.119.222 site.ru - [01/Oct/2016:12:44:37 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:18:28:23 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:18:28:23 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:18:50:11 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.178 site.ru - [01/Oct/2016:18:50:11 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

91.121.81.33 site.ru - [01/Oct/2016:21:39:01 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.81.33 site.ru - [01/Oct/2016:21:39:01 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

91.121.81.33 site.ru - [01/Oct/2016:22:07:13 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

91.121.81.33 site.ru - [01/Oct/2016:22:07:13 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [01/Oct/2016:23:15:57 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.86 site.ru - [01/Oct/2016:23:15:57 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.40.88 site.ru - [01/Oct/2016:23:39:35 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.40.88 site.ru - [01/Oct/2016:23:39:35 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

151.80.41.96 site.ru - [02/Oct/2016:04:48:51 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.96 site.ru - [02/Oct/2016:04:48:51 +0700] "GET / HTTP/1.0" 200 12641 "-" "Googlebot/Nutch-1.7"

151.80.41.95 site.ru - [02/Oct/2016:05:10:20 +0700] "GET /robots.txt HTTP/1.0" 200 67 "-" "Googlebot/Nutch-1.7"

151.80.41.95 site.ru - [02/Oct/2016:05:10:20 +0700] "GET /country/6 HTTP/1.0" 200 12226 "-" "Googlebot/Nutch-1.7"

C

65

chos

2 октября 2016, 06:19

#1

Многие пишут что это зло и надо блокировать

https://devcentral.f5.com/questions/how-to-block-googlebot-nutch-10-bot

Но сам не сталкивался

1218

Mik Foxi

2 октября 2016, 09:52

#2

это не гугл. смотрите whois ip адреса.

Антибот, антиспам, веб фаервол, защита от накрутки поведенческих: https://antibot.cloud/ (8 лет на защите сайтов). Форум на замену серчу: https://foxi.biz/

L

351

Ladycharm

2 октября 2016, 12:40

#3

trillannaa:
Заметил бота, который все время по одним урлам ходит. Что это?

Это какая-то спамилка или парсилка вашего сайта, роботы Google не ходят с хостинга OVH.

Вообще все IP хостингов(кроме вашего) должны быть заблокированы в ipfw

D

1114

Dram

2 октября 2016, 14:11

#4

Ladycharm:
Вообще все IP хостингов(кроме вашего) должны быть заблокированы в ipfw

А списком IP сетей не поделитесь?

K

7

Kacper

4 октября 2016, 23:22

#5

Где проверить это можно?

VK приобрела 70% в структуре компании-разработчика red_mad_robot

Что делать, чтобы попасть в ответы Google Bard