Форум Сайтостроение Администрирование серверов

Падает ВПС, помогите понять причину.

D
На сайте с 28.06.2008
Offline
1101
Dram
5140

Есть слабенький ВПС (1 ядро, 1 Гиг памяти. Дебиан, Апач, пхп 5,6, Эджинкс) у Хедзнера, на нем два тестовых сайта без посейщаемости и скприт Сергеича для блокировка шлака в адсент.

Все работало год примерно и кушать не просило. И вот вчера впс начал падать.

вот ошибки из логов

апач

[crit] Memory allocation failed, aborting process.
[Tue Aug 14 08:02:45.645931 2018] [core:notice] [pid 25807] AH00052: child pid 27402 exit signal Aborted (6)
[Tue Aug 14 08:03:19.984891 2018] [mpm_prefork:error] [pid 25807] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

я так понимаю что-то сожрало всю память. В пхп ини было всегда 128 мб, увеличение до 256 не помогло. Щас уменьшил до 64... наблюдаю.

Мускуль

2018-08-14 8:07:17 140432750342720 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2018-08-14 8:07:17 140432750342720 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: The InnoDB memory heap is disabled
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: Using Linux native AIO
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: Using SSE crc32 instructions
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: Completed initialization of buffer pool
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: Highest supported file format is Barracuda.
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: The log sequence number 2128676 in ibdata file do not match the log sequence number 2418164 in the ib_logfiles!
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: 128 rollback segment(s) are active.
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: Waiting for purge to start
2018-08-14 8:07:17 140432750342720 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 2418164
2018-08-14 8:07:18 140432130402048 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-08-14 8:07:18 140432750342720 [Note] Plugin 'FEEDBACK' is disabled.
2018-08-14 8:07:18 140432750342720 [Note] Recovering after a crash using tc.log
2018-08-14 8:07:18 140432750342720 [Note] Starting crash recovery...
2018-08-14 8:07:18 140432750342720 [Note] Crash recovery finished.
2018-08-14 8:07:18 140432750342720 [Note] Server socket created on IP: '127.0.0.1'.
2018-08-14 8:07:18 140432750342720 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.26-MariaDB-0+deb9u1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Debian 9.1
2018-08-14 8:07:19 140432749709056 [ERROR] mysqld: Table './test/jos_content' is marked as crashed and should be repaired
2018-08-14 8:07:19 140432749709056 [Warning] Checking table: './test/jos_content'
2018-08-14 8:07:20 140432749709056 [ERROR] mysqld: Table './test/jos_users' is marked as crashed and should be repaired
2018-08-14 8:07:20 140432749709056 [Warning] Checking table: './test/jos_users'
D
На сайте с 28.06.2008
Offline
1101
Dram
#1

прошло 4 часа ВПС опять завис.

ошибки теже.

в логе ошибок апач:

[Tue Aug 14 11:09:32.275518 2018] [mpm_prefork:error] [pid 25002] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

через пару минут в логе мускуля

2018-08-14 11:15:56 140024221164096 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

и далее тоже что и выше



---------- Добавлено 14.08.2018 в 12:21 ----------

Сис лог на момент падения

Aug 14 11:14:39 Debian-93-stretch-64-minimal CRON[27489]: (CRON) error (can't fork)
Aug 14 11:15:54 Debian-93-stretch-64-minimal keyboard-setup.sh[478]: cannot open file /tmp/tmpkbd.abk5Bo
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Load/Save Random Seed.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Starting Flush Journal to Persistent Storage...
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Starting udev Kernel Device Manager...
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Flush Journal to Persistent Storage.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started udev Coldplug all Devices.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started udev Kernel Device Manager.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Set the console keyboard layout.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Reached target Local File Systems (Pre).
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Reached target Local File Systems.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Starting Raise network interfaces...
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Starting Create Volatile Files and Directories...
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Starting Set console font and keymap...
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Set console font and keymap.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Create Volatile Files and Directories.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Starting Network Time Synchronization...
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Entropy daemon using the HAVEGE algorithm.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Update UTMP about System Boot/Shutdown.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Network Time Synchronization.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Reached target System Initialization.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Listening on D-Bus System Message Bus Socket.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Reached target Sockets.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Daily Cleanup of Temporary Directories.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Reached target Basic System.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Regular background program processing daemon.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Starting Login Service...
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started irqbalance daemon.
Aug 14 11:15:54 Debian-93-stretch-64-minimal cron[677]: (CRON) INFO (pidfile fd = 3)
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started D-Bus System Message Bus.
Aug 14 11:15:54 Debian-93-stretch-64-minimal cron[677]: (CRON) INFO (Running @reboot jobs)
Aug 14 11:15:54 Debian-93-stretch-64-minimal /usr/sbin/irqbalance: Balancing is ineffective on systems with a single cpu. Shutting down
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Starting System Logging Service...
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Starting LSB: Load kernel modules needed to enable cpufreq scaling...
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Reached target System Time Synchronized.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Clean PHP session files every 30 mins.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: apt-daily.timer: Adding 7h 8min 43.036909s random time.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Daily apt download activities.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: apt-daily-upgrade.timer: Adding 43min 19.972167s random time.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Daily apt upgrade and clean activities.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Reached target Timers.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started Login Service.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started LSB: Load kernel modules needed to enable cpufreq scaling.
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Starting LSB: set CPUFreq kernel parameters...
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started LSB: set CPUFreq kernel parameters.
Aug 14 11:15:54 Debian-93-stretch-64-minimal liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="690" x-info="http://www.rsyslog.com"] start
Aug 14 11:15:54 Debian-93-stretch-64-minimal systemd[1]: Started System Logging Service.
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] Linux version 4.9.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.65-3 (2017-12-03)
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-4-amd64 root=UUID=ece287f6-9958-41f5-88ac-c8645c17c284 ro nomodeset consoleblank=0 elevator=noop
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] x86/fpu: Using 'eager' FPU context switches.
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] e820: BIOS-provided physical RAM map:
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003ffdbfff] usable
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] BIOS-e820: [mem 0x000000003ffdc000-0x000000003fffffff] reserved
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] NX (Execute Disable) protection: active
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] SMBIOS 2.8 present.
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] DMI: Hetzner vServer, BIOS 1.10.2 04/01/2014
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] Hypervisor detected: KVM
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] e820: last_pfn = 0x3ffdc max_arch_pfn = 0x400000000
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] MTRR default type: write-back
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] MTRR fixed ranges enabled:
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] 00000-9FFFF write-back
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] A0000-BFFFF uncachable
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] C0000-FFFFF write-protect
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] MTRR variable ranges enabled:
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] 0 base 0080000000 mask FF80000000 uncachable
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] 1 disabled
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] 2 disabled
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] 3 disabled
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] 4 disabled
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] 5 disabled
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] 6 disabled
Aug 14 11:15:54 Debian-93-stretch-64-minimal kernel: [ 0.000000] 7 disabled
Plazik
На сайте с 29.07.2008
Offline
161
Plazik
#2

Смотрите в htop, что у вас память кушает.

danforth
На сайте с 18.12.2015
Offline
153
danforth
#3 

dmesg | egrep -i 'killed process'
Junior Web Developer
Mik Foxi
На сайте с 02.03.2011
Offline
1076
Mik Foxi
#4

Пришли боты парсеры, нагрузили парсингом память. Изучите еще аксесс логи nginx

Антибот, антиспам, веб файрвол, защита от накрутки поведенческих: https://antibot.cloud/ + партнерка, до 40$ с продажи.
D
На сайте с 28.06.2008
Offline
1101
Dram
#5
danforth:
dmesg | egrep -i 'killed process'

Сразу после ребута команда ничего не показала.

Я уже поставил перезагружать апач по крону раз в час. Это помогло продержаться пол дня. Но сейчас он опять завис.

Акцесс лог обычный, никакого наплыва ботов или парсинга там нет

LA все время не превышает 0,1

Я склоняюсь к тому что скрипт Сергеича блокирующий шлак в Адсент плавно отжирает память. Но не понятно почему он нормально работал год и вдруг ВПС начал падать.

Включил htop - вижу запуск этого скрипта, но он кушает всего 4% памяти.... не понимаю короч. (пометил его на скрине)

jpg 176386.jpg
danforth
На сайте с 18.12.2015
Offline
153
danforth
#6

VPS не должна погибать из-за недостатка памяти, должен приходить oom и убивать процесс, если это не какой-то уберхайлод где все стало раком. Если у вас в логах нету oom, мне кажется, что вы попали на какой-то дичайший оверселл. Возможно так-же проблемы железа.

Без системы мониторинга сложно что-то сказать.

D
На сайте с 28.06.2008
Offline
1101
Dram
#7

Установил мунин. Ночью впс опять упал. Дерка в графиках это падение.

Пока вижу две аномалии - на графике дисков и базы.

Напоминаю у меня 1 гиг памяти, вот конфиг мускуля

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/english
old_passwords = 0
bind-address = 127.0.0.1

skip-external-locking

max_allowed_packet = 16M
key_buffer_size = 16M
innodb_buffer_pool_size = 512M
innodb_file_per_table = 1
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 0

max_connections = 132

query_cache_size = 0

slow_query_log = /var/log/mysql/mysql-slow.log
long_query_time = 1

expire_logs_days = 10
max_binlog_size = 100M

[mysqldump]
quick
quote-names
max_allowed_packet = 16M
png diskstats_iops-day.png
png memory-day.png
D
На сайте с 28.06.2008
Offline
1101
Dram
#8

Пока сделал такие изменения:

max_connections = 50
thread_cache_size = 40
query_cache_size = 16M

уменьшил max_connections, включил квери кеш и thread_cache_size

Evas EvaSystems
На сайте с 31.05.2012
Offline
104
Evas EvaSystems
#9

Вам действительно нужен такой innodb_buffer_pool_size? или вы его просто поставили?

А вообще рекомендую перевести таблички в MyISAM и отключить InnoDB вовсе, если позволяет ситуация. Включить Swap, если он отключен.

Системный администратор Linux. Настройка, сопровождение и оптимизация серверов. Отзывы - searchengines.guru/ru/forum/1017473
D
На сайте с 28.06.2008
Offline
1101
Dram
#10

Пол дня полет нормальный... вероятно все же мускуль был неправильно настроен. Точнее конфиг по дефолту - зло, даже для тестовых сайтов.

Авторизуйтесь или зарегистрируйтесь, чтобы оставить комментарий