ipアドレス別でサーバーに接続している数を調べるコマンド
-----------------------------------------------------------
netstat -tan | awk '{print $5}' | cut -d : -f1 | sort -n | uniq -c | sort -n
-----------------------------------------------------------
オプションの説明
a → すべてのネットワーク接続を表示する
n → DSN逆引きを行わない（ipアドレスで表示）
t → TCPのみを表示する
awk '{print $5}' → 5フィールド目を取り出す
cut -d : -f1 → デリミタ（区切り文字）「:」を指定。フィールド１を取り出す
sort -n → データを文字ではなく、数値として並び変える。次のuniq が隣り合った行しか比較しないので必要
uniq -c → 各行の前に出現回数を出力する。隣り合った行しか比較しない
sort -n → データを文字ではなく、数値として並び変える。出現回数順に並べ替える
カテゴリー：linux
現在の設定を確認
--------------------------------------------
# sysctl vm.panic_on_oom kernel.panic
vm.panic_on_oom = 0
kernel.panic = 0
--------------------------------------------
vm.panic_on_oom
OOM Killer が実行される際に、カーネルパニックを起こさせるかを制御するパラメータ
0 → カーネルパニックしない
1 → カーネルパニックする。但し cgroup 制限により物理メモリがまだ残っている場合にはカーネルパニックしない。
2 → 必ずカーネルパニックする
カーネルパニックとは、オペレーティングシステム（OS）のカーネル部分において、何らかの理由で致命的なエラーが発生し、安全に復旧することができなくなった状態。
kernel.panic = 0 → なにもしない
kernel.panic = 30 → 30秒後にリブート
--------------------------------------------
sysctl -a | grep vm.over
--------------------------------------------
現状
--------------------------------------------
vm.overcommit_memory=1
vm.panic_on_oom=1
kernel.panic=10
--------------------------------------------
vm.overcommit_memory：オーバーコミットするかどうかの設定
vm.overcommit_memory = 0 → オーバーコミット有効。実メモリの大きさまで割り当てる。
vm.overcommit_memory = 1 → オーバーコミット有効。実メモリ以上に割り当てる。
sysctl.conf を下記のように変更
#oom になったらカーネルパニックにさせてサーバーを再起動
--------------------------------
vm.overcommit_memory=1
vm.panic_on_oom=1
#15秒後に再起動
kernel.panic=15
--------------------------------
設定を反映
--------------------------------
sysctl -p
--------------------------------
カテゴリー：CentOS, サーバートラブル・つながらない, サーバー設定
monit を導入する
ダウンロードできるか確認
-----------------------------------
yum list available | grep monit
-----------------------------------
monit をインストール
-----------------------------------
yum install monit
-----------------------------------
設定ファイルをいじるので原本をコピー
-----------------------------------
cp -p /etc/monit.conf /etc/monit.conf_org
-----------------------------------
※centOS7.x系では設定ファイル名は monitrc
起動しているか（monitの状態確認）
-----------------------------------
monit summary
-----------------------------------
もしくは
-----------------------------------
monit status
-----------------------------------
結果：起動していない
-----------------------------------
Monit: the monit daemon is not running
-----------------------------------
monit を起動させる（centOS6.x系）
-----------------------------------
service monit start
-----------------------------------
monit を起動させる（centOS7.x系）
-----------------------------------
systemctl start monit
-----------------------------------
設定ファイルの編集
-----------------------------------
vi /etc/monit.conf
-----------------------------------
monit のログ（デフォルト）
-----------------------------------
cat /var/log/monit
-----------------------------------
monit.conf を見ると、include /etc/monit.d/ 以下のファイルがインクルードされているのが分かる
-----------------------------------
# Include all files from /etc/monit.d/
include /etc/monit.d/*
-----------------------------------
監視対象のデーモンは /etc/monit.d/ に個別にファイルを作ればいい。
crond の監視をする場合、下記の内容で crond.conf というファイル名にして /etc/monit.d の直下に置く
-----------------------------------
check process crond with pidfile /var/run/crond.pid
start program = "/etc/init.d/crond start"
stop program = "/etc/init.d/crond stop"
if 5 restarts within 5 cycles then unmonitor
-----------------------------------
mysqldはこれ
-----------------------------------
check process mysqld with pidfile /var/run/mysqld/mysqld.pid
start program = "/etc/init.d/mysqld start"
stop program = "/etc/init.d/mysqld stop"
if 5 restarts within 5 cycles then unmonitor
-----------------------------------
httpd（apache）はこれ
-----------------------------------
check process httpd with pidfile /var/run/httpd/httpd.pid
start program = "/etc/init.d/httpd start"
stop program = "/etc/init.d/httpd stop"
if 5 restarts within 5 cycles then unmonitor
-----------------------------------
メールの設定。monit.conf に書く
-----------------------------------
#yahooメールの送信サーバーを指定
set mailserver localhost
#アラートメールの内容
set mail-format {
from: monit@example.com
subject: $HOST : $SERVICE - $EVENT
message: Monit
ACTION : $ACTION
SERVICE : $SERVICE
at $DATE on $HOST.
DESCRIPTION : $DESCRIPTION
}
#送信先
set alert 自分のメール@yahoo.co.jp
-----------------------------------
個別設定ファイルを作ったらシンタックスのチェックも忘れずに
-----------------------------------
monit -t
-----------------------------------
シンタックスに問題なければ設定を反映
-----------------------------------
monit reload
-----------------------------------
サーバーが起動した際に monit も自動起動するか確認
-----------------------------------
# chkconfig --list monit
monit 0:off 1:off 2:off 3:off 4:off 5:off 6:off
-----------------------------------
run level 3 ～ 5 が on になっていれば自動起動する。↑これでは起動しない。
ちなみに mysqld の run level を確認すると 3～ 5 が on になっていた。
----------------------------------
#chkconfig --list mysqld
mysqld 0:off 1:off 2:off 3:on 4:on 5:on 6:off
---------------------------------
monit を自動起動に設定する
---------------------------------
chkconfig monit on
---------------------------------
スクリプトにより unmonitor になった場合に monitor に戻すとき
-----------------------------------
monit monitor all
-----------------------------------
一時的に unmonitor するとき
-----------------------------------
monit unmonitor crond
もしくは
monit unmonitor all
-----------------------------------
一度 unmonitor になると自動で戻らないので cron で定期的に戻す
-----------------------------------
#60分ごとに unmonit をリフレッシュ
*/60 * * * * /usr/bin/monit monitor all
-----------------------------------
マニュアルを開く
-----------------------------------
man monit
-----------------------------------
カテゴリー：CentOS, サーバートラブル・つながらない
PoderosaをSSH2 形式で利用していて、PC に置いてある秘密鍵を iphone のターミナルでも再利用しようとコピペしたが使えない。
id_rsa というファイル名でこんな感じの内容だった
------------------------------------------------
---- BEGIN SSH2 ENCRYPTED PRIVATE KEY ----
Comment:
faejpg@gjewrewf:affewgwgrgfsdfdsfdsfsdffa
ejpg@gjewrewf:affewgwgrgfsdfdsfdsfsdffaej
pg@gjewrewf:affewgwgrgfsdfdsfdsfsdffaejpg
@gjewrewf:affewgwgrgfsdfdsfdsfsdffaejpg@g
jewrewf:affewgwgrgfsdfdsfdsfsdffaejpg@gjew
rewf:affewgwgrgfsdfdsfdsfsdf
---- END SSH2 ENCRYPTED PRIVATE KEY ----
------------------------------------------------
これはssh.com形式(SECSH)という形式の鍵で termius で使用するには OpenSSH形式への変換が必要らしい
認証鍵の形式についてはこちらを参考
「SSHの公開鍵ってなに?」の「認証鍵の形式」の章
ちなみにサーバーにアップロードされている公開キーは ssh-rsa～ から始まる一行の物でOpenSSH 形式だった。
------------------------------------------------
cat /home/ユーザー名/.ssh/authorized_keys
------------------------------------------------
ssh.com形式(SECSH)をOpenSSH形式へ鍵の形式変換
変換を試みる
ssh-keygen コマンドがあるか確認
------------------------------
which ssh-keygen
------------------------------
変換を実施
------------------------------
# ssh-keygen -i -f id_rsa > new_id_rsa
unsupported cipher 3des-cbc
decode blob failed.
------------------------------
失敗した。
別の方法で変換を試みる。
参考ページの方法で変換する↓
PoderosaのSSH2の秘密鍵をOpenSSHで使う
秘密鍵の変換は puttygen.exe を利用するとできるらしい。
１．ダウンロードしたputtygen.exe をダブルクリックで起動。
２． [Conversions]→[Import key] でPoderosaで作成した秘密鍵をインポート
３． [Conversions]→[Export OpenSSH key] で秘密鍵をエクスポート
出来上がった秘密鍵を iphone の termius に itunes のファイル共有から移動させて接続できた。
カテゴリー：CentOS, SSH, サーバートラブル・つながらない, サーバー設定
サーバーへの接続要求が失敗した数の味方
コマンド
-----------------------------------------
# netstat -s | grep Tcp: -A 10
-----------------------------------------
netstat -s は、システム起動後からのネットワークの統計値を表示するためコマンド
-A 10 はgrep のオプションで、十行うしろも表示
結果
-----------------------------------------
Tcp:
15707 active connections openings
1356123 passive connection openings
2425 failed connection attempts
33948 connection resets received
187 connections established
32977847 segments received
34238933 segments send out
298015 segments retransmited
14 bad segments received.
26367 resets sent
-----------------------------------------
failed connection attempts が失敗した接続の数
カテゴリー：CentOS, サーバートラブル・つながらない
error_log という名前のファイルを探したい場合
--------------------------------------
# find / -name error_log
--------------------------------------
ワイルドカードも使える
--------------------------------------
# find / -name error_*
--------------------------------------
カテゴリー：linux
１．ターミナルでサーバーにログインできるか。
poderosaを使用。サーバーにログインできるのであればサーバー自体は問題ないの判断。
２．HTTPサーバー（apache）は起動しているか確認。
---------------------------------
service httpd status
---------------------------------
もしくは
---------------------------------
/etc/init.d/httpd status
---------------------------------
/etc/init.dディレクトリ → デーモンを起動するためのファイル（起動スクリプト = rcスクリプト）が置いてある
rc は run command
結果（起動している場合）
---------------------------------
httpd (pid 1759) を実行中...
---------------------------------
２．HTTPサーバー（apache）が起動しているか確認。
---------------------------------
# service httpd status
---------------------------------
もしくは
---------------------------------
# /etc/init.d/httpd status
---------------------------------
/etc/init.dディレクトリ → デーモンを起動するためのファイル（起動スクリプト = rcスクリプト）が置いてある
rc は run command
結果（起動している場合）
---------------------------------
httpd (pid 1759) を実行中...
---------------------------------
３．mysql が起動しているか確認。
---------------------------------
# service mysqld status
---------------------------------
もしくは
---------------------------------
# /etc/init.d/mysqld status
---------------------------------
結果（起動している場合）
---------------------------------
mysqld (pid 1634) を実行中...
---------------------------------
apache は起動しているが mysql が落ちている時に、ページにアクセスすると php のエラーが出る
-----------------------------------------------------------
Warning: mysql_set_charset(): No such file or directory in /var/www/libs/～～～～
Warning: mysql_set_charset(): A link to the server could not be established in /var/www/～～～～～
ConnectError!DB Error: connect failed
-----------------------------------------------------------
この場合は、mysqld を再起動などする
-----------------------------------------------------------
sudo /etc/init.d/mysqld restart
-----------------------------------------------------------
４．webサービスに繋がらない原因はアクセス過多の場合が多いので現在のhttpdのプロセス数を確認する
-----------------------------------------------------------
ps aux |grep httpd |wc -l
65
-----------------------------------------------------------
この数字が http.cnf の ServerLimit を超えるとサーバーが停止する。
-----------------------------------------------------------
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
-----------------------------------------------------------
エラーログにもエラーが出る。
エラーログを調べる
-----------------------------------------------------------
cat /var/log/httpd/error_log
-----------------------------------------------------------
エラーログ一部抜粋
------------------------------------------------------------------------------
[Fri Aug 27 01:28:32 2021] [error] server reached MaxClients setting, consider raising the MaxClients setting
------------------------------------------------------------------------------
OS のエラーを調べると、上記エラーが出た時間の前後に oom-killer が発動しているのが分かる
-----------------------------------------------------------
cat messages | grep oom
-----------------------------------------------------------
一部抜粋
-----------------------------------------------------------
Oct 6 02:24:02 hostname kernel: httpd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Oct 6 02:24:02 hostname kernel: [] ? oom_kill_process+0x82/0x2a0
Oct 6 02:24:02 hostname kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
-----------------------------------------------------------
oom-killer が発動したときにサーバーに再起動が起こるように設定する
OOM発動時にサーバーを再起動
５．ネットワークの状態も確認する
-----------------------------------------------------------
# netstat -tpn
-----------------------------------------------------------
netstat で同じipアドレスからのたくさんの重複アクセスを確認したので、対策として mod_evasiveをインストールした
その時の記事
ただしあまり効果が感じられない。インストールしてもサーバーは止まるし、ipを遮断したらメールが来る設定にしているが遮断条件に満たなくてメールがあまり来ない。
ipアドレス別でサーバーに接続している数を調べる
-----------------------------------------------------------
# netstat -tan | awk '{print $5}' | cut -d : -f1 | sort -n | uniq -c | sort -n
-----------------------------------------------------------
-----------------------------------------------------------
37 66.249.68.××
42 157.55.39.××
42 40.77.167.××
43 157.55.39.××
45 207.46.13.××
51 66.249.68.××
2256 207.××.××.××
-----------------------------------------------------------
207.××.××.×× が異常な接続数なので ipアドレスを解析してクローラーでなければ ipアドレスを拒否する
netstat で大量のCLOSE_WAITを確認。その時に行った対策のメモ
サーバーがつながらない。CLOSE_WAITが大量発生
カテゴリー：CentOS, サーバートラブル・つながらない
2021-09-12 15:50:05 1634 [Note] Event Scheduler: Loaded 0 events
2021-09-12 15:50:05 1634 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 17:10:33 1634 [Warning] IP address '34.79.107.251' has been resolved to the host name '251.107.79.34.bc.googleusercontent.com', which resembles IPv4-address itself.
2021-09-12 18:57:11 1634 [Warning] IP address '161.35.186.198' could not be resolved: Name or service not known
2021-09-12 19:55:51 1634 [Warning] IP address '34.79.107.251' has been resolved to the host name '251.107.79.34.bc.googleusercontent.com', which resembles IPv4-address itself.
2021-09-12 20:23:05 1634 [Warning] Host name 'zg-0830b-148.stretchoid.com' could not be resolved: Name or service not known
2021-09-12 20:54:47 9078 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 20:54:47 9078 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 20:54:47 9078 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 20:54:47 9078 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 20:54:47 9078 [Note] InnoDB: Memory barrier is not used
2021-09-12 20:54:47 9078 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 20:54:47 9078 [Note] InnoDB: Using Linux native AIO
2021-09-12 20:54:47 9078 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 20:54:47 9078 [Note] InnoDB: Initializing buffer pool, size = 512.0M
2021-09-12 20:54:47 9078 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 20:54:47 9078 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 20:54:47 9078 [Note] InnoDB: The log sequence numbers 209070279820 and 209070279820 in ibdata files do not match the log sequence number 209071204930 in the ib_logfiles!
2021-09-12 20:54:47 9078 [Note] InnoDB: Database was not shutdown normally!
2021-09-12 20:54:47 9078 [Note] InnoDB: Starting crash recovery.
カテゴリー：サーバートラブル・つながらない
ある日突然、運営中のwebサービスに繋がらなくなった。
正確にいうと、webサービスには繋がっているが、mysql にアクセスできない。
参考になりそうなページ：https://totutotu.hatenablog.com/entry/2015/12/07/173936
サイトにアクセスしたときに表示された PHP のエラーメッセージ
-----------------------------------------------------------
Warning: mysql_set_charset(): No such file or directory in /var/www/libs/～～～～
Warning: mysql_set_charset(): A link to the server could not be established in /var/www/～～～～～
ConnectError!DB Error: connect failed
-----------------------------------------------------------
前回と同じエラー。
サーバーを再起動したら mysql に繋がらなくなった。
mysqld の状態を確認
-----------------------------------------------------------
service mysqld status
mysqld は停止していますがサブシステムがロックされています
-----------------------------------------------------------
（↑これはrootでない状態で確認した）
サーバーのerror_log
-----------------------------------------------------------
[Sun Sep 12 11:30:15 2021] [error] server reached MaxClients setting, consider raising the MaxClients setting
-----------------------------------------------------------
システムのログを調べる
-----------------------------------------------------------
cat /var/log/messages
-----------------------------------------------------------
サーバー再起動後だったため再起動前の情報が見えなかったので不明
ターミナルから mysqld のログを調べる
-----------------------------------------------------------
cat /var/log/mysqld.log
-----------------------------------------------------------
怪しいそうな部分だけ抜粋
-------------------------------------------------------------------------
2021-09-12 10:14:41 1623 [Note] - '::' resolves to '::';
2021-09-12 10:14:41 1623 [Note] Server socket created on IP: '::'.
2021-09-12 10:14:41 1623 [Note] Event Scheduler: Loaded 0 events
2021-09-12 10:14:41 1623 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:32:49 3983 [Note] Plugin 'FEDERATED' is disabled. ← おそらく起動開始がここ。disabled = 無効
2021-09-12 11:32:49 3983 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:32:49 3983 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:32:49 3983 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:32:49 3983 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:32:49 3983 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:32:49 3983 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:32:49 3983 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:32:49 3983 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:32:49 3983 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:32:50 3983 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:32:50 3983 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069557708 in the ib_logfiles!
2021-09-12 11:32:50 3983 [Note] InnoDB: Database was not shutdown normally!
2021-09-12 11:32:50 3983 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:32:50 3983 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:32:50 3983 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:32:50 3983 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:32:50 3983 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:32:50 3983 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:32:50 3983 [Note] InnoDB: 5.6.36 started; log sequence number 209069557708
2021-09-12 11:32:50 3983 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:32:50 3983 [Note] IPv6 is available.
2021-09-12 11:32:50 3983 [Note] - '::' resolves to '::';
2021-09-12 11:32:50 3983 [Note] Server socket created on IP: '::'.
2021-09-12 11:32:50 3983 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:32:50 3983 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:33:06 4025 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:06 4025 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:06 4025 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:06 4025 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:06 4025 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:06 4025 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:06 4025 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:06 4025 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:06 4025 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:07 4025 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:07 4025 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:07 4025 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069557927 in the ib_logfiles!
2021-09-12 11:33:07 4025 [Note] InnoDB: Database was not shutdown normally!
2021-09-12 11:33:07 4025 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:07 4025 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:07 4025 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:07 4025 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:33:07 4025 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:07 4025 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:07 4025 [Note] InnoDB: 5.6.36 started; log sequence number 209069557927
2021-09-12 11:33:07 4025 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:07 4025 [Note] IPv6 is available.
2021-09-12 11:33:07 4025 [Note] - '::' resolves to '::';
2021-09-12 11:33:07 4025 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:07 4025 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:07 4025 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:33:12 4068 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:12 4068 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:12 4068 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:12 4068 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:12 4068 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:12 4068 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:12 4068 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:12 4068 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:12 4068 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:12 4068 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:12 4068 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:12 4068 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069557937 in the ib_logfiles!
2021-09-12 11:33:12 4068 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:33:12 4068 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:12 4068 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:13 4068 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:13 4068 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:33:13 4068 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:13 4068 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:13 4068 [Note] InnoDB: 5.6.36 started; log sequence number 209069557937
2021-09-12 11:33:13 4068 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:13 4068 [Note] IPv6 is available.
2021-09-12 11:33:13 4068 [Note] - '::' resolves to '::';
2021-09-12 11:33:13 4068 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:13 4068 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:13 4068 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:33:17 4110 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:17 4110 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:17 4110 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:17 4110 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:17 4110 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:17 4110 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:17 4110 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:17 4110 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:17 4110 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:17 4110 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:17 4110 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:17 4110 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069557937
2021-09-12 11:33:17 4110 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:33:17 4110 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:17 4110 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:18 4110 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:18 4110 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069557947
2021-09-12 11:33:18 4110 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:18 4110 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:18 4110 [Note] InnoDB: 5.6.36 started; log sequence number 209069557947
2021-09-12 11:33:19 4110 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:19 4110 [Note] IPv6 is available.
2021-09-12 11:33:19 4110 [Note] - '::' resolves to '::';
2021-09-12 11:33:19 4110 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:19 4110 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:19 4110 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:33:22 4152 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:22 4152 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:22 4152 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:22 4152 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:22 4152 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:22 4152 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:22 4152 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:23 4152 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:23 4152 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:23 4152 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:23 4152 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:23 4152 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069557957 in the ib_logfiles!
2021-09-12 11:33:23 4152 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:33:23 4152 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:23 4152 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:23 4152 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:23 4152 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:33:24 4152 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:24 4152 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:24 4152 [Note] InnoDB: 5.6.36 started; log sequence number 209069557957
2021-09-12 11:33:24 4152 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:24 4152 [Note] IPv6 is available.
2021-09-12 11:33:24 4152 [Note] - '::' resolves to '::';
2021-09-12 11:33:24 4152 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:24 4152 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:24 4152 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:33:28 4201 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:28 4201 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:28 4201 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:28 4201 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:28 4201 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:28 4201 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:28 4201 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:28 4201 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:28 4201 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:28 4201 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:28 4201 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:28 4201 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069557957
2021-09-12 11:33:28 4201 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:33:28 4201 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:28 4201 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:28 4201 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:28 4201 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069558157
2021-09-12 11:33:29 4201 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2021-09-12 11:33:29 4201 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:29 4201 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:29 4201 [Note] InnoDB: 5.6.36 started; log sequence number 209069558157
2021-09-12 11:33:29 4201 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:29 4201 [Note] IPv6 is available.
2021-09-12 11:33:29 4201 [Note] - '::' resolves to '::';
2021-09-12 11:33:29 4201 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:29 4201 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:29 4201 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:33:37 4265 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:37 4265 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:37 4265 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:37 4265 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:37 4265 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:37 4265 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:37 4265 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:37 4265 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:37 4265 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:37 4265 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:37 4265 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:37 4265 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069558563 in the ib_logfiles!
2021-09-12 11:33:37 4265 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:33:37 4265 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:37 4265 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:37 4265 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:37 4265 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:33:38 4265 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:38 4265 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:38 4265 [Note] InnoDB: 5.6.36 started; log sequence number 209069558563
2021-09-12 11:33:38 4265 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:38 4265 [Note] IPv6 is available.
2021-09-12 11:33:38 4265 [Note] - '::' resolves to '::';
2021-09-12 11:33:38 4265 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:38 4265 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:38 4265 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:36:14 4355 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:14 4355 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:14 4355 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:14 4355 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:14 4355 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:14 4355 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:14 4355 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:14 4355 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:14 4355 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:14 4355 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:14 4355 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:14 4355 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069568015 in the ib_logfiles!
2021-09-12 11:36:14 4355 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:14 4355 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:14 4355 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:14 4355 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:14 4355 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:36:14 4355 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:14 4355 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:14 4355 [Note] InnoDB: 5.6.36 started; log sequence number 209069568015
2021-09-12 11:36:14 4355 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:14 4355 [Note] IPv6 is available.
2021-09-12 11:36:14 4355 [Note] - '::' resolves to '::';
2021-09-12 11:36:14 4355 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:14 4355 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:14 4355 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:36:21 4404 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:21 4404 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:21 4404 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:21 4404 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:21 4404 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:21 4404 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:21 4404 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:21 4404 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:21 4404 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:21 4404 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:21 4404 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:21 4404 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069568235 in the ib_logfiles!
2021-09-12 11:36:21 4404 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:21 4404 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:21 4404 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:21 4404 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:21 4404 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:36:21 4404 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:21 4404 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:21 4404 [Note] InnoDB: 5.6.36 started; log sequence number 209069568235
2021-09-12 11:36:21 4404 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:21 4404 [Note] IPv6 is available.
2021-09-12 11:36:21 4404 [Note] - '::' resolves to '::';
2021-09-12 11:36:21 4404 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:21 4404 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:21 4404 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:36:25 4449 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:25 4449 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:25 4449 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:25 4449 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:25 4449 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:25 4449 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:25 4449 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:25 4449 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:25 4449 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:25 4449 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:25 4449 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:25 4449 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568235
2021-09-12 11:36:25 4449 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:25 4449 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:25 4449 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:25 4449 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:25 4449 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568440
2021-09-12 11:36:25 4449 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2021-09-12 11:36:26 4449 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:26 4449 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:26 4449 [Note] InnoDB: 5.6.36 started; log sequence number 209069568440
2021-09-12 11:36:26 4449 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:26 4449 [Note] IPv6 is available.
2021-09-12 11:36:26 4449 [Note] - '::' resolves to '::';
2021-09-12 11:36:26 4449 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:26 4449 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:26 4449 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:36:28 4503 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:28 4503 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:28 4503 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:28 4503 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:28 4503 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:28 4503 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:28 4503 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:28 4503 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:28 4503 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:28 4503 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:28 4503 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:28 4503 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568235
2021-09-12 11:36:28 4503 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:28 4503 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:28 4503 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:28 4503 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:28 4503 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568450
2021-09-12 11:36:28 4503 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2021-09-12 11:36:29 4503 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:29 4503 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:29 4503 [Note] InnoDB: 5.6.36 started; log sequence number 209069568450
2021-09-12 11:36:29 4503 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:29 4503 [Note] IPv6 is available.
2021-09-12 11:36:29 4503 [Note] - '::' resolves to '::';
2021-09-12 11:36:29 4503 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:29 4503 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:29 4503 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:36:32 4551 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:32 4551 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:32 4551 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:32 4551 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:32 4551 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:32 4551 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:32 4551 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:32 4551 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:32 4551 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:32 4551 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:32 4551 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:32 4551 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568450
2021-09-12 11:36:32 4551 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:32 4551 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:32 4551 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:32 4551 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:32 4551 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568460
2021-09-12 11:36:32 4551 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:32 4551 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:32 4551 [Note] InnoDB: 5.6.36 started; log sequence number 209069568460
2021-09-12 11:36:32 4551 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:32 4551 [Note] IPv6 is available.
2021-09-12 11:36:32 4551 [Note] - '::' resolves to '::';
2021-09-12 11:36:32 4551 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:32 4551 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:32 4551 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:36:35 4601 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:35 4601 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:35 4601 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:35 4601 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:35 4601 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:35 4601 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:35 4601 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:35 4601 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:35 4601 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:35 4601 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:35 4601 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:35 4601 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568460
2021-09-12 11:36:35 4601 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:35 4601 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:35 4601 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:35 4601 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:35 4601 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568470
2021-09-12 11:36:36 4601 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:36 4601 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:36 4601 [Note] InnoDB: 5.6.36 started; log sequence number 209069568470
2021-09-12 11:36:36 4601 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:36 4601 [Note] IPv6 is available.
2021-09-12 11:36:36 4601 [Note] - '::' resolves to '::';
2021-09-12 11:36:36 4601 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:36 4601 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:36 4601 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:36:39 4641 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:39 4641 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:39 4641 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:39 4641 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:39 4641 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:39 4641 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:39 4641 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:39 4641 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:39 4641 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:39 4641 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:39 4641 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:39 4641 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568470
2021-09-12 11:36:39 4641 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:39 4641 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:39 4641 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:39 4641 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:39 4641 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568480
2021-09-12 11:36:39 4641 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:39 4641 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:39 4641 [Note] InnoDB: 5.6.36 started; log sequence number 209069568480
2021-09-12 11:36:39 4641 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:39 4641 [Note] IPv6 is available.
2021-09-12 11:36:39 4641 [Note] - '::' resolves to '::';
2021-09-12 11:36:39 4641 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:39 4641 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:39 4641 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 11:36:42 4689 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:42 4689 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:42 4689 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:42 4689 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:42 4689 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:42 4689 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:42 4689 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:42 4689 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:42 4689 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:42 4689 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:42 4689 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:42 4689 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568470
2021-09-12 11:36:42 4689 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:42 4689 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:42 4689 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:42 4689 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:42 4689 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568490 ← この後に mysqld が停止してログ無しみたい
2021-09-12 11:47:07 1622 [Note] Plugin 'FEDERATED' is disabled. ←サーバー手動再起動後
2021-09-12 11:47:07 1622 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:47:07 1622 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:47:07 1622 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
-------------------------------------------------------------------------
「InnoDB: Database was not shutdown normally!」というログが短時間のうちに複数あり、再起動を繰り返しているのが分かる。
これ以前を解析していけばシャットダウンの原因が分かるはず。
もう少しログをさかのぼったらこんな記述も見つけた。これも mysql が停止したタイミングもログと思われる
-------------------------------------------------------------------------
2021-09-12 09:01:57 1810 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 09:01:57 1810 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 09:01:57 1810 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 09:01:57 1810 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 09:01:57 1810 [Note] InnoDB: Memory barrier is not used
2021-09-12 09:01:57 1810 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 09:01:57 1810 [Note] InnoDB: Using Linux native AIO
2021-09-12 09:01:57 1810 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 09:01:57 1810 [Note] InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
2021-09-12 09:01:57 1810 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2021-09-12 09:01:57 1810 [ERROR] Plugin 'InnoDB' init function returned error.
2021-09-12 09:01:57 1810 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2021-09-12 09:01:57 1810 [ERROR] Unknown/unsupported storage engine: InnoDB
2021-09-12 09:01:57 1810 [ERROR] Aborting
2021-09-12 09:01:57 1810 [Note] Binlog end
2021-09-12 09:01:57 1810 [Note] Shutting down plugin 'partition'
2021-09-12 09:01:57 1810 [Note] Shutting down plugin 'ARCHIVE'
2021-09-12 09:01:57 1810 [Note] Shutting down plugin 'BLACKHOLE'
2021-09-12 09:01:57 1810 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
-------------------------------------------------------------------------
短い時間で解決策を調べたけど不明。
とりあえずバッファプールへのメモリ割り当てを上げておく。
参考ページ：https://tech.librastudio.co.jp/entry/index.php/2018/05/22/post-1891/
現状を確認。128Mであることが分かる
--------------------------------------------------------
mysql> SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
+-------------------------+-----------+
| Variable_name | Value |
+-------------------------+-----------+
| innodb_buffer_pool_size | 134217728 |
+-------------------------+-----------+
--------------------------------------------------------
my.cnf を編集（my.conf でない）
--------------------------------------------------------
vi my.cnf
--------------------------------------------------------
--------------------------------------------------------
[mysqld]
...
innodb_buffer_pool_size=512M
--------------------------------------------------------
mysql を再起動
--------------------------------------------------------
/etc/init.d/mysqld restart
--------------------------------------------------------
変更できていることを確認
--------------------------------------------------------
mysql> SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
+-------------------------+-----------+
| Variable_name | Value |
+-------------------------+-----------+
| innodb_buffer_pool_size | 536870912 |
+-------------------------+-----------+
--------------------------------------------------------
ここも後で見る
MySQLパフォーマンスチューニング -my.cnfの見直し-
カテゴリー：mysql, サーバートラブル・つながらない
参考ページ１：https://end0tknr.hateblo.jp/entry/20110724/1311490171
参考ページ２：https://hacknote.jp/archives/19502/
参考ページ３：https://seesaawiki.jp/w/dehio3/d/solaris%A1%C1CLOSE_WAIT%C8%AF%C0%B8%A1%C1
参考ページ４：https://hirofukami.com/2008/08/11/close-wait/
サーバーがつながらない事態が頻発
ネットワークの状況を確認
---------------------------------------
netstat -tpn
---------------------------------------
結果
---------------------------------------
tcp 0 0 160.16.63.○○○:443 207.46.13.76:15571 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:40457 CLOSE_WAIT 11805/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:11251 CLOSE_WAIT 11567/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:11982 CLOSE_WAIT 11753/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:54793 CLOSE_WAIT 12106/httpd
tcp 0 0 160.16.63.○○○:443 126.34.252.62:59582 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:53009 CLOSE_WAIT 12076/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:57869 CLOSE_WAIT 11894/httpd
tcp 521 0 160.16.63.○○○:443 36.13.167.104:37684 ESTABLISHED -
tcp 0 0 160.16.63.○○○:443 13.66.139.0:19846 TIME_WAIT -
tcp 0 0 160.16.63.○○○:443 112.137.96.56:59062 TIME_WAIT -
tcp 0 0 160.16.63.○○○:443 125.8.175.235:55073 TIME_WAIT -
tcp 0 0 160.16.63.○○○:443 110.232.6.71:62611 TIME_WAIT -
tcp 518 0 160.16.63.○○○:443 210.194.87.121:50331 CLOSE_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:9959 CLOSE_WAIT 12046/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:14902 CLOSE_WAIT 11726/httpd
tcp 0 0 160.16.63.○○○:443 60.153.198.22:42116 ESTABLISHED -
tcp 0 0 160.16.63.○○○:443 133.201.90.32:62121 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:13430 CLOSE_WAIT 11769/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:35291 CLOSE_WAIT 12005/httpd
tcp 0 0 160.16.63.○○○:443 13.66.139.0:19845 TIME_WAIT -
tcp 517 0 160.16.63.○○○:443 111.239.35.98:65278 ESTABLISHED -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:42621 CLOSE_WAIT 12102/httpd
tcp 0 0 160.16.63.○○○:443 211.127.83.36:57156 ESTABLISHED 12020/httpd
tcp 0 0 160.16.63.○○○:443 126.34.252.62:59581 TIME_WAIT -
tcp 0 0 160.16.63.○○○:443 111.64.18.75:62476 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:6615 CLOSE_WAIT 11750/httpd
tcp 0 0 160.16.63.○○○:443 72.14.199.110:37727 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:4216 CLOSE_WAIT 12023/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:15151 CLOSE_WAIT 12123/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:9227 CLOSE_WAIT 11850/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:13495 CLOSE_WAIT 11863/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:32764 CLOSE_WAIT 11987/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:17980 CLOSE_WAIT 11958/httpd
tcp 0 0 160.16.63.○○○:443 133.206.82.0:42008 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:38943 CLOSE_WAIT 12051/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:33367 CLOSE_WAIT 12138/httpd
tcp 517 0 160.16.63.○○○:443 112.137.96.56:59073 ESTABLISHED -
tcp 0 0 160.16.63.○○○:443 112.137.96.56:59064 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:43395 CLOSE_WAIT 12149/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:28548 CLOSE_WAIT 11806/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:39903 CLOSE_WAIT 12145/httpd
tcp 0 0 160.16.63.○○○:443 126.53.57.5:32806 ESTABLISHED 11930/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:49601 CLOSE_WAIT 11609/httpd
tcp 0 0 160.16.63.○○○:443 207.46.13.76:15559 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:15573 CLOSE_WAIT 12009/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:62567 CLOSE_WAIT 12033/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:7693 CLOSE_WAIT 11818/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:21720 CLOSE_WAIT 11801/httpd
tcp 0 0 160.16.63.○○○:443 207.46.13.76:15574 ESTABLISHED -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:49345 CLOSE_WAIT 11882/httpd
tcp 0 0 160.16.63.○○○:443 207.46.13.76:15569 TIME_WAIT -
tcp 0 0 160.16.63.○○○:443 203.133.150.50:55961 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:17553 CLOSE_WAIT 11804/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:23842 CLOSE_WAIT 11762/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:7292 CLOSE_WAIT 11814/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:23870 CLOSE_WAIT 11528/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:32023 CLOSE_WAIT 12062/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:62955 CLOSE_WAIT 11989/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:8371 CLOSE_WAIT 11919/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:22436 CLOSE_WAIT 11682/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:21309 CLOSE_WAIT 11913/httpd
---------------------------------------
同じIPアドレスからのアクセスで CLOSE_WAIT が大量発生しているのが分かる
/var/log/httpd/error_log には child process 13944 still did not exit, sending a SIGTERM が残っている。（関係あるかは不明）
---------------------------------------
[Wed Sep 08 21:23:17 2021] [warn] child process 13944 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 13951 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 13955 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 13970 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 13984 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 14001 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 14012 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 14031 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 14044 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:18 2021] [warn] child process 14070 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:18 2021] [warn] child process 14075 still did not exit, sending a SIGTERM
---------------------------------------
CLOSE_WAIT は通信相手から自分への通信はcloseしたが、自分側は完全にcloseしていない状態。
デフォルトの有効時間は7200秒（2時間）。
CLOSE_WAIT の有効時間は net.ipv4.tcp_keepalive_time ディレクティブで変更可能。
今の値を確認
---------------------------------------
# sysctl -n net.ipv4.tcp_keepalive_time
7200
---------------------------------------
利用できる全ての値を表示して net.ipv4.tcp_keepalive_time があることを確認
---------------------------------------
sysctl -a
---------------------------------------
7200の値を少なくすれば改善されるはず
一時的な変更方法
3600秒に変更
---------------------------------------
echo 3600 > /proc/sys/net/ipv4/tcp_keepalive_time
---------------------------------------
永続的な変更方法
---------------------------------------
# vi /etc/sysctl.conf
net.ipv4.tcp_keepalive_time = 10
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_intvl = 3
---------------------------------------
を追記
---------------------------------------
# sysctl -w
---------------------------------------
変更を反映
---------------------------------------
# sysctl -p
---------------------------------------
カテゴリー：サーバートラブル・つながらない, サーバー設定
