サーバートラブル・つながらない


poderosaのssh2接続で使用していた秘密鍵をtermiusで使う

PoderosaをSSH2 形式で利用していて、PC に置いてある秘密鍵を iphone のターミナルでも再利用しようとコピペしたが使えない。

id_rsa というファイル名でこんな感じの内容だった
------------------------------------------------

---- BEGIN SSH2 ENCRYPTED PRIVATE KEY ----
Comment:
faejpg@gjewrewf:affewgwgrgfsdfdsfdsfsdffa
ejpg@gjewrewf:affewgwgrgfsdfdsfdsfsdffaej
pg@gjewrewf:affewgwgrgfsdfdsfdsfsdffaejpg
@gjewrewf:affewgwgrgfsdfdsfdsfsdffaejpg@g
jewrewf:affewgwgrgfsdfdsfdsfsdffaejpg@gjew
rewf:affewgwgrgfsdfdsfdsfsdf
---- END SSH2 ENCRYPTED PRIVATE KEY ----
------------------------------------------------

これはssh.com形式(SECSH)という形式の鍵で termius で使用するには OpenSSH形式への変換が必要らしい
認証鍵の形式についてはこちらを参考
「SSHの公開鍵ってなに?」の「認証鍵の形式」の章

ちなみにサーバーにアップロードされている公開キーは ssh-rsa~ から始まる一行の物でOpenSSH 形式だった。
------------------------------------------------
cat /home/ユーザー名/.ssh/authorized_keys
------------------------------------------------

ssh.com形式(SECSH)をOpenSSH形式へ鍵の形式変換

変換を試みる

ssh-keygen コマンドがあるか確認
------------------------------
which ssh-keygen
------------------------------

変換を実施
------------------------------
# ssh-keygen -i -f id_rsa > new_id_rsa
unsupported cipher 3des-cbc
decode blob failed.
------------------------------

失敗した。

別の方法で変換を試みる。

参考ページの方法で変換する↓
PoderosaのSSH2の秘密鍵をOpenSSHで使う

秘密鍵の変換は puttygen.exe を利用するとできるらしい。

1.ダウンロードしたputtygen.exe をダブルクリックで起動。
2. [Conversions]→[Import key] でPoderosaで作成した秘密鍵をインポート
3. [Conversions]→[Export OpenSSH key] で秘密鍵をエクスポート

出来上がった秘密鍵を iphone の termius に itunes のファイル共有から移動させて接続できた。

サーバーへの接続要求が失敗した数の味方

サーバーへの接続要求が失敗した数の味方

コマンド
-----------------------------------------
# netstat -s | grep Tcp: -A 10
-----------------------------------------

netstat -s は、システム起動後からのネットワークの統計値を表示するためコマンド

-A 10 はgrep のオプションで、十行うしろも表示

結果
-----------------------------------------
Tcp:
15707 active connections openings
1356123 passive connection openings
2425 failed connection attempts
33948 connection resets received
187 connections established
32977847 segments received
34238933 segments send out
298015 segments retransmited
14 bad segments received.
26367 resets sent
-----------------------------------------

failed connection attempts が失敗した接続の数

サーバーがつながらない。webページが表示されないときに確認すること。

1.ターミナルでサーバーにログインできるか。

poderosaを使用。サーバーにログインできるのであればサーバー自体は問題ないの判断。

2.HTTPサーバー(apache)は起動しているか確認。

---------------------------------
service httpd status
---------------------------------

もしくは

---------------------------------
/etc/init.d/httpd status
---------------------------------
/etc/init.dディレクトリ → デーモンを起動するためのファイル(起動スクリプト = rcスクリプト)が置いてある
rc は run command

結果(起動している場合)
---------------------------------
httpd (pid 1759) を実行中...
---------------------------------

2.HTTPサーバー(apache)が起動しているか確認。

---------------------------------
# service httpd status
---------------------------------

もしくは

---------------------------------
# /etc/init.d/httpd status
---------------------------------
/etc/init.dディレクトリ → デーモンを起動するためのファイル(起動スクリプト = rcスクリプト)が置いてある
rc は run command

結果(起動している場合)
---------------------------------
httpd (pid 1759) を実行中...
---------------------------------

3.mysql が起動しているか確認。
---------------------------------
# service mysqld status
---------------------------------

もしくは

---------------------------------
# /etc/init.d/mysqld status
---------------------------------

結果(起動している場合)
---------------------------------
mysqld (pid 1634) を実行中...
---------------------------------

apache は起動しているが mysql が落ちている時に、ページにアクセスすると php のエラーが出る

-----------------------------------------------------------
Warning: mysql_set_charset(): No such file or directory in /var/www/libs/~~~~

Warning: mysql_set_charset(): A link to the server could not be established in /var/www/~~~~~
ConnectError!DB Error: connect failed
-----------------------------------------------------------

この場合は、mysqld を再起動などする
-----------------------------------------------------------
sudo /etc/init.d/mysqld restart
-----------------------------------------------------------

4.webサービスに繋がらない原因はアクセス過多の場合が多いので現在のhttpdのプロセス数を確認する

-----------------------------------------------------------
ps aux |grep httpd |wc -l
65
-----------------------------------------------------------

この数字が http.cnf の ServerLimit を超えるとサーバーが停止する。

-----------------------------------------------------------

StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000

-----------------------------------------------------------

エラーログにもエラーが出る。

エラーログを調べる
-----------------------------------------------------------
cat /var/log/httpd/error_log
-----------------------------------------------------------

エラーログ一部抜粋
------------------------------------------------------------------------------
[Fri Aug 27 01:28:32 2021] [error] server reached MaxClients setting, consider raising the MaxClients setting
------------------------------------------------------------------------------

5.ネットワークの状態も確認する
-----------------------------------------------------------
# netstat -tpn
-----------------------------------------------------------

netstat で同じipアドレスからのたくさんの重複アクセスを確認したので、対策として mod_evasiveをインストールした
その時の記事
ただしあまり効果が感じられない。インストールしてもサーバーは止まるし、ipを遮断したらメールが来る設定にしているが遮断条件に満たなくてメールがあまり来ない。

ipアドレス別でサーバーに接続している数を調べる
-----------------------------------------------------------
# netstat -tan | awk '{print $5}' | cut -d : -f1 | sort -n | uniq -c | sort -n
-----------------------------------------------------------

-----------------------------------------------------------
37 66.249.68.××
42 157.55.39.××
42 40.77.167.××
43 157.55.39.××
45 207.46.13.××
51 66.249.68.××
2256 207.××.××.××
-----------------------------------------------------------
207.××.××.×× が異常な接続数なので ipアドレスを解析してクローラーでなければ ipアドレスを拒否する

netstat で大量のCLOSE_WAITを確認。その時に行った対策のメモ

サーバーがつながらない。CLOSE_WAITが大量発生

後で見る

2021-09-12 15:50:05 1634 [Note] Event Scheduler: Loaded 0 events
2021-09-12 15:50:05 1634 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2021-09-12 17:10:33 1634 [Warning] IP address '34.79.107.251' has been resolved to the host name '251.107.79.34.bc.googleusercontent.com', which resembles IPv4-address itself.
2021-09-12 18:57:11 1634 [Warning] IP address '161.35.186.198' could not be resolved: Name or service not known
2021-09-12 19:55:51 1634 [Warning] IP address '34.79.107.251' has been resolved to the host name '251.107.79.34.bc.googleusercontent.com', which resembles IPv4-address itself.
2021-09-12 20:23:05 1634 [Warning] Host name 'zg-0830b-148.stretchoid.com' could not be resolved: Name or service not known
2021-09-12 20:54:47 9078 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 20:54:47 9078 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 20:54:47 9078 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 20:54:47 9078 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 20:54:47 9078 [Note] InnoDB: Memory barrier is not used
2021-09-12 20:54:47 9078 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 20:54:47 9078 [Note] InnoDB: Using Linux native AIO
2021-09-12 20:54:47 9078 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 20:54:47 9078 [Note] InnoDB: Initializing buffer pool, size = 512.0M
2021-09-12 20:54:47 9078 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 20:54:47 9078 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 20:54:47 9078 [Note] InnoDB: The log sequence numbers 209070279820 and 209070279820 in ibdata files do not match the log sequence number 209071204930 in the ib_logfiles!
2021-09-12 20:54:47 9078 [Note] InnoDB: Database was not shutdown normally!
2021-09-12 20:54:47 9078 [Note] InnoDB: Starting crash recovery.

サーバーがつながらない。mysqlにつながらない

ある日突然、運営中のwebサービスに繋がらなくなった。
正確にいうと、webサービスには繋がっているが、mysql にアクセスできない。

参考になりそうなページ:https://totutotu.hatenablog.com/entry/2015/12/07/173936
サイトにアクセスしたときに表示された PHP のエラーメッセージ
-----------------------------------------------------------
Warning: mysql_set_charset(): No such file or directory in /var/www/libs/~~~~

Warning: mysql_set_charset(): A link to the server could not be established in /var/www/~~~~~
ConnectError!DB Error: connect failed
-----------------------------------------------------------
前回と同じエラー。
サーバーを再起動したら mysql に繋がらなくなった。

mysqld の状態を確認
-----------------------------------------------------------
service mysqld status
mysqld は停止していますがサブシステムがロックされています
-----------------------------------------------------------
(↑これはrootでない状態で確認した)

サーバーのerror_log
-----------------------------------------------------------
[Sun Sep 12 11:30:15 2021] [error] server reached MaxClients setting, consider raising the MaxClients setting
-----------------------------------------------------------

システムのログを調べる
-----------------------------------------------------------
cat /var/log/messages
-----------------------------------------------------------
サーバー再起動後だったため再起動前の情報が見えなかったので不明

ターミナルから mysqld のログを調べる
-----------------------------------------------------------
cat /var/log/mysqld.log
-----------------------------------------------------------

 
怪しいそうな部分だけ抜粋
-------------------------------------------------------------------------
2021-09-12 10:14:41 1623 [Note] - '::' resolves to '::';
2021-09-12 10:14:41 1623 [Note] Server socket created on IP: '::'.
2021-09-12 10:14:41 1623 [Note] Event Scheduler: Loaded 0 events
2021-09-12 10:14:41 1623 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:32:49 3983 [Note] Plugin 'FEDERATED' is disabled. ← おそらく起動開始がここ。disabled = 無効
2021-09-12 11:32:49 3983 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:32:49 3983 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:32:49 3983 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:32:49 3983 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:32:49 3983 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:32:49 3983 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:32:49 3983 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:32:49 3983 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:32:49 3983 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:32:50 3983 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:32:50 3983 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069557708 in the ib_logfiles!
2021-09-12 11:32:50 3983 [Note] InnoDB: Database was not shutdown normally!
2021-09-12 11:32:50 3983 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:32:50 3983 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:32:50 3983 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:32:50 3983 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:32:50 3983 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:32:50 3983 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:32:50 3983 [Note] InnoDB: 5.6.36 started; log sequence number 209069557708
2021-09-12 11:32:50 3983 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:32:50 3983 [Note] IPv6 is available.
2021-09-12 11:32:50 3983 [Note] - '::' resolves to '::';
2021-09-12 11:32:50 3983 [Note] Server socket created on IP: '::'.
2021-09-12 11:32:50 3983 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:32:50 3983 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:33:06 4025 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:06 4025 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:06 4025 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:06 4025 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:06 4025 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:06 4025 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:06 4025 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:06 4025 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:06 4025 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:07 4025 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:07 4025 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:07 4025 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069557927 in the ib_logfiles!
2021-09-12 11:33:07 4025 [Note] InnoDB: Database was not shutdown normally!
2021-09-12 11:33:07 4025 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:07 4025 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:07 4025 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:07 4025 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:33:07 4025 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:07 4025 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:07 4025 [Note] InnoDB: 5.6.36 started; log sequence number 209069557927
2021-09-12 11:33:07 4025 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:07 4025 [Note] IPv6 is available.
2021-09-12 11:33:07 4025 [Note] - '::' resolves to '::';
2021-09-12 11:33:07 4025 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:07 4025 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:07 4025 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:33:12 4068 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:12 4068 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:12 4068 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:12 4068 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:12 4068 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:12 4068 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:12 4068 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:12 4068 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:12 4068 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:12 4068 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:12 4068 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:12 4068 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069557937 in the ib_logfiles!
2021-09-12 11:33:12 4068 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:33:12 4068 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:12 4068 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:13 4068 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:13 4068 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:33:13 4068 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:13 4068 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:13 4068 [Note] InnoDB: 5.6.36 started; log sequence number 209069557937
2021-09-12 11:33:13 4068 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:13 4068 [Note] IPv6 is available.
2021-09-12 11:33:13 4068 [Note] - '::' resolves to '::';
2021-09-12 11:33:13 4068 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:13 4068 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:13 4068 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:33:17 4110 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:17 4110 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:17 4110 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:17 4110 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:17 4110 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:17 4110 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:17 4110 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:17 4110 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:17 4110 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:17 4110 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:17 4110 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:17 4110 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069557937
2021-09-12 11:33:17 4110 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:33:17 4110 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:17 4110 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:18 4110 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:18 4110 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069557947
2021-09-12 11:33:18 4110 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:18 4110 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:18 4110 [Note] InnoDB: 5.6.36 started; log sequence number 209069557947
2021-09-12 11:33:19 4110 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:19 4110 [Note] IPv6 is available.
2021-09-12 11:33:19 4110 [Note] - '::' resolves to '::';
2021-09-12 11:33:19 4110 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:19 4110 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:19 4110 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:33:22 4152 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:22 4152 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:22 4152 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:22 4152 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:22 4152 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:22 4152 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:22 4152 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:23 4152 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:23 4152 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:23 4152 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:23 4152 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:23 4152 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069557957 in the ib_logfiles!
2021-09-12 11:33:23 4152 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:33:23 4152 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:23 4152 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:23 4152 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:23 4152 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:33:24 4152 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:24 4152 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:24 4152 [Note] InnoDB: 5.6.36 started; log sequence number 209069557957
2021-09-12 11:33:24 4152 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:24 4152 [Note] IPv6 is available.
2021-09-12 11:33:24 4152 [Note] - '::' resolves to '::';
2021-09-12 11:33:24 4152 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:24 4152 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:24 4152 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:33:28 4201 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:28 4201 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:28 4201 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:28 4201 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:28 4201 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:28 4201 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:28 4201 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:28 4201 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:28 4201 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:28 4201 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:28 4201 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:28 4201 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069557957
2021-09-12 11:33:28 4201 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:33:28 4201 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:28 4201 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:28 4201 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:28 4201 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069558157
2021-09-12 11:33:29 4201 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2021-09-12 11:33:29 4201 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:29 4201 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:29 4201 [Note] InnoDB: 5.6.36 started; log sequence number 209069558157
2021-09-12 11:33:29 4201 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:29 4201 [Note] IPv6 is available.
2021-09-12 11:33:29 4201 [Note] - '::' resolves to '::';
2021-09-12 11:33:29 4201 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:29 4201 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:29 4201 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:33:37 4265 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:33:37 4265 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:33:37 4265 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:33:37 4265 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:33:37 4265 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:33:37 4265 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:33:37 4265 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:33:37 4265 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:33:37 4265 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:33:37 4265 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:33:37 4265 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:33:37 4265 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069558563 in the ib_logfiles!
2021-09-12 11:33:37 4265 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:33:37 4265 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:33:37 4265 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:33:37 4265 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:33:37 4265 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:33:38 4265 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:33:38 4265 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:33:38 4265 [Note] InnoDB: 5.6.36 started; log sequence number 209069558563
2021-09-12 11:33:38 4265 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:33:38 4265 [Note] IPv6 is available.
2021-09-12 11:33:38 4265 [Note] - '::' resolves to '::';
2021-09-12 11:33:38 4265 [Note] Server socket created on IP: '::'.
2021-09-12 11:33:38 4265 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:33:38 4265 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:36:14 4355 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:14 4355 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:14 4355 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:14 4355 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:14 4355 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:14 4355 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:14 4355 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:14 4355 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:14 4355 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:14 4355 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:14 4355 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:14 4355 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069568015 in the ib_logfiles!
2021-09-12 11:36:14 4355 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:14 4355 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:14 4355 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:14 4355 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:14 4355 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:36:14 4355 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:14 4355 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:14 4355 [Note] InnoDB: 5.6.36 started; log sequence number 209069568015
2021-09-12 11:36:14 4355 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:14 4355 [Note] IPv6 is available.
2021-09-12 11:36:14 4355 [Note] - '::' resolves to '::';
2021-09-12 11:36:14 4355 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:14 4355 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:14 4355 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:36:21 4404 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:21 4404 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:21 4404 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:21 4404 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:21 4404 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:21 4404 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:21 4404 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:21 4404 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:21 4404 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:21 4404 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:21 4404 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:21 4404 [Note] InnoDB: The log sequence numbers 209069373861 and 209069373861 in ibdata files do not match the log sequence number 209069568235 in the ib_logfiles!
2021-09-12 11:36:21 4404 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:21 4404 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:21 4404 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:21 4404 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:21 4404 [Note] InnoDB: from the doublewrite buffer...
2021-09-12 11:36:21 4404 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:21 4404 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:21 4404 [Note] InnoDB: 5.6.36 started; log sequence number 209069568235
2021-09-12 11:36:21 4404 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:21 4404 [Note] IPv6 is available.
2021-09-12 11:36:21 4404 [Note] - '::' resolves to '::';
2021-09-12 11:36:21 4404 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:21 4404 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:21 4404 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:36:25 4449 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:25 4449 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:25 4449 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:25 4449 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:25 4449 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:25 4449 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:25 4449 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:25 4449 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:25 4449 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:25 4449 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:25 4449 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:25 4449 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568235
2021-09-12 11:36:25 4449 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:25 4449 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:25 4449 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:25 4449 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:25 4449 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568440
2021-09-12 11:36:25 4449 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2021-09-12 11:36:26 4449 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:26 4449 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:26 4449 [Note] InnoDB: 5.6.36 started; log sequence number 209069568440
2021-09-12 11:36:26 4449 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:26 4449 [Note] IPv6 is available.
2021-09-12 11:36:26 4449 [Note] - '::' resolves to '::';
2021-09-12 11:36:26 4449 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:26 4449 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:26 4449 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:36:28 4503 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:28 4503 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:28 4503 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:28 4503 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:28 4503 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:28 4503 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:28 4503 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:28 4503 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:28 4503 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:28 4503 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:28 4503 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:28 4503 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568235
2021-09-12 11:36:28 4503 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:28 4503 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:28 4503 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:28 4503 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:28 4503 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568450
2021-09-12 11:36:28 4503 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2021-09-12 11:36:29 4503 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:29 4503 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:29 4503 [Note] InnoDB: 5.6.36 started; log sequence number 209069568450
2021-09-12 11:36:29 4503 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:29 4503 [Note] IPv6 is available.
2021-09-12 11:36:29 4503 [Note] - '::' resolves to '::';
2021-09-12 11:36:29 4503 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:29 4503 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:29 4503 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:36:32 4551 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:32 4551 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:32 4551 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:32 4551 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:32 4551 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:32 4551 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:32 4551 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:32 4551 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:32 4551 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:32 4551 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:32 4551 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:32 4551 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568450
2021-09-12 11:36:32 4551 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:32 4551 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:32 4551 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:32 4551 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:32 4551 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568460
2021-09-12 11:36:32 4551 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:32 4551 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:32 4551 [Note] InnoDB: 5.6.36 started; log sequence number 209069568460
2021-09-12 11:36:32 4551 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:32 4551 [Note] IPv6 is available.
2021-09-12 11:36:32 4551 [Note] - '::' resolves to '::';
2021-09-12 11:36:32 4551 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:32 4551 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:32 4551 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:36:35 4601 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:35 4601 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:35 4601 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:35 4601 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:35 4601 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:35 4601 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:35 4601 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:35 4601 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:35 4601 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:35 4601 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:35 4601 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:35 4601 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568460
2021-09-12 11:36:35 4601 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:35 4601 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:35 4601 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:35 4601 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:35 4601 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568470
2021-09-12 11:36:36 4601 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:36 4601 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:36 4601 [Note] InnoDB: 5.6.36 started; log sequence number 209069568470
2021-09-12 11:36:36 4601 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:36 4601 [Note] IPv6 is available.
2021-09-12 11:36:36 4601 [Note] - '::' resolves to '::';
2021-09-12 11:36:36 4601 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:36 4601 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:36 4601 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:36:39 4641 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:39 4641 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:39 4641 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:39 4641 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:39 4641 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:39 4641 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:39 4641 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:39 4641 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:39 4641 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:39 4641 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:39 4641 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:39 4641 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568470
2021-09-12 11:36:39 4641 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:39 4641 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:39 4641 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:39 4641 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:39 4641 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568480
2021-09-12 11:36:39 4641 [Note] InnoDB: 128 rollback segment(s) are active.
2021-09-12 11:36:39 4641 [Note] InnoDB: Waiting for purge to start
2021-09-12 11:36:39 4641 [Note] InnoDB: 5.6.36 started; log sequence number 209069568480
2021-09-12 11:36:39 4641 [Note] Server hostname (bind-address): '*'; port: 3306
2021-09-12 11:36:39 4641 [Note] IPv6 is available.
2021-09-12 11:36:39 4641 [Note] - '::' resolves to '::';
2021-09-12 11:36:39 4641 [Note] Server socket created on IP: '::'.
2021-09-12 11:36:39 4641 [Note] Event Scheduler: Loaded 0 events
2021-09-12 11:36:39 4641 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.36' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)

2021-09-12 11:36:42 4689 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 11:36:42 4689 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:36:42 4689 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:36:42 4689 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 11:36:42 4689 [Note] InnoDB: Memory barrier is not used
2021-09-12 11:36:42 4689 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 11:36:42 4689 [Note] InnoDB: Using Linux native AIO
2021-09-12 11:36:42 4689 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 11:36:42 4689 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-09-12 11:36:42 4689 [Note] InnoDB: Completed initialization of buffer pool
2021-09-12 11:36:42 4689 [Note] InnoDB: Highest supported file format is Barracuda.
2021-09-12 11:36:42 4689 [Note] InnoDB: Log scan progressed past the checkpoint lsn 209069568470
2021-09-12 11:36:42 4689 [Note] InnoDB: InnoDB: Database was not shutdown normally!
2021-09-12 11:36:42 4689 [Note] InnoDB: Starting crash recovery.
2021-09-12 11:36:42 4689 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-09-12 11:36:42 4689 [Note] InnoDB: Restoring possible half-written data pages
2021-09-12 11:36:42 4689 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 209069568490 ← この後に mysqld が停止してログ無しみたい

2021-09-12 11:47:07 1622 [Note] Plugin 'FEDERATED' is disabled. ←サーバー手動再起動後
2021-09-12 11:47:07 1622 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 11:47:07 1622 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 11:47:07 1622 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
-------------------------------------------------------------------------
「InnoDB: Database was not shutdown normally!」というログが短時間のうちに複数あり、再起動を繰り返しているのが分かる。
これ以前を解析していけばシャットダウンの原因が分かるはず。

もう少しログをさかのぼったらこんな記述も見つけた。これも mysql が停止したタイミングもログと思われる
-------------------------------------------------------------------------
2021-09-12 09:01:57 1810 [Note] Plugin 'FEDERATED' is disabled.
2021-09-12 09:01:57 1810 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-09-12 09:01:57 1810 [Note] InnoDB: The InnoDB memory heap is disabled
2021-09-12 09:01:57 1810 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-12 09:01:57 1810 [Note] InnoDB: Memory barrier is not used
2021-09-12 09:01:57 1810 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-09-12 09:01:57 1810 [Note] InnoDB: Using Linux native AIO
2021-09-12 09:01:57 1810 [Note] InnoDB: Using CPU crc32 instructions
2021-09-12 09:01:57 1810 [Note] InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
2021-09-12 09:01:57 1810 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2021-09-12 09:01:57 1810 [ERROR] Plugin 'InnoDB' init function returned error.
2021-09-12 09:01:57 1810 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2021-09-12 09:01:57 1810 [ERROR] Unknown/unsupported storage engine: InnoDB
2021-09-12 09:01:57 1810 [ERROR] Aborting

2021-09-12 09:01:57 1810 [Note] Binlog end
2021-09-12 09:01:57 1810 [Note] Shutting down plugin 'partition'
2021-09-12 09:01:57 1810 [Note] Shutting down plugin 'ARCHIVE'
2021-09-12 09:01:57 1810 [Note] Shutting down plugin 'BLACKHOLE'
2021-09-12 09:01:57 1810 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
-------------------------------------------------------------------------

短い時間で解決策を調べたけど不明。

とりあえずバッファプールへのメモリ割り当てを上げておく。
参考ページ:https://tech.librastudio.co.jp/entry/index.php/2018/05/22/post-1891/

現状を確認。128Mであることが分かる
--------------------------------------------------------
mysql> SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
+-------------------------+-----------+
| Variable_name | Value |
+-------------------------+-----------+
| innodb_buffer_pool_size | 134217728 |
+-------------------------+-----------+
--------------------------------------------------------

my.cnf を編集(my.conf でない)
--------------------------------------------------------
vi my.cnf
--------------------------------------------------------

--------------------------------------------------------
[mysqld]
...
innodb_buffer_pool_size=512M
--------------------------------------------------------

mysql を再起動
--------------------------------------------------------
/etc/init.d/mysqld restart
--------------------------------------------------------

変更できていることを確認
--------------------------------------------------------
mysql> SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
+-------------------------+-----------+
| Variable_name | Value |
+-------------------------+-----------+
| innodb_buffer_pool_size | 536870912 |
+-------------------------+-----------+
--------------------------------------------------------

ここも後で見る
MySQLパフォーマンスチューニング -my.cnfの見直し-

サーバーがつながらない。CLOSE_WAITが大量発生

参考ページ1:https://end0tknr.hateblo.jp/entry/20110724/1311490171
参考ページ2:https://hacknote.jp/archives/19502/
参考ページ3:https://seesaawiki.jp/w/dehio3/d/solaris%A1%C1CLOSE_WAIT%C8%AF%C0%B8%A1%C1
参考ページ4:https://hirofukami.com/2008/08/11/close-wait/

 

サーバーがつながらない事態が頻発

ネットワークの状況を確認
---------------------------------------
netstat -tpn
---------------------------------------

結果
---------------------------------------
tcp 0 0 160.16.63.○○○:443 207.46.13.76:15571 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:40457 CLOSE_WAIT 11805/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:11251 CLOSE_WAIT 11567/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:11982 CLOSE_WAIT 11753/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:54793 CLOSE_WAIT 12106/httpd
tcp 0 0 160.16.63.○○○:443 126.34.252.62:59582 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:53009 CLOSE_WAIT 12076/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:57869 CLOSE_WAIT 11894/httpd
tcp 521 0 160.16.63.○○○:443 36.13.167.104:37684 ESTABLISHED -
tcp 0 0 160.16.63.○○○:443 13.66.139.0:19846 TIME_WAIT -
tcp 0 0 160.16.63.○○○:443 112.137.96.56:59062 TIME_WAIT -
tcp 0 0 160.16.63.○○○:443 125.8.175.235:55073 TIME_WAIT -
tcp 0 0 160.16.63.○○○:443 110.232.6.71:62611 TIME_WAIT -
tcp 518 0 160.16.63.○○○:443 210.194.87.121:50331 CLOSE_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:9959 CLOSE_WAIT 12046/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:14902 CLOSE_WAIT 11726/httpd
tcp 0 0 160.16.63.○○○:443 60.153.198.22:42116 ESTABLISHED -
tcp 0 0 160.16.63.○○○:443 133.201.90.32:62121 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:13430 CLOSE_WAIT 11769/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:35291 CLOSE_WAIT 12005/httpd
tcp 0 0 160.16.63.○○○:443 13.66.139.0:19845 TIME_WAIT -
tcp 517 0 160.16.63.○○○:443 111.239.35.98:65278 ESTABLISHED -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:42621 CLOSE_WAIT 12102/httpd
tcp 0 0 160.16.63.○○○:443 211.127.83.36:57156 ESTABLISHED 12020/httpd
tcp 0 0 160.16.63.○○○:443 126.34.252.62:59581 TIME_WAIT -
tcp 0 0 160.16.63.○○○:443 111.64.18.75:62476 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:6615 CLOSE_WAIT 11750/httpd
tcp 0 0 160.16.63.○○○:443 72.14.199.110:37727 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:4216 CLOSE_WAIT 12023/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:15151 CLOSE_WAIT 12123/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:9227 CLOSE_WAIT 11850/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:13495 CLOSE_WAIT 11863/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:32764 CLOSE_WAIT 11987/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:17980 CLOSE_WAIT 11958/httpd
tcp 0 0 160.16.63.○○○:443 133.206.82.0:42008 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:38943 CLOSE_WAIT 12051/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:33367 CLOSE_WAIT 12138/httpd
tcp 517 0 160.16.63.○○○:443 112.137.96.56:59073 ESTABLISHED -
tcp 0 0 160.16.63.○○○:443 112.137.96.56:59064 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:43395 CLOSE_WAIT 12149/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:28548 CLOSE_WAIT 11806/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:39903 CLOSE_WAIT 12145/httpd
tcp 0 0 160.16.63.○○○:443 126.53.57.5:32806 ESTABLISHED 11930/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:49601 CLOSE_WAIT 11609/httpd
tcp 0 0 160.16.63.○○○:443 207.46.13.76:15559 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:15573 CLOSE_WAIT 12009/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:62567 CLOSE_WAIT 12033/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:7693 CLOSE_WAIT 11818/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:21720 CLOSE_WAIT 11801/httpd
tcp 0 0 160.16.63.○○○:443 207.46.13.76:15574 ESTABLISHED -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:49345 CLOSE_WAIT 11882/httpd
tcp 0 0 160.16.63.○○○:443 207.46.13.76:15569 TIME_WAIT -
tcp 0 0 160.16.63.○○○:443 203.133.150.50:55961 TIME_WAIT -
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:17553 CLOSE_WAIT 11804/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:23842 CLOSE_WAIT 11762/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:7292 CLOSE_WAIT 11814/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:23870 CLOSE_WAIT 11528/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:32023 CLOSE_WAIT 12062/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:62955 CLOSE_WAIT 11989/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:8371 CLOSE_WAIT 11919/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:22436 CLOSE_WAIT 11682/httpd
tcp 1 0 160.16.63.○○○:443 94.177.○○.○○:21309 CLOSE_WAIT 11913/httpd
---------------------------------------

同じIPアドレスからのアクセスで CLOSE_WAIT が大量発生しているのが分かる

/var/log/httpd/error_log には child process 13944 still did not exit, sending a SIGTERM が残っている。(関係あるかは不明)
---------------------------------------
[Wed Sep 08 21:23:17 2021] [warn] child process 13944 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 13951 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 13955 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 13970 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 13984 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 14001 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 14012 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 14031 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:17 2021] [warn] child process 14044 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:18 2021] [warn] child process 14070 still did not exit, sending a SIGTERM
[Wed Sep 08 21:23:18 2021] [warn] child process 14075 still did not exit, sending a SIGTERM
---------------------------------------

 

CLOSE_WAIT  は通信相手から自分への通信はcloseしたが、自分側は完全にcloseしていない状態。
デフォルトの有効時間は7200秒(2時間)。

CLOSE_WAIT  の有効時間は net.ipv4.tcp_keepalive_time ディレクティブで変更可能。

今の値を確認
---------------------------------------
# sysctl -n net.ipv4.tcp_keepalive_time
7200
---------------------------------------

 

利用できる全ての値を表示して net.ipv4.tcp_keepalive_time があることを確認
---------------------------------------
sysctl -a
---------------------------------------

7200の値を少なくすれば改善されるはず

一時的な変更方法
3600秒に変更
---------------------------------------
echo 3600 > /proc/sys/net/ipv4/tcp_keepalive_time
---------------------------------------

永続的な変更方法
---------------------------------------
# vi /etc/sysctl.conf

net.ipv4.tcp_keepalive_time = 10

net.ipv4.tcp_keepalive_probes = 2

net.ipv4.tcp_keepalive_intvl = 3
---------------------------------------

を追記
---------------------------------------
# sysctl -w
---------------------------------------

変更を反映
---------------------------------------
# sysctl -p
---------------------------------------

apacheでDoS攻撃対策。mod_evasiveのインストール

apacheでDoS攻撃対策。mod_evasiveのインストール

同一IPアドレスからのアクセスに回数制限を設けるのにApache のモジュールがいくつかある。

有名なのはここらへん。
mod_dosdetector
mod_evasive
Fail2ban ← 人気
簡単そうなので mod_evasive をインストールする。

インストール参考:https://okochang.hatenablog.jp/entry/2014/03/15/161134

以下手順。
読み込まれているモジュールの一覧
-----------------
httpd -M
-----------------

sharedのモジュールは、/etc/httpd/conf.modules.d配下の定義ファイルを編集することで、モジュールをロードしたり、ロードしないようにすることができる。

mod_evasive が無いことを確認
-----------------
httpd -M | grep evasive
-----------------

EPEL(イーペル)リポジトリにあるらしいので設定されいてるリポジトリを見る

-----------------
yum repolist all
-----------------

EPELがあることを確認する
mod_evasiveのインストール
-----------------
yum --enablerepo=epel install mod_evasive
-----------------

エラーが出る
-----------------
# yum --enablerepo=epel install mod_evasive
読み込んだプラグイン:fastestmirror, security
インストール処理の設定をしています
Determining fastest mirrors
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
Eg. Invalid release/repo/arch combination/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/6/base/mirrorlist.txt
エラー: Cannot find a valid baseurl for repo: base
-----------------

インストール可能なパッケージとインストール済みのパッケージを見ようとしてもエラーが出る
-----------------
yum list
-----------------

CentOSのバージョンが6.9と古いから出るエラーらしい。

対処参考URL1:https://shobon.hatenablog.com/entry/2020/12/03/212209
対処参考URL2:https://ex1.m-yabe.com/archives/5066

ページを参考に設定ファイルを書き変え
-----------------
/etc/yum.repos.d/CentOS-Base.repo
-----------------
書き変え後、一覧が出るか確認 → OK。
-----------------
yum list | grep evasive
-----------------

再度インストール実行
-----------------
yum --enablerepo=epel install mod_evasive
-----------------

インストールできているか確認
-----------------
httpd -M | grep evasive
-----------------

結果
-----------------
Syntax OK
evasive20_module (shared)
-----------------

設定ファイルの編集
うちのサーバーはここにあった
-----------------
cat /etc/httpd/conf.d/mod_evasive.conf
-----------------

モジュールに付属のテストファイルを実行
-----------------
# perl /usr/share/doc/mod_evasive-1.10.1/test.pl
-----------------

エラーが出る。最近の apache ではリクエストの改行コードがCRLFでないため 400 Bad Request が出る。

下記を参考に書き変え

https://www.kowloonet.net/2020/07/centos8-10apachedosmodevasive.html

再度実行してok。

 

設定ファイルからログファイルの保存場所を変更
------------------------------------------------
# vi /etc/httpd/conf.d/mod_evasive.conf
LoadModule evasive20_module modules/mod_evasive24.so
<IfModule mod_evasive24.c>
DOSHashTableSize 3097
DOSPageCount 10
DOSSiteCount 30
DOSPageInterval 2
DOSSiteInterval 1
DOSBlockingPeriod 60
DOSLogDir "/var/log/mod_evasive"
DOSWhitelist 127.0.0.1
</IfModule>
------------------------------------------------
保存先のディレクトリを作成して権限を与える
------------------------------------------------
# mkdir /var/log/mod_evasive/
# chown apache:apache /var/log/mod_evasive/
------------------------------------------------
二つあるサーバーのうち、一方は test.pl を実行すると 301 が返ってきてしまう。調べたけど理由不明。

MaxClients及びServerLimitのベストな設定値

参考ページ:https://www.netmarvs.com/archives/3518

上記ページでは次のような式で MaxClients の値を求めている。
-----------------------------------------------
MaxClients = サーバ合計メモリ * (80%〜70%) / httpdプロセスの最大メモリ使用量(子httpdピーク時の平均値です。)
-----------------------------------------------

これを自分の環境の置き換えると
-----------------------------------------------
MaxClients = 4G(さくらのvpsサーバ合計メモリ) * (80%〜70%) / httpdプロセスの最大メモリ使用量(子httpdピーク時の平均値です。)
-----------------------------------------------

 

 

1プロセス当たりのメモリ使用量の調べ方

-----------------------------------------------
ps aux
-----------------------------------------------

結果(一部)
-----------------------------------------------
# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 19356 624 ? Ss Aug27 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S Aug27 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Aug27 0:00 [migration/0]
root 4 0.0 0.0 0 0 ? S Aug27 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S Aug27 0:00 [stopper/0]
root 6 0.0 0.0 0 0 ? S Aug27 0:00 [watchdog/0]
root 7 0.0 0.0 0 0 ? S Aug27 0:00 [migration/1]
root 8 0.0 0.0 0 0 ? S Aug27 0:00 [stopper/1]
root 9 0.0 0.0 0 0 ? S Aug27 0:00 [ksoftirqd/1]
root 10 0.0 0.0 0 0 ? S Aug27 0:00 [watchdog/1]
root 11 0.0 0.0 0 0 ? S Aug27 0:00 [migration/2]
root 12 0.0 0.0 0 0 ? S Aug27 0:00 [stopper/2]
root 13 0.0 0.0 0 0 ? S Aug27 0:00 [ksoftirqd/2]
root 14 0.0 0.0 0 0 ? S Aug27 0:00 [watchdog/2]
root 15 0.0 0.0 0 0 ? S Aug27 0:00 [migration/3]
root 16 0.0 0.0 0 0 ? S Aug27 0:00 [stopper/3]
root 17 0.0 0.0 0 0 ? S Aug27 0:00 [ksoftirqd/3]
root 18 0.0 0.0 0 0 ? S Aug27 0:00 [watchdog/3]
root 19 0.0 0.0 0 0 ? S Aug27 0:03 [events/0]
root 20 0.0 0.0 0 0 ? S Aug27 0:04 [events/1]
root 21 0.0 0.0 0 0 ? S Aug27 0:06 [events/2]
root 22 0.0 0.0 0 0 ? S Aug27 0:04 [events/3]
root 23 0.0 0.0 0 0 ? S Aug27 0:00 [events/0]

(中略)

apache 5406 0.2 0.3 376408 14648 ? S 13:36 0:01 /usr/sbin/httpd
apache 5407 0.3 0.3 376316 15168 ? S 13:36 0:02 /usr/sbin/httpd
apache 5408 0.1 0.3 377308 15624 ? S 13:36 0:00 /usr/sbin/httpd
apache 5409 0.1 0.3 376404 14676 ? S 13:36 0:01 /usr/sbin/httpd
apache 5412 0.3 0.3 376896 15140 ? S 13:36 0:01 /usr/sbin/httpd
apache 5413 0.2 0.4 379216 17620 ? S 13:36 0:01 /usr/sbin/httpd
apache 5420 0.4 0.4 380156 18808 ? S 13:37 0:02 /usr/sbin/httpd
apache 5422 0.3 0.3 376148 14596 ? S 13:37 0:01 /usr/sbin/httpd
apache 5423 0.5 0.4 430376 16852 ? S 13:38 0:02 /usr/sbin/httpd
apache 5424 0.3 0.3 430116 16212 ? S 13:38 0:01 /usr/sbin/httpd
apache 5455 0.5 0.3 376884 15288 ? S 13:42 0:01 /usr/sbin/httpd
apache 5456 0.2 0.3 376132 14268 ? S 13:42 0:00 /usr/sbin/httpd
apache 5457 0.4 0.3 376644 14904 ? S 13:42 0:00 /usr/sbin/httpd
apache 5458 0.4 0.3 376144 14480 ? S 13:42 0:00 /usr/sbin/httpd
apache 5459 0.1 0.3 375904 13836 ? S 13:43 0:00 /usr/sbin/httpd
apache 5460 0.4 0.3 377724 15548 ? S 13:43 0:00 /usr/sbin/httpd
apache 5461 0.1 0.3 375220 13224 ? S 13:43 0:00 /usr/sbin/httpd
apache 5462 0.2 0.3 375472 13428 ? S 13:43 0:00 /usr/sbin/httpd
apache 5463 0.4 0.3 374864 12720 ? S 13:44 0:00 /usr/sbin/httpd
apache 5464 0.2 0.3 375224 13160 ? S 13:44 0:00 /usr/sbin/httpd
apache 5468 3.0 0.3 374876 12584 ? S 13:45 0:00 /usr/sbin/httpd
root 5471 0.0 0.0 110252 1132 pts/0 R+ 13:45 0:00 ps aux
-----------------------------------------------

書きかけだけどまだまだメモリに余裕があったため MaxClients は適当に増やしたので、生地は未完

サーバーが重い、ページが開かない 20210819

運営サイトでサーバーが重くなった。タイムアウトでページが開かないことが多発した。
ipアドレスでアクセスした場合は問題なし。ドメインでアクセスした場合のみ問題が発生。

ニ三日前からページにアクセスしづらい状態がたびたび起こっている。ちょうどデーターベースの書き換えでサーバーに負荷のかかる作業だったので、それが原因かとデータベースを削除したり解決策を検索したが解決に至らず。

さくらインターネットに問い合わせて見る。

 

送ったメール内容
---------------------------------------------------------

サーバーに利用制限がかかっていませんか?
二日ほど前多少サーバーに負荷のかかる処理をしました。そのあとからページの表示がされにくくなりました。
ブラウザからページにアクセスしても「サーバーが応答しません」と出ます。
ipアドレスでアクセスした場合は問題なく表示されますが、ドメイン名でアクセスした場合のみ問題が起こります。
http://160.○○○.○○○.○○○/index.html ← 問題ない
http://独自ドメイン/index.html ← 表示されない
調査をお願いします。
---------------------------------------------------------

一時間ほどで返信あり。
返信の内容
---------------------------------------------------------
現時点で弊社にて実施している制限はございません。

また、IPアドレスでブラウズ可能ということでございましたら仮想サーバ
自体の障害や制限によるものとは考えにくいものでございます。

恐れ入りますが「さくらのVPS」は、お客様に管理者権限をお渡ししている
サービスとなり仮想サーバの構築・運用・保守にいたる管理全般をお客様にて
実施いただいております。

そのため、弊社ではお客様のご利用いただく仮想サーバ内のシステム状況
などを把握や調査、トラブルシューティング等の技術的なサポートは承って
おりません。

お手数ですが、仮想サーバ内の設定やドメインのゾーンを管理するネーム
サーバに問題がないなど、サーバ管理者様にてご確認いただきますよう
お願いいたします。
---------------------------------------------------------
このメールを受け取った前後に再度ページにアクセスしたら滞りなくページが開いた。
時間が解決してくれたということか?

一週間後・・・。

再びアクセスエラー。

再調査する。

このページが参考になりそう

https://meteoricstream.com/tips_detail/45.html

したこと。

手元の windows コマンドプロンプトから ping 送信 → 問題なし。

 

アクセスログを調べる。
rootで
cat /var/log/httpd/access_log

エラーログを調べる。
cat /var/log/httpd/error_log

 

エラーログ一部抜粋
-------------------------------------------------------------------------------------------
[Fri Aug 27 01:28:32 2021] [error] server reached MaxClients setting, consider raising the MaxClients setting
-------------------------------------------------------------------------------------------
訳:サーバーは MaxClients の設定値に達しました。MaxClients の設定値を大きくすることを検討してください。
setting → 設定(値)

↑ 上記メッセージは、Webサーバの起動後、リクエスト数がはじめてクライアントの同時接続数に達した際に一度だけ出るみたい。

 

MaxClients の値を上げるには httpd.conf の値を変えればいいらしい。

cat /etc/httpd/conf/httpd.conf

 

httpd.conf を開くと MaxClients ディレクティブが二つあった
httpd.conf 一部抜粋
-------------------------------------------------------------------------------------------
# prefork MPM

# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers 4
MaxClients 300
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
-------------------------------------------------------------------------------------------

workerで動いてるか、preforkで動いてるか確認

コンソールで apachectl -l もしくは httpd -l を実行
apachectl と httpd はほぼ同じ内容で、どちらも httpd の軌道や停止をするコマンド

-------------------------------------------------------------------------------------------
# apachectl -l
Compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c
-------------------------------------------------------------------------------------------

prefork で動いているという事らしい
デフォルトでは prefork ということ

httpd -V もしくは apachectl -V コマンドでも確認できる
Server MPM: の部分を確認

結果
-------------------------------------------------------------------------------------------
# httpd -V
Server version: Apache/2.2.15 (Unix)
Server built: Oct 19 2017 16:43:38
Server's Module Magic Number: 20051115:25
Server loaded: APR 1.3.9, APR-Util 1.3.9
Compiled using: APR 1.3.9, APR-Util 1.3.9
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
-------------------------------------------------------------------------------------------

 

httpd.confより抜粋
-------------------------------------------------------------------------------------------
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>
-------------------------------------------------------------------------------------------

同時接続の上限はこの場合 256 になる。

 

問題が起きているときのプロセスの数を確認すると 258 であり、上限を超えている
-------------------------------------------------------------------------------------------
ps aux |grep httpd |wc -l
-------------------------------------------------------------------------------------------

httpd.conf を書き変えて、MaxClientsを増やす。

ServerLimit ディレクティブも同じように増やす必要がある。

ではベストな MaxClients はいくつなのか

MaxClients及びServerLimitのベストな設定値

 

また、ネットワークの状態を確認して、同一ipアドレスから大量のリクエストが来ていないか確認する

コマンド
-------------------------------------------------------------------------------------------
netstat -tpn
-------------------------------------------------------------------------------------------

結果
-------------------------------------------------------------------------------------------
tcp 0 0 160.16.63.○○○:443 154.21.212.○○:52041 SYN_RECV -
tcp 0 0 160.16.63.○○○:443 154.21.212.○○:65281 SYN_RECV -
tcp 0 0 160.16.63.○○○:443 154.21.212.○○:50095 SYN_RECV -
tcp 0 0 160.16.63.○○○:443 154.21.212.○○:62694 SYN_RECV -
tcp 0 0 160.16.63.○○○:443 154.21.212.○○:49490 SYN_RECV -
tcp 0 0 160.16.63.○○○:443 154.21.212.○○:64857 SYN_RECV -
tcp 0 0 160.16.63.○○○:443 154.21.212.○○:57942 SYN_RECV -
tcp 0 0 160.16.63.○○○:443 154.21.212.○○:51261 SYN_RECV -
-------------------------------------------------------------------------------------------
同一ipアドレスからの大量アクセスを確認。

.htaccess を編集して ipアドレスに制限を掛ける

-------------------------------------------------------------------------------------------
#ipアドレスの制限
order allow,deny
allow from all
deny from 154.21.212.○○
-------------------------------------------------------------------------------------------

禁止ipアドレスでアクセスするとcentOS のテストページが表示されてしまうので、forbidden が出るよう設定を変更

参考:https://www.dataplan.jp/blog/server/2089

問題のipアドレスを探す方法として、tcpdump でパケットキャプチャを仕掛けて分析するという方法もあるらしい。

↓サーバーを再起動するとerror_log に残る
-------------------------------------------------------------------------------------------
[Fri Aug 27 02:02:07 2021] [notice] caught SIGTERM, shutting down
-------------------------------------------------------------------------------------------

 

【さくらのVPS】WEBページが表示されない

ある日突然WEBページが表示されなくなった。
表示されなくなったサイトとは別のサイトのお問い合わせメールから知る。

アクセスするが確かに表示されない。2つ契約しているVPSサーバーのもう片方は問題なく表示できる。表示できないサイトと同じサーバーで運営している別サイトも表示できず。

さくらのVPSサーバーのコントロールパネルにはログイン可能。
特に問題も当たらず。

さくらのコールセンターに問い合わせる。

サーバーには問題ないとの答え。

ただ、ポート80(http)と443(https)が不通で通信ができてないとのこと。

問題を解決するにはポートを開放するか、apache の状態を確認しろと助言をもらう。

ターミナルからサーバーにアクセス。

apacheの状態を確認

//コマンド
----------------------------
/etc/init.d/httpd status
----------------------------

//結果
----------------------------
httpd status unknown due to insufficient privileges.
----------------------------

権限がないという事で root に変更して確認

//結果
----------------------------
httpd は停止しています。
----------------------------

やっぱり動いてなかった。

//apache を起動
----------------------------
service httpd start
----------------------------

//結果
----------------------------
httpd を起動中
----------------------------

無事WEBページを表示できた。

1 / 212