Linux VPS服务器内存不够用的情况

自从把我的一个网站搬家到linode 1G 的vps上后，总感觉有些不太正常。首先dnspod监控上显示网站有时能访问，有时不行。然后wp supercache插件在执行预缓存任务时经常失败后重启，邮件内容如“[http://blog.nbqykj.cn] 预缓存可能已失去响应。预缓存已经重新启动。”。还有一个情况是linode有时会发来磁盘io使用高的报警邮件。

今天抽空检查了下服务器，发现一个现象是mysql进程频繁无故重启。

150424 17:41:14 [Note] Event Scheduler: Loaded 0 events
150424 17:41:14 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.5.37' socket: '/tmp/mysql.sock' port: 3306 Source distribution
150424 19:27:16 mysqld_safe Number of processes running now: 0
150424 19:27:16 mysqld_safe mysqld restarted
150424 19:27:24 [Note] Plugin 'InnoDB' is disabled.
150424 19:27:24 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
150424 19:27:24 [Note] - '0.0.0.0' resolves to '0.0.0.0';
150424 19:27:24 [Note] Server socket created on IP: '0.0.0.0'.
150424 19:27:24 [Warning] 'user' entry 'root@li676-235' ignored in --skip-name-resolve mode.
150424 19:27:24 [Warning] 'proxies_priv' entry '@ root@li676-235' ignored in --skip-name-resolve mode.
150424 19:27:25 [Note] Event Scheduler: Loaded 0 events
150424 19:27:25 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.5.37' socket: '/tmp/mysql.sock' port: 3306 Source distribution

通过这个日志，mysql在提示他运行困难，也就是表示服务器资源不够用了，接下来开始检查。

The error log message “mysqld_safe Number of processes running now: 0″ indicates scarcity for resources to pursue the operations.

运行free -m ，发现空闲内存还有很多，差不多才用一半。

[root@li676-235 ~]# free -m
 total used free shared buffers cached
Mem: 990 903 87 0 114 351
-/+ buffers/cache: 436 554
Swap: 255 53 202

为了确定服务器资源是否真的不够用了，方法是查看系统日志中是否有oom（Out Of memory） killer运行过，果然在日志中发现有很多类似日志。

[root@li676-235 var]# egrep -i "oom|kill|mysql" /var/log/messages |more
Apr 23 13:36:16 li676-235 kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Apr 23 13:36:16 li676-235 kernel: mysqld cpuset=/ mems_allowed=0
Apr 23 13:36:16 li676-235 kernel: CPU: 0 PID: 16020 Comm: mysqld Not tainted 3.18.5-x86_64-linode52 #1
Apr 23 13:36:16 li676-235 kernel: [<ffffffff8112695f>] ? oom_kill_process+0x65/0x32f
Apr 23 13:36:16 li676-235 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 23 13:36:16 li676-235 kernel: [12128] 0 12128 26564 1 12 71 0 mysqld_safe
Apr 23 13:36:16 li676-235 kernel: [12405] 501 12405 155926 2868 120 3676 0 mysqld
Apr 23 13:36:16 li676-235 kernel: Out of memory: Kill process 9703 (php-fpm) score 41 or sacrifice child
Apr 23 13:36:16 li676-235 kernel: Killed process 9703 (php-fpm) total-vm:266976kB, anon-rss:38932kB, file-rss:0kB
Apr 23 13:36:23 li676-235 kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Apr 23 13:36:24 li676-235 kernel: mysqld cpuset=/ mems_allowed=0
Apr 23 13:36:24 li676-235 kernel: CPU: 0 PID: 12405 Comm: mysqld Not tainted 3.18.5-x86_64-linode52 #1
Apr 23 13:36:24 li676-235 kernel: [<ffffffff8112695f>] ? oom_kill_process+0x65/0x32f
Apr 23 13:36:24 li676-235 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 23 13:36:24 li676-235 kernel: [12128] 0 12128 26564 1 12 71 0 mysqld_safe
Apr 23 13:36:24 li676-235 kernel: [12405] 501 12405 156056 2873 120 3676 0 mysqld
Apr 23 13:36:24 li676-235 kernel: Out of memory: Kill process 18168 (php-fpm) score 38 or sacrifice child
Apr 23 13:36:24 li676-235 kernel: Killed process 18168 (php-fpm) total-vm:263724kB, anon-rss:24872kB, file-rss:0kB

通过这个日志，可以清楚看到oom kill在“Apr 23 13:36:16”被php-fpm触发，也就是php在那时被系统强制kill掉了。我设置php-fpm进程最大可以打开的数量是20。按照20*30 = 600M计算，php占用的资源最大可能会大于600M 。

“Apr 23 13:36:16”时nginx日志如下图所示：

nginx_log_20150424203816