One of my websites was constantly throwing "Internal Server Error" errors, and that error appeared as follows in /var/log/apache2/error.log:
Thu Oct 16 17:59:14 2014 (19446): Fatal Error Unable to allocate shared memory segment of 67108864 bytes: mmap: Cannot allocate memory (12)
And that even though 9 GiB of memory was free. Also, only one website was affected, others did run fine. The error appeared even when requesting files that did not exist at all (independent of file type: JPG, PHP etc.). After ten minutes or so, the error would disappear on its own, for a while. Also, after restarting Apache the problem disappeared, for a while.
Reason
The problem was due to hitting the system limit of shmpage (shared memory allocation), similar to this case. This limit effectively does only exist in virtualized (VPS) environments, not on physical machines. Hitting this limit can be confirmed by running "cat /proc/user_beancounters", which in our case would output right after the above error situation: shmpages held=247122 [...] limit=262144 failcnt=10067
. At 4 kiB page size, the max. allowed 262144 shared memory pages correspond to (262144*4096)/1024^3 = 1 GiB of shared memory.
When restarting Apache, the problem would be temporarily resolved because shared memory usage would initially be lower: it reduced the number of shared memory pages held form about 247000 to 148000, as per cat /proc/user_beancounters
.
The number of 67108864 bytes of shared memory to allocated, mentioned in the error message, gives a hint what consumes this much shared memory: it is just the default value of PHP's opcache size of 64 MiB, configured in /etc/php5/cgi/php.ini (because 67108864 / 1024 / 1024 = 64). The problem is not that PHP-FCGI would try to allocate 64 MiB of opcache shared memory once, or once per site, but once per process [source]. PHP-FastCGI processes are reused for several requests [source], so the opcache caching makes still some sense, contrary to this opinion. However, they are relatively short-lived as their number varies depending on site load, so the caching does not add much benefit. And worse, the opcache caches are not shared between the PHP-FastCGI processes, but each one gets its own. With about 10 processes per site, each consuming 64 MiB of shared memory, we quickly hit the 1 GiB shared memory limit of the VPS, like above. To illustrate, these were the values in my case: the free
command indicated the following shared memory usage:
- 69 MB some seconds after stopping apache2
- 136 MB immediately after starting apache2 (
service apache2 start; free
) - 500 – 700 MB some 30 – 180 s after starting apache2
- 989 MB typically when apache2 is running for a long time, very close to the 1 GiB limit already
Nearly all this shared memory usage is created by the php-cgi
processes, as can be seen in top
output (use "b" to toggle to background highlighting, "x" to toggle to highlighting the sort column, and "<" / ">" to switch to SHR as the sort column). Namely, when 991 MB shared memory are consumed, 620 MB (= 9 * 64 MB + 2 * 40 MB) of this was consumed by 11 php-cgi
processes.
Solution
The solution is to use PHP-FPM instead of PHP-FastCGI [source]. Contrary to that source, this works independently of the webserver, also working with Apache2.
After deploying this solution, you can compare the "Zend OPcache" section of phpinfo() scripts from different sites on your server. As you can see from the "Cache hits", "Cache misses", "Cached scripts" and "Cached keys" numbers, there is only one single OPcache for all your PHP sites. However, you can configure opcache parameters different for different sites (like memory usage etc.), and this is also reflected in the phpinfo() output. I can't really make sense of that so far, but assume that memory usage etc. are indeed configured per PHP-FPM process pool. So 128 MiB for one site and 64 MiB for another would allow for 192 MiB total shared memory usage for opcache.
The shared memory situation after switching the site to PHP-FPM was only a 70 MiB max. difference in shared memory between Apache2 and php-fpm running and not running, compared to the 900 MiB earlier. This was generated by five php-fpm
processes running simultaneously, with 40-50 MB shared memory "each". So clearly, the shared memory is indeed shared between php-fpm
processes instead of each having its own.
(Another tip: there are other important ways to optimize OPcache, see this blog post.)
Leave a Reply