I'm running Apache 2.4.25 on a Raspberry Pi 3A+ with Raspbian Stretch and every first request after booting takes several minutes. The connection does not break it just loads forever.
So far I have:
- made sure that the system is reachable via SSH. So I don't think it's a network issue.
- made sure apache is up-to-date and running via
systemctl status apache2 - checked syslog for apache via
journalctl -u apache2.service. Neither this nor apaches access and error logs show anything before the first request is served. - monitored cpu and memory use using
htop. System is not overwhelmed.
Looking at the output of systemctl status apache2 makes me think it has something to do with number of worker processes being spawned by apache. After boot it shows just the main process:
●くろまる apache2.service - The Apache HTTP Server Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2019年04月15日 10:54:08 UTC; 3min 33s ago Process: 482 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS) Main PID: 775 (apache2) CPU: 303ms CGroup: /system.slice/apache2.service └─775 /usr/sbin/apache2 -k start Apr 15 10:54:07 a_test systemd[1]: Starting The Apache HTTP Server... Apr 15 10:54:08 a_test systemd[1]: Started The Apache HTTP Server.
When I try to connect via http (using Opera 58 on Win 10) the page will load for somewhere between 2 and 10 minutes. The moment the page loads is also the moment apache spawns child processes. It will spawn 6 processes which I guess is 1 to serve the request and 5 spare as is the default in mpm_prefork.conf. It will also spawn these child processes without any requests after 5-10 minutes:
●くろまる apache2.service - The Apache HTTP Server Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2019年04月15日 11:07:03 UTC; 10min ago Process: 481 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS) Main PID: 780 (apache2) CPU: 418ms CGroup: /system.slice/apache2.service ├─ 780 /usr/sbin/apache2 -k start ├─3682 /usr/sbin/apache2 -k start ├─3683 /usr/sbin/apache2 -k start ├─3684 /usr/sbin/apache2 -k start ├─3685 /usr/sbin/apache2 -k start └─3686 /usr/sbin/apache2 -k start Apr 15 11:07:02 a_test systemd[1]: Starting The Apache HTTP Server... Apr 15 11:07:03 a_test systemd[1]: Started The Apache HTTP Server.
But why does it wait so long? I say wait, because the system does not seem busy - virtually no CPU use and lots of free memory.
Once it has child processes the web page is working fine. I also tried restarting or stopping and starting the service via systemctl. However in these cases apache will start with 6 child processes and serve requests immediately. After a reboot it will act as described above.
At this point I don't really know what to do anymore. Web searches for apache and workers or processes or things like that almost always lead to discussion about how to avoid more spawning rather than the opposite.
Configurations are mostly default:
apache.conf: https://hastebin.com/pasajesuqe.apache
only virtualhost: https://hastebin.com/onozijadud.apache
mpm_prefork:
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxRequestWorkers 150
MaxConnectionsPerChild 0
</IfModule>
mods-enabled:
access_compat.load authz_core.load deflate.load mime.load php7.0.load status.load alias.conf authz_host.load dir.conf mpm_prefork.conf reqtimeout.conf alias.load authz_user.load dir.load mpm_prefork.load reqtimeout.load auth_basic.load autoindex.conf env.load negotiation.conf setenvif.conf authn_core.load autoindex.load filter.load negotiation.load setenvif.load authn_file.load deflate.conf mime.conf php7.0.conf status.conf
Additional information requested by user grawity:
$ sudo cat /proc/775/stack [] 0xffffffff
$ sudo cat /proc/775/task/775/stack [] 0xffffffff
$ sudo cat /proc/sys/kernel/random/entropy_avail 94
1 Answer 1
Going of the comment made by @grawity (Thanks!) and what I read in an article on hackaday.com, I was able to fix my problem by installing the package rng-tools.
As far as I now understand it, this was the problem:
Apache needs a minimum amount of entropy to spawn workers. The RPi not having any input devices and having just booted up was struggling to collect enough entropy. While it does have a hardware (pseudo) random number generator, the RPi needs to have the package rng-tools installed to actually feed that entropy into the system entropy pool.
Installing the rng-tools package immediately increased available entropy from less than 100 to over 2000 and apache workers were started.
Workers are now also started directly on boot.
/proc/<pid>/stackand/proc/<pid>/task/*/stackfor the Apache master process (the one with lowest PID), while it's still "hung" immediately after a reboot?/proc/sys/kernel/random/entropy_availas that's probably the most common problem)