Petr Gajdos changed bug 1026212
What Removed Added
Priority P5 - None P4 - Low
CC   pgajdos@suse.com

Comment # 1 on bug 1026212 from 
(In reply to George Anchev from comment #0)
> I have been experiencing this so many times and I can't find a pattern. The
> system boots normally but the few sites which I have on the local machine
> don't load (404). Running 'rcapache2 restart' makes everything work but I
> can't find why it sometimes happens and based on what factors.
> 
> Looking at /var/log/apache2/error_log:
> 
> -------------------------------------
> [Sun Feb 19 16:28:30.000427 2017] [:notice] [pid 1767] ModSecurity for
> Apache/2.9.0 (http://www.modsecurity.org/) configured.
> [Sun Feb 19 16:28:30.469773 2017] [:notice] [pid 1767] ModSecurity: APR
> compiled version="1.5.1"; loaded version="1.5.1"
> [Sun Feb 19 16:28:30.469792 2017] [:notice] [pid 1767] ModSecurity: PCRE
> compiled version="8.33 "; loaded version="8.39 2016-06-14"
> [Sun Feb 19 16:28:30.469798 2017] [:warn] [pid 1767] ModSecurity: Loaded
> PCRE do not match with compiled!
> [Sun Feb 19 16:28:30.469802 2017] [:notice] [pid 1767] ModSecurity: LUA
> compiled version="Lua 5.2"
> [Sun Feb 19 16:28:30.469807 2017] [:notice] [pid 1767] ModSecurity: LIBXML
> compiled version="2.9.1"
> [Sun Feb 19 16:28:30.469811 2017] [:notice] [pid 1767] ModSecurity: Status
> engine is currently disabled, enable it by set SecStatusEngine to On.
> [Sun Feb 19 16:28:30.495914 2017] [core:error] [pid 1767] (EAI 2)Name or
> service not known: AH00547: Could not resolve host name site1.local --
> ignoring!
> [Sun Feb 19 16:28:30.495979 2017] [core:error] [pid 1767] (EAI 2)Name or
> service not known: AH00547: Could not resolve host name site2.local --
> ignoring!
> [Sun Feb 19 16:28:30.496037 2017] [core:error] [pid 1767] (EAI 2)Name or
> service not known: AH00547: Could not resolve host name site3.local --
> ignoring!
> [Sun Feb 19 16:28:31.018255 2017] [mpm_prefork:notice] [pid 1767] AH00163:
> Apache/2.4.16 (Linux/SUSE) OpenSSL/1.0.1i-fips configured -- resuming normal
> operations
> [Sun Feb 19 16:28:31.018285 2017] [core:notice] [pid 1767] AH00094: Command
> line: '/usr/sbin/httpd-prefork -f /etc/apache2/httpd.conf -D SYSCONFIG -D
> SSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d/ -D
> SYSTEMD -D FOREGROUND'
> -------------------------------------
> 
> I ran 'rcapache2 restart' and in the same log I see:

Please use 'systemctl restart apache2' instead.

> 
> -------------------------------------
> [Sun Feb 19 16:33:38.465418 2017] [mpm_prefork:notice] [pid 1767] AH00170:
> caught SIGWINCH, shutting down gracefully
> [Sun Feb 19 16:33:39.000468 2017] [:notice] [pid 3473] ModSecurity for
> Apache/2.9.0 (http://www.modsecurity.org/) configured.
> [Sun Feb 19 16:33:39.000561 2017] [:notice] [pid 3473] ModSecurity: APR
> compiled version="1.5.1"; loaded version="1.5.1"
> [Sun Feb 19 16:33:39.000568 2017] [:notice] [pid 3473] ModSecurity: PCRE
> compiled version="8.33 "; loaded version="8.39 2016-06-14"
> [Sun Feb 19 16:33:39.000572 2017] [:warn] [pid 3473] ModSecurity: Loaded
> PCRE do not match with compiled!
> [Sun Feb 19 16:33:39.000575 2017] [:notice] [pid 3473] ModSecurity: LUA
> compiled version="Lua 5.2"
> [Sun Feb 19 16:33:39.000579 2017] [:notice] [pid 3473] ModSecurity: LIBXML
> compiled version="2.9.1"
> [Sun Feb 19 16:33:39.000582 2017] [:notice] [pid 3473] ModSecurity: Status
> engine is currently disabled, enable it by set SecStatusEngine to On.
> [Sun Feb 19 16:33:40.013885 2017] [mpm_prefork:notice] [pid 3473] AH00163:
> Apache/2.4.16 (Linux/SUSE) OpenSSL/1.0.1i-fips configured -- resuming normal
> operations
> [Sun Feb 19 16:33:40.013934 2017] [core:notice] [pid 3473] AH00094: Command
> line: '/usr/sbin/httpd-prefork -f /etc/apache2/httpd.conf -D SYSCONFIG -D
> SSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d/ -D
> SYSTEMD -D FOREGROUND'
> [Sun Feb 19 16:33:45.022714 2017] [core:notice] [pid 3473] AH00052: child
> pid 3485 exit signal Segmentation fault (11)
> [Sun Feb 19 16:33:47.024813 2017] [core:notice] [pid 3473] AH00052: child
> pid 3486 exit signal Segmentation fault (11)
> [Sun Feb 19 16:33:52.029721 2017] [core:notice] [pid 3473] AH00052: child
> pid 3481 exit signal Segmentation fault (11)
> [Sun Feb 19 16:34:16.054527 2017] [core:notice] [pid 3473] AH00052: child
> pid 3505 exit signal Segmentation fault (11)
> [Sun Feb 19 16:34:17.056118 2017] [core:notice] [pid 3473] AH00052: child
> pid 3501 exit signal Segmentation fault (11)
> [Sun Feb 19 16:34:17.056145 2017] [core:notice] [pid 3473] AH00052: child
> pid 3502 exit signal Segmentation fault (11)
> [Sun Feb 19 16:36:13.176170 2017] [core:notice] [pid 3473] AH00052: child
> pid 3508 exit signal Segmentation fault (11)
> [Sun Feb 19 16:36:28.191208 2017] [core:notice] [pid 3473] AH00052: child
> pid 3506 exit signal Segmentation fault (11)
> [Sun Feb 19 16:36:28.191241 2017] [core:notice] [pid 3473] AH00052: child
> pid 3507 exit signal Segmentation fault (11)
> [Sun Feb 19 16:36:38.201731 2017] [core:notice] [pid 3473] AH00052: child
> pid 3617 exit signal Segmentation fault (11)
> [Sun Feb 19 16:36:40.204770 2017] [core:notice] [pid 3473] AH00052: child
> pid 3620 exit signal Segmentation fault (11)
> [Sun Feb 19 16:36:41.207193 2017] [core:notice] [pid 3473] AH00052: child
> pid 3621 exit signal Segmentation fault (11)
> [Sun Feb 19 16:36:51.219528 2017] [core:notice] [pid 3473] AH00052: child
> pid 3555 exit signal Segmentation fault (11)
> [Sun Feb 19 16:36:54.222686 2017] [core:notice] [pid 3473] AH00052: child
> pid 3480 exit signal Segmentation fault (11)
> [Sun Feb 19 16:36:54.222723 2017] [core:notice] [pid 3473] AH00052: child
> pid 3489 exit signal Segmentation fault (11)
> [Sun Feb 19 16:37:13.241850 2017] [core:notice] [pid 3473] AH00052: child
> pid 3642 exit signal Segmentation fault (11)

If I were you, I would start with debugging the segmentation fault. There are
so many possibilities why the child could encounter it, so there is no proof it
is bug in httpd. Unfortunately I have no time to help you with that
intensively. 

You can inspire yourself how to obtain core dump with systemd coredumpctl:
https://github.com/pgajdos/apache-rex/tree/master/debug-coredump

> -------------------------------------
> 
> and after that everything works as expected.
> 
> All the HDD and SSD drives on the system are healthy. site1.local,
> site2.local and site3.local are properly configured in /etc/hosts and point
> to 127.0.0.1
> 
> When this happens I notice that Apache starts before wicked:
> 
> -------------------------------------
> [~]: systemd-analyze blame                                                  
> 
>           6.767s nut-driver.service                                         
> 
>           2.222s apache2.service                                            
> 
>           2.124s mysql.service                                              
> 
>           2.027s wicked.service                                             
> 
>           1.353s ModemManager.service                                       
> 
>           1.292s SuSEfirewall2_init.service                                 
> 
>            302ms php-fpm.service                                            
> 
>            275ms avahi-daemon.service                                       
> 
>            266ms libvirtd.service                                           
> 
>            242ms SuSEfirewall2.service 
> ...
> -------------------------------------

I am not expert on systemd, but I think that does not sort services by point of
time, where they begin to start.

> But this is strange considering that the service says the unit should start
> after network.target:
> 
> -------------------------------------
> cat /usr/lib/systemd/system/apache2*
> [Unit]
> Description=The Apache Webserver
> After=network.target nss-lookup.target time-sync.target remote-fs.target
> Before=getty@tty1.service plymouth-quit.service xdm.service
> 
> 
> [Service]
> Type=notify
> PrivateTmp=true
> ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start
> ExecReload=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful
> ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful-stop
> KillMode=mixed
> 
> 
> [Install]
> WantedBy=multi-user.target
> Alias=httpd.service apache.service
> 
> 
> [Unit]
> Description=The Apache Webserver %I
> After=network.target nss-lookup.target time-sync.target
> Before=getty@tty1.service plymouth-quit.service xdm.service
> 
> 
> [Service]
> Type=notify
> PrivateTmp=true
> Environment="HTTPD_INSTANCE=%i"
> ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start
> ExecReload=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful
> ExecStop=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k graceful-stop
> KillMode=mixed
> 
> 
> [Install]
> WantedBy=multi-user.target
> Alias=httpd@.service apache@.service
> -------------------------------------
> 
> BTW in /usr/lib/systemd/system I have 2 services: apache2.service and
> apache2@.service - is that normal and why are they 2?

Yes, see
/usr/share/doc/packages/apache2/README-instances.txt

You are receiving this mail because: