[opensuse-buildservice] PassengerAgent running amok on obs 2.6.6 (SLE12)
Hi, since the update to 2.6.5 or 2.6.6, we have the problem that requests stall and the PassengerAgent process uses 100% and more cpu time. Before that update, we were running 2.6.3 since quite some time without any issue. requests that came in are stuck and nothing reaches production log. here’s an strace: https://www.dropbox.com/s/egx5w370z077rg2/strace-passenger.out.bz2?dl=0 and this is how strace -c looks like: buildservice:/srv/www/obs/api/log # strace -cf -p 27893 Process 27893 attached with 34 threads Process 28584 attached Process 28585 attached Process 28607 attached Process 28608 attached Process 28613 attached Process 28614 attached Process 28664 attached Process 28665 attached Process 28686 attached Process 28687 attached 27893^CProcess 27893 detached Process 27895 detached Process 27896 detached Process 27898 detached Process 27899 detached Process 27900 detached Process 27901 detached Process 27903 detached Process 27904 detached Process 27949 detached Process 27958 detached Process 27964 detached Process 27972 detached Process 27982 detached Process 27989 detached Process 27996 detached Process 28003 detached Process 28010 detached Process 28017 detached Process 28024 detached Process 28031 detached Process 28038 detached Process 28045 detached Process 28052 detached Process 28059 detached Process 28066 detached Process 28073 detached Process 28080 detached Process 28087 detached Process 28094 detached Process 31631 detached Process 31632 detached Process 10797 detached Process 10798 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 47.75 0.110718 6 20025 read 26.44 0.061317 6132 10 wait4 24.15 0.056000 14000 4 nanosleep 1.59 0.003685 1 5269 write 0.04 0.000104 0 4560 15 close 0.02 0.000050 0 4155 20 open 0.00 0.000009 0 1325 stat 0.00 0.000000 0 180 fstat 0.00 0.000000 0 5 poll 0.00 0.000000 0 10 lseek 0.00 0.000000 0 295 mmap 0.00 0.000000 0 115 mprotect 0.00 0.000000 0 120 munmap 0.00 0.000000 0 15 brk 0.00 0.000000 0 250 5 rt_sigaction 0.00 0.000000 0 10 rt_sigprocmask 0.00 0.000000 0 10 rt_sigreturn 0.00 0.000000 0 20 20 ioctl 0.00 0.000000 0 15 15 access 0.00 0.000000 0 10 pipe 0.00 0.000000 0 5 dup2 0.00 0.000000 0 10 clone 0.00 0.000000 0 20 15 execve 0.00 0.000000 0 5 kill 0.00 0.000000 0 5 uname 0.00 0.000000 0 10 fcntl 0.00 0.000000 0 20 getdents 0.00 0.000000 0 5 getrlimit 0.00 0.000000 0 5 geteuid 0.00 0.000000 0 5 getpriority 0.00 0.000000 0 5 setpriority 0.00 0.000000 0 5 arch_prctl 0.00 0.000000 0 10 futex 0.00 0.000000 0 5 set_tid_address 0.00 0.000000 0 1 restart_syscall 0.00 0.000000 0 10 openat 0.00 0.000000 0 15 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 0.231883 36544 90 total any idea what could be the problem? Only way to solve it is to restart apache(+passenger). I already switched passenger into debug logging to see whether that is going to reveal anything more specific. -- mit freundlichen Gruessen/with best regards, Carsten Hoeger Open-Xchange GmbH -------------------------------------------------------------------------------- Open-Xchange AG, Rollnerstr. 14, 90408 Nürnberg, Amtsgericht Nürnberg HRB 24738 Vorstand: Rafael Laguna de la Vera, Carsten Dirks Aufsichtsratsvorsitzender: Richard Seibt European Office: Open-Xchange GmbH, Martinstr. 41, D-57462 Olpe, Germany Amtsgericht Siegen, HRB 8718, Geschäftsführer: Frank Hoberg, Martin Kauss US Office: Open-Xchange. Inc., 530 Lytton Avenue, Palo Alto, CA 94301, USA --------------------------------------------------------------------------------
Hi,
On 16 Oct 2015, at 12:41 , Carsten Höger
wrote: Hi,
since the update to 2.6.5 or 2.6.6, we have the problem that requests stall and the PassengerAgent process uses 100% and more cpu time.
Before that update, we were running 2.6.3 since quite some time without any issue.
[...] FYI: after we updated the passenger version to 5.0.20 from http://download.opensuse.org/repositories/OBS:/Server:/Unstable/SLE_12/x86_6..., as Adrian suggested, the problem seems to be gone. rubygem-passenger-5.0.20-5.1.x86_64 rubygem-passenger-apache2-5.0.20-5.1.x86_64 ruby2.1-rubygem-passenger-5.0.20-5.1.x86_64 http://download.opensuse.org/repositories/OBS:/Server:/2.6/SLE_12/x86_64/ contains version 5.0.7 which seem to be broken somehow. -- mit freundlichen Gruessen/with best regards, Carsten Hoeger Open-Xchange GmbH -------------------------------------------------------------------------------- Open-Xchange AG, Rollnerstr. 14, 90408 Nürnberg, Amtsgericht Nürnberg HRB 24738 Vorstand: Rafael Laguna de la Vera, Carsten Dirks Aufsichtsratsvorsitzender: Richard Seibt European Office: Open-Xchange GmbH, Martinstr. 41, D-57462 Olpe, Germany Amtsgericht Siegen, HRB 8718, Geschäftsführer: Frank Hoberg, Martin Kauss US Office: Open-Xchange. Inc., 530 Lytton Avenue, Palo Alto, CA 94301, USA --------------------------------------------------------------------------------
participants (1)
-
Carsten Höger