http://bugzilla.suse.com/show_bug.cgi?id=921577 --- Comment #2 from Michel Normand --- I am able to recreate a similar error in an opensuse guest: failure if guest started with "-smp 4,sockets=1,cores=1,threads=8" no error if guest started with "-smp 16,sockets=1,cores=2,threads=8" === libvirt started qemu command: /usr/bin/qemu-system-ppc64 -name opensuse_13_2_ppc64le -S -machine pseries-2.2,accel=kvm,usb=off -m 32768 -realtime mlock=off -smp 4,sockets=1,cores=1,threads=8 -uuid 4c973584-8ee9-410f-97bd-e03f5deff297 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/home/normand/.config/libvirt/qemu/lib/opensuse_13_2_ppc64le.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device pci-ohci,id=usb,bus=pci.0,addr=0x1 -device spapr-vscsi,id=scsi0,reg=0x2000 -drive file=/home/normand/images/opensuse_13_2_ppc64le.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive file=/home/normand/images/opensuse_13_2_ppc64le.disk2.qcow2,if=none,id=drive-scsi0-0-0-1,format=qcow2,cache=none -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1 -drive file=/home/normand/images/opensuse_13_2_ppc64le.disk3.qcow2,if=none,id=drive-scsi0-0-0-2,format=qcow2,cache=none -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi0-0-0-2,id=scsi0-0-0-2 -drive file=http://sf1.test.toulouse-stg.fr.ibm.com:80/pub/linux/opensuse/13/ppc64le/iso... -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=4,drive=drive-scsi0-0-0-4,id=scsi0-0-0-4,bootindex=2 -netdev tap,fd=24,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:89:84:34,bus=pci.0,addr=0x2 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,reg=0x30001000 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -msg timestamp=on === ... $cd open work/openSUSE:Factory:PowerPC/doxygen/ @opensuse_13_2_ppc64le:/home/michel/work/openSUSE:Factory:PowerPC/doxygen[michel@opensuse_13_2_ppc64le:~/work/openSUSE:Factory:PowerPC/doxygen] $osc build --clean ... [ 325s] Generating docs for compound B/home/abuild/rpmbuild/BUILD/doxygen-1.8.9.1/vhdlparser/TokenManager.h:19: warning: Unsupported xml/html tag <EOF> found [ 342s] /var/tmp/rpm-tmp.xsDZ1b: line 48: 4026 Segmentation fault ./bin/doxygen [ 342s] error: Bad exit status from /var/tmp/rpm-tmp.xsDZ1b (%build) [ 342s] [ 342s] [ 342s] RPM build errors: [ 342s] Bad exit status from /var/tmp/rpm-tmp.xsDZ1b (%build) ==== -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 Michel Normand changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mjwolf@us.ibm.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 --- Comment #3 from Michel Normand --- (In reply to Michel Normand from comment #2)

[CUT] ... $cd open work/openSUSE:Factory:PowerPC/doxygen/ @opensuse_13_2_ppc64le:/home/michel/work/openSUSE:Factory:PowerPC/ doxygen[michel@opensuse_13_2_ppc64le:~/work/openSUSE:Factory:PowerPC/doxygen] $osc build --clean [CUT] ...

note that unable to recreate the failure if manually executing ./bin/doxygen from inside chroot environment. But each time recreating the failure by executing the osc build command. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 --- Comment #5 from Michel Normand --- In my Power8 guest (started with -smp of comments #2) I am able to recreate the failure with following steps * the step1 + step2 is creating the problem * the step3 is to recreate the problem after step2. === step1: $osc co openSUSE:Factory:PowerPC doxygen $cd openSUSE:Factory:PowerPC/doxygen $quilt setup doxygen.spec $cd doxygen-1.8.9.1 $quilt push -a $cd ../ $mv doxygen-1.8.9.1 ~/ $cd ~/doxygen-1.8.9.1 $sed -i~ '/^TMAKE_CFLAGS_RELEASE.*/s@^.*@TMAKE_CFLAGS_RELEASE = -fmessage length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g @' tmake/lib/linuxg++/tmake.conf $ulimit -c unlimited === step2: $./configure --prefix /usr --install /usr/bin/install && make -j4 && ./bin/doxygen === === step3: $make clean && rm -rf doxygen_docs && rm -f doxygen.tag $./configure --prefix /usr --install /usr/bin/install && make -j4 && ./bin/doxygen === The generated core file and gdb gives the following backtrace: === $gdb -c ./core ./bin/doxygen ... Core was generated by `./bin/doxygen '. Program terminated with signal SIGABRT, Aborted. #0 0x00003fffa5acd194 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55 55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. Missing separate debuginfos, use: zypper install libgcc_s1-debuginfo-4.8.3+r218481-2.1.ppc64le libstdc++6-debuginfo-4.8.3+r218481-2.1.ppc64le (gdb) bt #0 0x00003fffa5acd194 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x00003fffa5acf184 in __GI_abort () at abort.c:78 #2 0x00003fffa5b136c4 in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:175 #3 0x00003fffa5b1ba84 in malloc_printerr (action=<optimized out>, str=0x3fffa5c06b50 "double free or corruption (fasttop)", ptr=<optimized out>) at malloc.c:4960 #4 0x00003fffa5b1cadc in _int_free (av=<optimized out>, p=<optimized out>, have_lock=<optimized out>) at malloc.c:3831 #5 0x00003fffa5dece10 in operator delete(void*) () from /usr/lib64/libstdc++.so.6 #6 0x00000000106620e4 in QGList::takeFirst (this=<optimized out>) at qglist.cpp:628 #7 0x000000001053ba84 in dequeue (this=<optimized out>) at ../qtools/qqueue.h:59 #8 DotRunnerQueue::dequeue (this=0x1001910fcc0) at dot.cpp:1170 #9 0x000000001053bb18 in DotWorkerThread::run (this=0x10019112a50) at dot.cpp:1191 #10 0x00000000106a0a44 in QThreadPrivate::start (arg=0x10019112a50) at qthread_unix.cpp:87 #11 0x00003fffa5ee9454 in start_thread (arg=0x3fffa38bf180) at pthread_create.c:335 #12 0x00003fffa5b9e0c4 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96 === -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 Alexey Kardashevskiy changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aik@au1.ibm.com --- Comment #6 from Alexey Kardashevskiy --- I could not figure out where you got the sources from so I tried the latest doxygen git which is 1.8.9.1 as well. The reproduce steps did not work for me either - there is no "tmake/lib/linuxg++/tmake.conf" in doxygen's git (missing a minus in the name); "-fmessage length=0" needs to be "-fmessage-length=0". With the commands below I get a linker error. What do I miss? === sed -i~ '/^TMAKE_CFLAGS_RELEASE.*/s@^.*@TMAKE_CFLAGS_RELEASE = -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g @' tmake/lib/linux-g++/tmake.conf make clean && rm -rf doxygen_docs && rm -f doxygen.tag ./configure --prefix /usr --install /usr/bin/install && make -j4 && ./bin/doxygen === g++ -o ../bin/doxygen ../objects/doxygen/main.o -L../lib -ldoxygen -lvhdlparser -ldoxycfg -lqtools -lmd5 -lpthread ../lib/libdoxycfg.a(config.o): In function `Config::create()': /home/aik/doxygen/src/config.l:1702: undefined reference to `addConfigOptions(Config*)' /home/aik/doxygen/src/config.l:1702: undefined reference to `addConfigOptions(Config*)' /home/aik/doxygen/src/config.l:1702: undefined reference to `addConfigOptions(Config*)' collect2: error: ld returned 1 exit status Makefile.doxygen:57: recipe for target '../bin/doxygen' failed gmake[2]: *** [../bin/doxygen] Error 1 gmake[2]: Leaving directory '/home/aik/doxygen/src' Makefile:41: recipe for target 'all' failed gmake[1]: *** [all] Error 2 gmake[1]: Leaving directory '/home/aik/doxygen/src' Makefile:24: recipe for target 'all' failed make: *** [all] Error 2 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 --- Comment #8 from Michel Normand --- Created attachment 627590 --> http://bugzilla.suse.com/attachment.cgi?id=627590&action=edit valgrind output of doxygen the attached valgrind output is reporting a set of lost memory blocks by doxygen that executed to its end without failure. Another confirmation that previous reported failure is timing dependant. I do not know how and if lost memory blocks could be related to failure (when not executed with valgrind) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 Mike Wolf changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bergner@vnet.ibm.com --- Comment #10 from Mike Wolf --- Nish, We think that there is a race in locking in the kernel. Could you or someone from your team take a look at this -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 --- Comment #12 from Michel Normand --- I did not found the source of the problem, but as previously done in comment #7 I retrieved the doxygen source code from git tree, and found that current master head do not failed anymore: * with commit id 796d6585be58d1c9112422a919f49d1998dc672c I still have the double mfree reported error * with commit id 7a0c06d1745739cb7f9753e75cb46f4a431b0eaa I do not have any failure. But there is not yet available tagged release with this commit id may be we could ask for one to upstream maintainer if we are ready to use it for opensuse. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c14 --- Comment #14 from Michel Normand --- Created attachment 644387 --> http://bugzilla.suse.com/attachment.cgi?id=644387&action=edit doxygen_1810_twppc64le_cpu_1_threads_8_failure.log (In reply to Petr Gajdos from comment #13)

Sorry for the delay.

I have submitted doxygen 1.8.10 into Factory. Michel, please confirm that it resolve the issue.

Too bad the new version doxygen 1.8.10 do not solve this problem as reported by attachment (doxygen_1810_twppc64le_cpu_1_threads_8_failure.log) The reported failure is now an invalid pointer but still when a glibc function is called by QGlist::takeFirst So very similar to initial call stack. === *** Error in `./build/bin/doxygen': free(): invalid pointer: 0x000001001610d980 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x84c44)[0x3fff9ae74c44] /lib64/libc.so.6(+0x8d298)[0x3fff9ae7d298] /lib64/libc.so.6(cfree+0xd0)[0x3fff9ae822f0] /usr/lib64/libstdc++.so.6(_ZdlPv+0x18)[0x3fff9b16e238] ./build/bin/doxygen(_ZN6QGList9takeFirstEv+0x114)[0x1087d804] ./build/bin/doxygen(_ZN15DotWorkerThread3runEv+0x84)[0x107aba44] ./build/bin/doxygen(_ZN14QThreadPrivate5startEPv+0x144)[0x108cfc04] /lib64/libpthread.so.0(+0x7fb8)[0x3fff9b2c7fb8] /lib64/libc.so.6(clone+0xe4)[0x3fff9af003b4] === -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 Michel Normand changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(normand@linux.vne | |t.ibm.com) | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c17 Petr Gajdos changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(normand@linux.vne | |t.ibm.com) --- Comment #17 from Petr Gajdos --- I don't mind to turn off multithreading globally (~ change default value of said variable) for ppc64le as a workaround. See sr#336660 if this is what you want and if it works for you. It does work for me, see home:pgajdos:doxygen-ppc64le. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 Michel Normand changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(normand@linux.vne | |t.ibm.com) | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c20 --- Comment #20 from Petr Gajdos --- (And the patch is not included in doxygen-doc.spec, so even if it wouldn't be specified, DOT_NUM_THREAD would be 0 by default, as the doxygen-doc is not using /usr/bin/doxygen from doxygen package.) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c22 Michel Normand changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(normand@linux.vne | |t.ibm.com) | --- Comment #22 from Michel Normand --- Created attachment 650515 --> http://bugzilla.suse.com/attachment.cgi?id=650515&action=edit doxygen_ppc64le_ignore_DOT_NUM_THREADS_to_0.patch in reply to Comment #21 the suggested change in home:pgajdos:doxygen-ppc64le/doxygen is working specifically for doxygen-doc package. But I would suggest the attached doxygen_ppc64le_ignore_DOT_NUM_THREADS_to_0.patch as replacement to be able to have the bypass for all packages using doxygen without the need to change their configuration file. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c24 Michel Normand changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(normand@linux.vne | |t.ibm.com) | --- Comment #24 from Michel Normand --- (In reply to Petr Gajdos from comment #23)

So the double free/corruption doesn't happen, say, for DOT_NUM_THREADS=10?

it would failed (because DOT_NUM_THREADS not 0 or 1) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c26 --- Comment #26 from Michel Normand --- (In reply to Petr Gajdos from comment #25)

So wouldn't be better to just hardcode numThreads to 1? That is

- int numThreads = QMIN(32,Config_getInt("DOT_NUM_THREADS")); + int numThreads = 1;

by the patch?

I don't think so. The advantage of the doxygen_ppc64le_ignore_DOT_NUM_THREADS_to_0.patch (1) is that it is sufficient as a bypass for the packages build in OBS (that have config with DOT_NUM_THREADS=0) But we continue to support DOT_NUM_THREADS when using the doxygen command manually for test purpose. (1) https://bugzilla.suse.com/attachment.cgi?id=650515 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 Petr Gajdos changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P4 - Low |P3 - Medium -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c30 --- Comment #30 from Petr Gajdos --- Thank you too. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c32 Michel Normand changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED |--- Assignee|pgajdos@suse.com |normand@linux.vnet.ibm.com --- Comment #32 from Michel Normand --- I am re-opening this bug and assigned it to me. Current status: * the provided patch of comment #29 is only a bypass for ppc64le. * still able to have reported failure if DOT_NUM_THREADS set to value different from 0 or 1. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c34 --- Comment #34 from Michel Normand --- Problem initially reported for ppc64le, is reported now for ppc64 with doxygen version 1.8.11. At least failure for OBS ppc64 build of two packages that are using doxygen; gwenhywfar, vdr So apply the same patch in spec. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=921577 Michel Normand changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(normand@linux.vne | |t.ibm.com) | -- You are receiving this mail because: You are on the CC list for the bug.