http://bugzilla.suse.com/show_bug.cgi?id=921577

--- Comment #2 from Michel Normand normand@linux.vnet.ibm.com --- I am able to recreate a similar error in an opensuse guest: failure if guest started with "-smp 4,sockets=1,cores=1,threads=8" no error if guest started with "-smp 16,sockets=1,cores=2,threads=8"

=== libvirt started qemu command: /usr/bin/qemu-system-ppc64 -name opensuse_13_2_ppc64le -S -machine pseries-2.2,accel=kvm,usb=off -m 32768 -realtime mlock=off -smp 4,sockets=1,cores=1,threads=8 -uuid 4c973584-8ee9-410f-97bd-e03f5deff297 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/home/normand/.config/libvirt/qemu/lib/opensuse_13_2_ppc64le.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device pci-ohci,id=usb,bus=pci.0,addr=0x1 -device spapr-vscsi,id=scsi0,reg=0x2000 -drive file=/home/normand/images/opensuse_13_2_ppc64le.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive file=/home/normand/images/opensuse_13_2_ppc64le.disk2.qcow2,if=none,id=drive-scsi0-0-0-1,format=qcow2,cache=none -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1 -drive file=/home/normand/images/opensuse_13_2_ppc64le.disk3.qcow2,if=none,id=drive-scsi0-0-0-2,format=qcow2,cache=none -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi0-0-0-2,id=scsi0-0-0-2 -drive file=http://sf1.test.toulouse-stg.fr.ibm.com:80/pub/linux/opensuse/13/ppc64le/iso... -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=4,drive=drive-scsi0-0-0-4,id=scsi0-0-0-4,bootindex=2 -netdev tap,fd=24,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:89:84:34,bus=pci.0,addr=0x2 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,reg=0x30001000 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -msg timestamp=on === ... $cd open work/openSUSE:Factory:PowerPC/doxygen/ @opensuse_13_2_ppc64le:/home/michel/work/openSUSE:Factory:PowerPC/doxygen[michel@opensuse_13_2_ppc64le:~/work/openSUSE:Factory:PowerPC/doxygen] $osc build --clean ... [ 325s] Generating docs for compound B/home/abuild/rpmbuild/BUILD/doxygen-1.8.9.1/vhdlparser/TokenManager.h:19: warning: Unsupported xml/html tag <EOF> found [ 342s] /var/tmp/rpm-tmp.xsDZ1b: line 48: 4026 Segmentation fault ./bin/doxygen [ 342s] error: Bad exit status from /var/tmp/rpm-tmp.xsDZ1b (%build) [ 342s] [ 342s] [ 342s] RPM build errors: [ 342s] Bad exit status from /var/tmp/rpm-tmp.xsDZ1b (%build) ====

http://bugzilla.suse.com/show_bug.cgi?id=921577

Michel Normand normand@linux.vnet.ibm.com changed:

What |Removed |Added ---------------------------------------------------------------------------- CC| |mjwolf@us.ibm.com

http://bugzilla.suse.com/show_bug.cgi?id=921577

--- Comment #3 from Michel Normand normand@linux.vnet.ibm.com --- (In reply to Michel Normand from comment #2)

[CUT] ... $cd open work/openSUSE:Factory:PowerPC/doxygen/ @opensuse_13_2_ppc64le:/home/michel/work/openSUSE:Factory:PowerPC/ doxygen[michel@opensuse_13_2_ppc64le:~/work/openSUSE:Factory:PowerPC/doxygen] $osc build --clean [CUT] ...

note that unable to recreate the failure if manually executing ./bin/doxygen from inside chroot environment. But each time recreating the failure by executing the osc build command.

http://bugzilla.suse.com/show_bug.cgi?id=921577

--- Comment #5 from Michel Normand normand@linux.vnet.ibm.com --- In my Power8 guest (started with -smp of comments #2) I am able to recreate the failure with following steps

* the step1 + step2 is creating the problem * the step3 is to recreate the problem after step2.

=== step1: $osc co openSUSE:Factory:PowerPC doxygen $cd openSUSE:Factory:PowerPC/doxygen $quilt setup doxygen.spec $cd doxygen-1.8.9.1 $quilt push -a $cd ../ $mv doxygen-1.8.9.1 ~/ $cd ~/doxygen-1.8.9.1 $sed -i~ '/^TMAKE_CFLAGS_RELEASE.*/s@^.*@TMAKE_CFLAGS_RELEASE = -fmessage length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g @' tmake/lib/linuxg++/tmake.conf $ulimit -c unlimited === step2: $./configure --prefix /usr --install /usr/bin/install && make -j4 && ./bin/doxygen === === step3: $make clean && rm -rf doxygen_docs && rm -f doxygen.tag $./configure --prefix /usr --install /usr/bin/install && make -j4 && ./bin/doxygen ===

The generated core file and gdb gives the following backtrace: === $gdb -c ./core ./bin/doxygen ... Core was generated by `./bin/doxygen '. Program terminated with signal SIGABRT, Aborted. #0 0x00003fffa5acd194 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55 55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. Missing separate debuginfos, use: zypper install libgcc_s1-debuginfo-4.8.3+r218481-2.1.ppc64le libstdc++6-debuginfo-4.8.3+r218481-2.1.ppc64le (gdb) bt #0 0x00003fffa5acd194 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x00003fffa5acf184 in __GI_abort () at abort.c:78 #2 0x00003fffa5b136c4 in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:175 #3 0x00003fffa5b1ba84 in malloc_printerr (action=<optimized out>, str=0x3fffa5c06b50 "double free or corruption (fasttop)", ptr=<optimized out>) at malloc.c:4960 #4 0x00003fffa5b1cadc in _int_free (av=<optimized out>, p=<optimized out>, have_lock=<optimized out>) at malloc.c:3831 #5 0x00003fffa5dece10 in operator delete(void*) () from /usr/lib64/libstdc++.so.6 #6 0x00000000106620e4 in QGList::takeFirst (this=<optimized out>) at qglist.cpp:628 #7 0x000000001053ba84 in dequeue (this=<optimized out>) at ../qtools/qqueue.h:59 #8 DotRunnerQueue::dequeue (this=0x1001910fcc0) at dot.cpp:1170 #9 0x000000001053bb18 in DotWorkerThread::run (this=0x10019112a50) at dot.cpp:1191 #10 0x00000000106a0a44 in QThreadPrivate::start (arg=0x10019112a50) at qthread_unix.cpp:87 #11 0x00003fffa5ee9454 in start_thread (arg=0x3fffa38bf180) at pthread_create.c:335 #12 0x00003fffa5b9e0c4 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96 ===

http://bugzilla.suse.com/show_bug.cgi?id=921577

Alexey Kardashevskiy aik@au1.ibm.com changed:

What |Removed |Added ---------------------------------------------------------------------------- CC| |aik@au1.ibm.com

--- Comment #6 from Alexey Kardashevskiy aik@au1.ibm.com --- I could not figure out where you got the sources from so I tried the latest doxygen git which is 1.8.9.1 as well. The reproduce steps did not work for me either - there is no "tmake/lib/linuxg++/tmake.conf" in doxygen's git (missing a minus in the name); "-fmessage length=0" needs to be "-fmessage-length=0".

With the commands below I get a linker error. What do I miss?

=== sed -i~ '/^TMAKE_CFLAGS_RELEASE.*/s@^.*@TMAKE_CFLAGS_RELEASE = -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g @' tmake/lib/linux-g++/tmake.conf

make clean && rm -rf doxygen_docs && rm -f doxygen.tag

./configure --prefix /usr --install /usr/bin/install && make -j4 && ./bin/doxygen ===

g++ -o ../bin/doxygen ../objects/doxygen/main.o -L../lib -ldoxygen -lvhdlparser -ldoxycfg -lqtools -lmd5 -lpthread ../lib/libdoxycfg.a(config.o): In function `Config::create()': /home/aik/doxygen/src/config.l:1702: undefined reference to `addConfigOptions(Config*)' /home/aik/doxygen/src/config.l:1702: undefined reference to `addConfigOptions(Config*)' /home/aik/doxygen/src/config.l:1702: undefined reference to `addConfigOptions(Config*)' collect2: error: ld returned 1 exit status Makefile.doxygen:57: recipe for target '../bin/doxygen' failed gmake[2]: *** [../bin/doxygen] Error 1 gmake[2]: Leaving directory '/home/aik/doxygen/src' Makefile:41: recipe for target 'all' failed gmake[1]: *** [all] Error 2 gmake[1]: Leaving directory '/home/aik/doxygen/src' Makefile:24: recipe for target 'all' failed make: *** [all] Error 2

http://bugzilla.suse.com/show_bug.cgi?id=921577

--- Comment #8 from Michel Normand normand@linux.vnet.ibm.com --- Created attachment 627590 --> http://bugzilla.suse.com/attachment.cgi?id=627590&action=edit valgrind output of doxygen

the attached valgrind output is reporting a set of lost memory blocks by doxygen that executed to its end without failure. Another confirmation that previous reported failure is timing dependant.

I do not know how and if lost memory blocks could be related to failure (when not executed with valgrind)

http://bugzilla.suse.com/show_bug.cgi?id=921577

Mike Wolf mjw@us.ibm.com changed:

What |Removed |Added ---------------------------------------------------------------------------- CC| |bergner@vnet.ibm.com

--- Comment #10 from Mike Wolf mjw@us.ibm.com --- Nish, We think that there is a race in locking in the kernel. Could you or someone from your team take a look at this

http://bugzilla.suse.com/show_bug.cgi?id=921577

--- Comment #12 from Michel Normand normand@linux.vnet.ibm.com --- I did not found the source of the problem, but as previously done in comment #7 I retrieved the doxygen source code from git tree, and found that current master head do not failed anymore:

* with commit id 796d6585be58d1c9112422a919f49d1998dc672c I still have the double mfree reported error * with commit id 7a0c06d1745739cb7f9753e75cb46f4a431b0eaa I do not have any failure. But there is not yet available tagged release with this commit id may be we could ask for one to upstream maintainer if we are ready to use it for opensuse.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c14

--- Comment #14 from Michel Normand normand@linux.vnet.ibm.com --- Created attachment 644387 --> http://bugzilla.suse.com/attachment.cgi?id=644387&action=edit doxygen_1810_twppc64le_cpu_1_threads_8_failure.log

(In reply to Petr Gajdos from comment #13)

Sorry for the delay.

I have submitted doxygen 1.8.10 into Factory. Michel, please confirm that it resolve the issue.

Too bad the new version doxygen 1.8.10 do not solve this problem as reported by attachment (doxygen_1810_twppc64le_cpu_1_threads_8_failure.log) The reported failure is now an invalid pointer but still when a glibc function is called by QGlist::takeFirst So very similar to initial call stack. ===

*** Error in `./build/bin/doxygen': free(): invalid pointer: 0x000001001610d980 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x84c44)[0x3fff9ae74c44] /lib64/libc.so.6(+0x8d298)[0x3fff9ae7d298] /lib64/libc.so.6(cfree+0xd0)[0x3fff9ae822f0] /usr/lib64/libstdc++.so.6(_ZdlPv+0x18)[0x3fff9b16e238] ./build/bin/doxygen(_ZN6QGList9takeFirstEv+0x114)[0x1087d804] ./build/bin/doxygen(_ZN15DotWorkerThread3runEv+0x84)[0x107aba44] ./build/bin/doxygen(_ZN14QThreadPrivate5startEPv+0x144)[0x108cfc04] /lib64/libpthread.so.0(+0x7fb8)[0x3fff9b2c7fb8] /lib64/libc.so.6(clone+0xe4)[0x3fff9af003b4] ===

http://bugzilla.suse.com/show_bug.cgi?id=921577

Michel Normand normand@linux.vnet.ibm.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(normand@linux.vne | |t.ibm.com) |

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c17

Petr Gajdos pgajdos@suse.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(normand@linux.vne | |t.ibm.com)

--- Comment #17 from Petr Gajdos pgajdos@suse.com --- I don't mind to turn off multithreading globally (~ change default value of said variable) for ppc64le as a workaround.

See sr#336660 if this is what you want and if it works for you.

It does work for me, see home:pgajdos:doxygen-ppc64le.

http://bugzilla.suse.com/show_bug.cgi?id=921577

Michel Normand normand@linux.vnet.ibm.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(normand@linux.vne | |t.ibm.com) |

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c20

--- Comment #20 from Petr Gajdos pgajdos@suse.com --- (And the patch is not included in doxygen-doc.spec, so even if it wouldn't be specified, DOT_NUM_THREAD would be 0 by default, as the doxygen-doc is not using /usr/bin/doxygen from doxygen package.)

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c22

Michel Normand normand@linux.vnet.ibm.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(normand@linux.vne | |t.ibm.com) |

--- Comment #22 from Michel Normand normand@linux.vnet.ibm.com --- Created attachment 650515 --> http://bugzilla.suse.com/attachment.cgi?id=650515&action=edit doxygen_ppc64le_ignore_DOT_NUM_THREADS_to_0.patch

in reply to Comment #21 the suggested change in home:pgajdos:doxygen-ppc64le/doxygen is working specifically for doxygen-doc package.

But I would suggest the attached doxygen_ppc64le_ignore_DOT_NUM_THREADS_to_0.patch as replacement to be able to have the bypass for all packages using doxygen without the need to change their configuration file.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c24

Michel Normand normand@linux.vnet.ibm.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(normand@linux.vne | |t.ibm.com) |

--- Comment #24 from Michel Normand normand@linux.vnet.ibm.com --- (In reply to Petr Gajdos from comment #23)

So the double free/corruption doesn't happen, say, for DOT_NUM_THREADS=10?

it would failed (because DOT_NUM_THREADS not 0 or 1)

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c26

--- Comment #26 from Michel Normand normand@linux.vnet.ibm.com --- (In reply to Petr Gajdos from comment #25)

So wouldn't be better to just hardcode numThreads to 1? That is

• int numThreads = QMIN(32,Config_getInt("DOT_NUM_THREADS"));

• int numThreads = 1;

by the patch?

I don't think so. The advantage of the doxygen_ppc64le_ignore_DOT_NUM_THREADS_to_0.patch (1) is that it is sufficient as a bypass for the packages build in OBS (that have config with DOT_NUM_THREADS=0) But we continue to support DOT_NUM_THREADS when using the doxygen command manually for test purpose.

(1) https://bugzilla.suse.com/attachment.cgi?id=650515

http://bugzilla.suse.com/show_bug.cgi?id=921577

Petr Gajdos pgajdos@suse.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Priority|P4 - Low |P3 - Medium

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c30

--- Comment #30 from Petr Gajdos pgajdos@suse.com --- Thank you too.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c32

Michel Normand normand@linux.vnet.ibm.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED |--- Assignee|pgajdos@suse.com |normand@linux.vnet.ibm.com

--- Comment #32 from Michel Normand normand@linux.vnet.ibm.com --- I am re-opening this bug and assigned it to me. Current status: * the provided patch of comment #29 is only a bypass for ppc64le. * still able to have reported failure if DOT_NUM_THREADS set to value different from 0 or 1.

http://bugzilla.suse.com/show_bug.cgi?id=921577 http://bugzilla.suse.com/show_bug.cgi?id=921577#c34

--- Comment #34 from Michel Normand normand@linux.vnet.ibm.com --- Problem initially reported for ppc64le, is reported now for ppc64 with doxygen version 1.8.11. At least failure for OBS ppc64 build of two packages that are using doxygen; gwenhywfar, vdr So apply the same patch in spec.

https://bugzilla.suse.com/show_bug.cgi?id=921577

Michel Normand normand@linux.vnet.ibm.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(normand@linux.vne | |t.ibm.com) |