[opensuse] Motif application crashes with 12.1 (glibc problem?)
Hi, I have to use a motif application that relies on openmotif22-libs. I have only a binary of the application. The problem is that the application crashes on startup on opensuse 12.1. It works on 11.2, 11.3 and 11.4. It also worked on older opensuse and redhat versions. The debugger gives me the following traceback: #0 0x0000000000000000 in ?? () #1 0x00007fb32d0b6ec8 in _IO_vfprintf_internal at vfprintf.c:1739 #2 0x00007fb32d15edb7 in ___vsprintf_chk at vsprintf_chk.c:87 #3 0x00007fb32d15ecfd in ___sprintf_chk at sprintf_chk.c:33 #4 0x00007fb32cb79d7d in sprintf at /usr/include/bits/stdio2.h:34 #5 _XmOSInitPath at Xmos.c:1271 #6 0x00007fb32cba9cce in XmGetIconFileName at IconFile.c:614 #7 0x00007fb32caca05f in GetImage at ImageCache.c:1310 I've checked the call to sprintf in motif and I am sure that it is ok. Motif also did not change that much from 11.4 to 12.1. Therefore I assume that a change in glibc causes the problem. Has anyone an idea what can cause the problem? Was there any suspicious change in glibc? Christoph -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 24/11/11 19:14, Christoph Bartoschek wrote: . Therefore I
assume that a change in glibc causes the problem.
That's unlikely, what is more probable is that either motif or your application has a buffer overflow somewhere. if you can reproduce with other motif apps which sources are available open a bug report. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 24/11/11 19:14, Christoph Bartoschek wrote:
Hi,
I have to use a motif application that relies on openmotif22-libs. I have only a binary of the application. The problem is that the application crashes on startup on opensuse 12.1. It works on 11.2, 11.3 and 11.4. It also worked on older opensuse and redhat versions.
The debugger gives me the following traceback:
#0 0x0000000000000000 in ?? () #1 0x00007fb32d0b6ec8 in _IO_vfprintf_internal at vfprintf.c:1739 #2 0x00007fb32d15edb7 in ___vsprintf_chk at vsprintf_chk.c:87 #3 0x00007fb32d15ecfd in ___sprintf_chk at sprintf_chk.c:33 #4 0x00007fb32cb79d7d in sprintf at /usr/include/bits/stdio2.h:34 #5 _XmOSInitPath at Xmos.c:1271 #6 0x00007fb32cba9cce in XmGetIconFileName at IconFile.c:614 #7 0x00007fb32caca05f in GetImage at ImageCache.c:1310
can you provide the full backtrace ? after installing the relevant debuginfo packages ? bt full -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Am 25.11.2011 04:04, schrieb Cristian Rodríguez:
On 24/11/11 19:14, Christoph Bartoschek wrote:
Hi,
I have to use a motif application that relies on openmotif22-libs. I have only a binary of the application. The problem is that the application crashes on startup on opensuse 12.1. It works on 11.2, 11.3 and 11.4. It also worked on older opensuse and redhat versions.
The debugger gives me the following traceback:
#0 0x0000000000000000 in ?? () #1 0x00007fb32d0b6ec8 in _IO_vfprintf_internal at vfprintf.c:1739 #2 0x00007fb32d15edb7 in ___vsprintf_chk at vsprintf_chk.c:87 #3 0x00007fb32d15ecfd in ___sprintf_chk at sprintf_chk.c:33 #4 0x00007fb32cb79d7d in sprintf at /usr/include/bits/stdio2.h:34 #5 _XmOSInitPath at Xmos.c:1271 #6 0x00007fb32cba9cce in XmGetIconFileName at IconFile.c:614 #7 0x00007fb32caca05f in GetImage at ImageCache.c:1310
can you provide the full backtrace ? after installing the relevant debuginfo packages ?
bt full
I have attached the backtrace. Frame #0 is at address 0x0. In frame #1 a jump table is used and the index obviously point to a null pointer. I've started valgrind to check whether there are heap overflows. I expect a result in one hour. Christoph
Am 25.11.2011 09:39, schrieb Christoph Bartoschek:
I have attached the backtrace. Frame #0 is at address 0x0. In frame #1 a jump table is used and the index obviously point to a null pointer.
I've started valgrind to check whether there are heap overflows. I expect a result in one hour.
Valgrind reports no errors that can lead to heap overflows. So there might only be a buffer overflow on the stack. I suspect that this is related to glibc because with opensuse 11.4 some applications stopped to work because of memcpy. Maybe there is a similar change that causes my crashes. I do not say that glibc is broken. Maybe it is now stricter than in earlier versions. Christoph -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 11/25/2011 10:19 AM, Christoph Bartoschek wrote:
Am 25.11.2011 09:39, schrieb Christoph Bartoschek:
I have attached the backtrace. Frame #0 is at address 0x0. In frame #1 a jump table is used and the index obviously point to a null pointer.
I've started valgrind to check whether there are heap overflows. I expect a result in one hour.
Valgrind reports no errors that can lead to heap overflows. So there might only be a buffer overflow on the stack.
I suspect that this is related to glibc because with opensuse 11.4 some applications stopped to work because of memcpy. Maybe there is a similar change that causes my crashes.
I do not say that glibc is broken. Maybe it is now stricter than in earlier versions.
If you have a self-contained testcase - best without involving glibc - the glibc maintainer (that's me;) will look at a bugreport via bugzilla.novell.com and tell you what's broken... right now with the information you've given, it's not clear what the problem is. Btw. the memcpy change was not in 11.4, we disabled it, Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Am 25.11.2011 10:22, schrieb Andreas Jaeger:
On 11/25/2011 10:19 AM, Christoph Bartoschek wrote:
Am 25.11.2011 09:39, schrieb Christoph Bartoschek:
I have attached the backtrace. Frame #0 is at address 0x0. In frame #1 a jump table is used and the index obviously point to a null pointer.
I've started valgrind to check whether there are heap overflows. I expect a result in one hour.
Valgrind reports no errors that can lead to heap overflows. So there might only be a buffer overflow on the stack.
I suspect that this is related to glibc because with opensuse 11.4 some applications stopped to work because of memcpy. Maybe there is a similar change that causes my crashes.
I do not say that glibc is broken. Maybe it is now stricter than in earlier versions.
If you have a self-contained testcase - best without involving glibc - the glibc maintainer (that's me;) will look at a bugreport via bugzilla.novell.com and tell you what's broken...
I try to create one. But this could take a week or more because I do not have the source code of the application. And just simulating the sprintf call does not lead to a crash.
right now with the information you've given, it's not clear what the problem is.
Btw. the memcpy change was not in 11.4, we disabled it,
I am sure that it was in the first release of 11.4 because I had an application that crashed only in 11.4. After changing memcpy to memmove at a single appropriate place it did not crash anymore. Christoph -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
I try to create one. But this could take a week or more because I do not have the source code of the application. And just simulating the sprintf call does not lead to a crash.
could be an issue with gcc -version and glibc -version, Could you try to upgrade your glibc version.
I am sure that it was in the first release of 11.4 because I had an application that crashed only in 11.4. After changing memcpy to memmove at a single appropriate place it did not crash anymore.
could you try the above and confirm. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Am 25.11.2011 11:07, schrieb Sujit Karatparambil:
I try to create one. But this could take a week or more because I do not have the source code of the application. And just simulating the sprintf call does not lead to a crash.
could be an issue with gcc -version and glibc -version, Could you try to upgrade your glibc version.
I am sure that it was in the first release of 11.4 because I had an application that crashed only in 11.4. After changing memcpy to memmove at a single appropriate place it did not crash anymore.
could you try the above and confirm.
I'm sorry. I do not understand what you suggest. I should compile the test with an older compiler? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
I'm sorry. I do not understand what you suggest. I should compile the test with an older compiler?
I am not asking you to compile the test, as you need to come up with testcase. I think the problem might be because of compiler optimizations when compiling the glibc version from 11.4 when you have reported some changes memcpy to memmove. You could take the current glibc version source and compile it with [1][2]. Thanks, Sujit K M 1. http://en.opensuse.org/Product_highlights (Look for gcc.) 2. http://en.wikipedia.org/wiki/Link-time_optimization -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, 25 Nov 2011 09:39:36 +0100
Christoph Bartoschek
Am 25.11.2011 04:04, schrieb Cristian Rodríguez:
On 24/11/11 19:14, Christoph Bartoschek wrote:
Hi,
I have to use a motif application that relies on openmotif22-libs. I have only a binary of the application. The problem is that the application crashes on startup on opensuse 12.1. It works on 11.2, 11.3 and 11.4. It also worked on older opensuse and redhat versions.
The debugger gives me the following traceback:
#0 0x0000000000000000 in ?? () #1 0x00007fb32d0b6ec8 in _IO_vfprintf_internal at vfprintf.c:1739 #2 0x00007fb32d15edb7 in ___vsprintf_chk at vsprintf_chk.c:87 #3 0x00007fb32d15ecfd in ___sprintf_chk at sprintf_chk.c:33 #4 0x00007fb32cb79d7d in sprintf at /usr/include/bits/stdio2.h:34 #5 _XmOSInitPath at Xmos.c:1271 #6 0x00007fb32cba9cce in XmGetIconFileName at IconFile.c:614 #7 0x00007fb32caca05f in GetImage at ImageCache.c:1310
can you provide the full backtrace ? after installing the relevant debuginfo packages ?
bt full
I have attached the backtrace. Frame #0 is at address 0x0. In frame #1 a jump table is used and the index obviously point to a null pointer.
I've started valgrind to check whether there are heap overflows. I expect a result in one hour.
Christoph
The program below is similar to the code calling sprintf in motif.
There's something going wrong because in the crash condition, in vfprintf,
it's processing the format string in 'the slow mode' but when the test case
runs, it never executes that code.
When the app crashes, can you check that the format string in vfprintf
frame #1, 'lead_str_end' is the same as specified below ?
There should be 18 %s formats.
#include
The program below is similar to the code calling sprintf in motif. There's something going wrong because in the crash condition, in vfprintf, it's processing the format string in 'the slow mode' but when the test case runs, it never executes that code.
When the app crashes, can you check that the format string in vfprintf frame #1, 'lead_str_end' is the same as specified below ? There should be 18 %s formats.
Thanks for your thorough analysis. What do you mean by slow mode? How can it be entered? (gdb) set print elements 700 (gdb) p lead_str_end $3 = (const unsigned char *) 0x7fb36b732500 "%%P%%S:%s/%%L/%%T/%%N/%%P%%S:%s/%%l_%%t/%%T/%%N/%%P%%S:%s/%%l/%%T/%%N/%%P%%S:%s/%%T/%%N/%%P%%S:%s/%%L/%%T/%%P%%S:%s/%%l_%%t/%%T/%%P%%S:%s/%%l/%%T/%%P%%S:%s/%%T/%%P%%S:%s/%%P%%S:%s/%%L/%%T/%%N/%%P%%S:%s/%%l_%%t/%%T/%%N/%%P%%S:%s/%%l/%%T/%%N/%%P%%S:%s/%%T/%%N/%%P%%S:%s/%%L/%%T/%%P%%S:%s/%%l_%%t/%%T/%%P%%S:%s/%%l/%%T/%%P%%S:%s/%%T/%%P%%S:%s/%%T/%%P%%S" The string seems to be complete. I count 18 %s formats. Christoph -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sat, 26 Nov 2011 17:01:13 +0100
Christoph Bartoschek
What do you mean by slow mode? How can it be entered?
There's two modes in vfprintf(), a 'normal' mode for the easy cases, and a 'slow' mode for everything else. It handles positional parameters, and some extra functionality. The format string looks like it should. The other way to enter the 'slow mode' is with the extra functionality. I wonder if TCL is setting that up. Can you check these values in the debugger when the problem occurs ? p __printf_function_table p __printf_modifier_table p __printf_va_arg_table Rich -- Rich Coe rcoe@wi.rr.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Am 26.11.2011 18:20, schrieb Rich Coe:
On Sat, 26 Nov 2011 17:01:13 +0100 Christoph Bartoschek
wrote: What do you mean by slow mode? How can it be entered?
There's two modes in vfprintf(), a 'normal' mode for the easy cases, and a 'slow' mode for everything else. It handles positional parameters, and some extra functionality.
The format string looks like it should.
The other way to enter the 'slow mode' is with the extra functionality. I wonder if TCL is setting that up.
Can you check these values in the debugger when the problem occurs ?
p __printf_function_table p __printf_modifier_table p __printf_va_arg_table
Rich
(gdb) p __printf_function_table $4 = (printf_function **) 0x8124600 (gdb) p __printf_modifier_table $5 = (struct printf_modifier_record **) 0xc3c2cb0 (gdb) p __printf_va_arg_table $6 = (printf_va_arg_function **) 0x1911c190 (gdb) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sat, 26 Nov 2011 18:55:43 +0100
Christoph Bartoschek
Am 26.11.2011 18:20, schrieb Rich Coe:
Can you check these values in the debugger when the problem occurs ?
p __printf_function_table p __printf_modifier_table p __printf_va_arg_table
Rich
(gdb) p __printf_function_table $4 = (printf_function **) 0x8124600 (gdb) p __printf_modifier_table $5 = (struct printf_modifier_record **) 0xc3c2cb0 (gdb) p __printf_va_arg_table $6 = (printf_va_arg_function **) 0x1911c190 (gdb)
I think I see the issue. The format string passed in has %% specifier, which is supposed to print a literal % instead of being a format specifier. In this case, the code does not recognize the '%%' construct, and fails. I think it should be easy to construct a test case that shows this. Rich -- Rich Coe rcoe@wi.rr.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sat, 26 Nov 2011 15:04:43 -0600
Rich Coe
On Sat, 26 Nov 2011 18:55:43 +0100 Christoph Bartoschek
wrote: I think I see the issue. The format string passed in has %% specifier, which is supposed to print a literal % instead of being a format specifier. In this case, the code does not recognize the '%%' construct, and fails.
I think it should be easy to construct a test case that shows this.
I was wrong. Here's a test case showing that it's working.
It looks like one of the registered printf extension handlers in your
problem is being called because there's a specifier for %s (not sure)?
The info handler is specifying that there is 2 or more args expected and
the info handler is calling through to a null pointer.
Which is really weird, because the info handler has already been called once
in order to get to this point.
I'll have to think if there's a way to avoid or debug and fix this problem.
One possibility is to override the register_printf_function so that it
doesn't cause an issue in the calling program.
#include
Am 27.11.2011 16:40, schrieb Rich Coe:
On Sat, 26 Nov 2011 15:04:43 -0600 Rich Coe
wrote: On Sat, 26 Nov 2011 18:55:43 +0100 Christoph Bartoschek
wrote: I think I see the issue. The format string passed in has %% specifier, which is supposed to print a literal % instead of being a format specifier. In this case, the code does not recognize the '%%' construct, and fails.
I think it should be easy to construct a test case that shows this.
I was wrong. Here's a test case showing that it's working.
It looks like one of the registered printf extension handlers in your problem is being called because there's a specifier for %s (not sure)? The info handler is specifying that there is 2 or more args expected and the info handler is calling through to a null pointer.
Which is really weird, because the info handler has already been called once in order to get to this point.
I'll have to think if there's a way to avoid or debug and fix this problem. One possibility is to override the register_printf_function so that it doesn't cause an issue in the calling program.
I've set a breakpoint on register_printf_function. There is no call to it. The tables you mentioned in an earlier post seem to be empty: (gdb) p __printf_function_table[0] $4 = (printf_function *) 0 (gdb) p __printf_modifier_table[0] $5 = (struct printf_modifier_record *) 0x0 (gdb) p __printf_va_arg_table[0] $6 = (printf_va_arg_function *) 0x7fffcf1ab0b0 Is there anything else I could try? Besides recompiling glibc and the program I use I think I could add debug statements in any code. Maybe I could also recompile glibc if this can be easily done from the srpm. Christoph -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sun, 27 Nov 2011 21:13:05 +0100
Christoph Bartoschek
Am 27.11.2011 16:40, schrieb Rich Coe:
On Sat, 26 Nov 2011 15:04:43 -0600 Rich Coe
wrote: On Sat, 26 Nov 2011 18:55:43 +0100 I've set a breakpoint on register_printf_function. There is no call to it.
It might also be calling register_printf_specifier() which is an alias for register_printf_function(), or calling __register_printf_specifier() which is the real entry point.
The tables you mentioned in an earlier post seem to be empty:
You're only printing the zero-th element of the table. Since they are allocated, they are being used.
(gdb) p __printf_function_table[0] $4 = (printf_function *) 0 (gdb) p __printf_modifier_table[0] $5 = (struct printf_modifier_record *) 0x0 (gdb) p __printf_va_arg_table[0] $6 = (printf_va_arg_function *) 0x7fffcf1ab0b0
Is there anything else I could try? Besides recompiling glibc and the program I use I think I could add debug statements in any code.
You could override the __register_printf_specifier() without recompiling glibc. I'll see if I can make a working example. Rich -- Rich Coe rcoe@wi.rr.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Am 27.11.2011 22:19, schrieb Rich Coe:
On Sun, 27 Nov 2011 21:13:05 +0100 Christoph Bartoschek
wrote: Am 27.11.2011 16:40, schrieb Rich Coe:
On Sat, 26 Nov 2011 15:04:43 -0600 Rich Coe
wrote: On Sat, 26 Nov 2011 18:55:43 +0100 I've set a breakpoint on register_printf_function. There is no call to it.
It might also be calling register_printf_specifier() which is an alias for register_printf_function(), or calling __register_printf_specifier() which is the real entry point.
The tables you mentioned in an earlier post seem to be empty:
You're only printing the zero-th element of the table. Since they are allocated, they are being used.
(gdb) p __printf_function_table[0] $4 = (printf_function *) 0 (gdb) p __printf_modifier_table[0] $5 = (struct printf_modifier_record *) 0x0 (gdb) p __printf_va_arg_table[0] $6 = (printf_va_arg_function *) 0x7fffcf1ab0b0
Is there anything else I could try? Besides recompiling glibc and the program I use I think I could add debug statements in any code.
You could override the __register_printf_specifier() without recompiling glibc. I'll see if I can make a working example.
Rich
I see one four calls to __register_printf_specifier: #0 __register_printf_specifier (spec=102, converter=0x7fffcf1ab060, arginfo=0x7fffcf1aafe0) at reg-printf.c:46 #1 0x00007fffcf1928a2 in ?? () from /usr/lib64/libquadmath.so.0 #0 __register_printf_specifier (spec=101, converter=0x7fffcf1ab060, arginfo=0x7fffcf1aafe0) at reg-printf.c:46 #1 0x00007fffcf1928d2 in ?? () from /usr/lib64/libquadmath.so.0 #0 __register_printf_specifier (spec=69, converter=0x7fffcf1ab060, arginfo=0x7fffcf1aafe0) at reg-printf.c:46 #1 0x00007fffcf1928ea in ?? () from /usr/lib64/libquadmath.so.0 #0 __register_printf_specifier (spec=103, converter=0x7fffcf1ab060, arginfo=0x7fffcf1aafe0) at reg-printf.c:46 #1 0x00007fffcf192902 in ?? () from /usr/lib64/libquadmath.so.0 Christoph -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hi, Rich is our hero. He was able to isolate the bug. He wrote a small test program that triggers the segfault and proves that it is a glibc problem. I've opened a bugreport on the issue: https://bugzilla.novell.com/show_bug.cgi?id=733140 I hope this will be fixed soon because it has in my opinion security implications. Rich's program is attached to the bugreport. Thanks Christoph -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Monday, November 28, 2011 19:32:37 Christoph Bartoschek wrote:
Hi,
Rich is our hero. He was able to isolate the bug. He wrote a small test program that triggers the segfault and proves that it is a glibc problem.
I've opened a bugreport on the issue:
https://bugzilla.novell.com/show_bug.cgi?id=733140
I hope this will be fixed soon because it has in my opinion security implications.
Rich's program is attached to the bugreport.
Thanks a lot - and I see there's even a patch! Great, I'll take care of the online update now! Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (5)
-
Andreas Jaeger
-
Christoph Bartoschek
-
Cristian Rodríguez
-
Rich Coe
-
Sujit Karatparambil