[Bug 656197] New: Unable to log in due to disk full because unbootable because syslog registered one message many thousands of times
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c0 Summary: Unable to log in due to disk full because unbootable because syslog registered one message many thousands of times Classification: openSUSE Product: openSUSE 11.2 Version: Final Platform: 32bit OS/Version: openSUSE 11.2 Status: NEW Severity: Normal Priority: P5 - None Component: Printing AssignedTo: jsmeix@novell.com ReportedBy: dmp5@uwm.edu QAContact: jsmeix@novell.com Found By: --- Blocker: --- Created an attachment (id=402380) --> (http://bugzilla.novell.com/attachment.cgi?id=402380) The log file for messages and wern (same as paste) User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101026 SUSE/3.6.12-0.7.1 Firefox/3.6.12 Here is the type of printer used: Description: HP Deskjet 6940 series Location: Office Printer Driver: HP Deskjet 6940 Series, hpcups 3.9.8 Printer State: idle, accepting jobs, published. Printer works fine, wireless The problem was I was out of black ink (I didn't realize this) and I had 6 jobs waiting to print, 5 were test pages because my 1 document wouldn't print, so I kept trying to see if it was a wireless issue with each new test page, eventually I gave up. I'm not sure when but a few days laster I could not log in. I would get to my splash screen. I would type my username and password and the process would keep up and never log in. I used a a live disc to learn than my /var/log was at 27g. Messages and wern held the majority 13.8g each. As going through the forums process in opensuse, Here is my post http://forums.opensuse.org/english/get-help-here/install-boot-login/450335-c... I learned, "syslog registered one message many thousands of times, several gigabytes of logs, instead of filtering the repeated message. Here are the two files that were the trouble http://susepaste.org/860249 After learning the two files were able to be deleted, I am back up and running as usual. Reproducible: Didn't try Steps to Reproduce: 1. 2. 3. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c1 Johannes Meixner <jsmeix@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |DUPLICATE Summary|Unable to log in due to |hpijs/services.cpp fills up |disk full because |log files with thousands of |unbootable because syslog |same messages |registered one message many | |thousands of times | --- Comment #1 from Johannes Meixner <jsmeix@novell.com> 2010-11-30 09:15:39 UTC --- http://susepaste.org/860249 shows that prnt/hpijs/services.cpp logs zillions of same messages in one same second like --------------------------------------------------------------------- Nov 24 18:51:30 linux-icwg Deskjet-6940-series: prnt/hpijs/services.cpp 386: unable to write to output, fd=1, count=4096: Broken pipe --------------------------------------------------------------------- See https://bugzilla.novell.com/show_bug.cgi?id=557000#c2 There is already a matching upstream bug report: https://bugs.launchpad.net/hplip/+bug/439550 and there should be the workaround: Stop cups printer and restart cups printer, then the job should be processed correctly. *** This bug has been marked as a duplicate of bug 557000 *** http://bugzilla.novell.com/show_bug.cgi?id=557000 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c2 Carlos Robinson <carlos.e.r@opensuse.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED CC| |carlos.e.r@opensuse.org Component|Printing |Basesystem Resolution|DUPLICATE | AssignedTo|jsmeix@novell.com |bnc-team-screening@forge.pr | |ovo.novell.com QAContact|jsmeix@novell.com |qa@suse.de --- Comment #2 from Carlos Robinson <carlos.e.r@opensuse.org> 2010-11-30 18:12:04 UTC --- Please allow me. Besides the printing problem, there is a problem with syslog: it did not filter out the thousands of repeated log entries till the system crashed. That is, IMO, a more important problem. I'll change component to "basesystem" and reopen for re-consideration. David: Please add the result of this command: grep SYSLOG_DAEMON /etc/sysconfig/syslog -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c3 Johannes Meixner <jsmeix@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jsmeix@novell.com Summary|hpijs/services.cpp fills up |syslog does not filter that |log files with thousands of |hpijs/services.cpp fills up |same messages |log files with thousands of | |same messages --- Comment #3 from Johannes Meixner <jsmeix@novell.com> 2010-12-01 09:38:40 UTC --- FYI: In the HPLIP sources in /prnt/hpijs/services.cpp there is the line ------------------------------------------------------------------------------ BUG("unable to write to output, fd=%d, count=%d: %m\n", OutputPath, *Count); ------------------------------------------------------------------------------ and "BUG" is defined in prnt/hpijs/bug.h (long line wrapped here): ------------------------------------------------------------------------------ #include <syslog.h> #include <stdio.h> #define _STRINGIZE(x) #x #define STRINGIZE(x) _STRINGIZE(x) #define BUG(args...) {syslog(LOG_ERR, __FILE__ " " STRINGIZE(__LINE__) ": " args); \ fprintf(stderr, __FILE__ " " STRINGIZE(__LINE__) ": " args);} ------------------------------------------------------------------------------ -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c Johannes Meixner <jsmeix@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |NEEDINFO InfoProvider| |dmp5@uwm.edu -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c4 Johannes Meixner <jsmeix@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |RESOLVED CC| |mt@novell.com InfoProvider|dmp5@uwm.edu | Resolution| |UPSTREAM Summary|syslog does not filter that |syslog-ng does not filter |hpijs/services.cpp fills up |that hpijs/services.cpp |log files with thousands of |fills up log files with |same messages |thousands of same messages --- Comment #4 from Johannes Meixner <jsmeix@novell.com> 2010-12-02 09:59:20 UTC --- By default we use syslog-ng (I don't know since which openSUSE version) but syslog-ng does not filter out duplicates (what the old syslogd did). This syslog-ng behaviour is not a bug in syslog-ng but at least currently it is the intended behaviour of syslog-ng which could be only changed by upstream syslog-ng. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c6 Carlos Robinson <carlos.e.r@opensuse.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|UPSTREAM | --- Comment #6 from Carlos Robinson <carlos.e.r@opensuse.org> 2010-12-02 11:57:10 UTC --- (In reply to comment #4)
By default we use syslog-ng (I don't know since which openSUSE version) but syslog-ng does not filter out duplicates (what the old syslogd did).
That is not entirely correct. 11.2 and 11.3 use rsyslog by default (I just checked three installs of 11.2 and one of 11.3). 11.0 used syslog-ng. 11.1 I don't know. Please, lets wait till David can confirm what he has before closing this one. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c7 Carlos Robinson <carlos.e.r@opensuse.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |NEEDINFO InfoProvider| |dmp5@uwm.edu --- Comment #7 from Carlos Robinson <carlos.e.r@opensuse.org> 2010-12-02 11:58:04 UTC --- David: Please add the result of this command: grep SYSLOG_DAEMON /etc/sysconfig/syslog -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c8 Marius Tomaschewski <mt@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pczanik@genesi-usa.com --- Comment #8 from Marius Tomaschewski <mt@novell.com> 2010-12-02 15:55:10 UTC --- As workaround for the duplicate message problem, you have following options: - add a filter discarding the messsage to the syslog-ng.conf - install rsyslog (default) and set "$RepeatedMsgReduction on" in /etc/rsyslog.conf - install syslogd. That the syslog-ng does not filter duplicates, is known upstream since many many years. People ask for this feature on the mailing lists and there are several patches for different syslog-ng versions (almost each version is a complete rewrite) on the mailing lists [I wrote one for 1.6.x version as well], but the patches AFAIK never found their way into the upstream versions. https://lists.balabit.hu/pipermail/syslog-ng/2004-February/005817.html https://lists.balabit.hu/pipermail/syslog-ng/2008-May/011698.html So from syslog-ng side => Resolve as UPSTREAM. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c9 --- Comment #9 from peter czanik <pczanik@genesi-usa.com> 2010-12-02 20:27:52 UTC --- Please check the suppress() option at http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guid... It is not used as default, but available. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c10 --- Comment #10 from Marius Tomaschewski <mt@novell.com> 2010-12-02 22:00:02 UTC --- Ah... this means, it has been implemented in 3.1 after all. But there is a 2.0.9 on 11.2... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c11 --- Comment #11 from David Pulkowski <dmp5@uwm.edu> 2010-12-05 22:46:25 UTC --- (In reply to comment #7)
David:
Please add the result of this command:
grep SYSLOG_DAEMON /etc/sysconfig/syslog
I'm not sure if you got this or not, The return was this
SYSLOG_DAEMON="rsyslogd"
-- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c Xinli Niu <xlniu@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED CC| |xlniu@novell.com InfoProvider|dmp5@uwm.edu | AssignedTo|bnc-team-screening@forge.pr |carlos.e.r@opensuse.org |ovo.novell.com | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c12 Carlos Robinson <carlos.e.r@opensuse.org> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|carlos.e.r@opensuse.org |bnc-team-screening@forge.pr | |ovo.novell.com Summary|syslog-ng does not filter |rsyslogd does not filter |that hpijs/services.cpp |that hpijs/services.cpp |fills up log files with |fills up log files with |thousands of same messages |thousands of same messages --- Comment #12 from Carlos Robinson <carlos.e.r@opensuse.org> 2010-12-06 13:37:53 UTC --- Thanks David. Yes, that's what I thought. Changing the tittle accordingly. WAS: Bug 656197 - syslog-ng does not filter that hpijs/services.cpp fills up log files with thousands of same messages NEW: Bug 656197 - rsyslogd does not filter that hpijs/services.cpp fills up log files with thousands of same messages P.S: I see this was assigned to me. I'm not the maintainer, just a helper trying to get the correct information. Reverting to defaults. This is a bug either in rsyslog or in its openSUSE configuration -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c13 Marius Tomaschewski <mt@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|bnc-team-screening@forge.pr |mt@novell.com |ovo.novell.com | --- Comment #13 from Marius Tomaschewski <mt@novell.com> 2010-12-06 14:34:48 UTC --- Thanks Carlos, David! (In reply to comment #12)
This is a bug either in rsyslog or in its openSUSE configuration
It is neither a bug in rsyslog, nor in openSUSE, but as already written in comment 8, it is a configration option and openSUSE is using the rsyslogd default setting of "$RepeatedMsgReduction off". I'm quite sure, somebody will report the change as bug [that messages dups are dropped], but I've changed to explicit "on" in Base:System / Factory, because it useful to catch amok running programs logging exactly same log message... But this IMO does not change the initial resolution as UPSTREAM: the bug is in hpijs/services.cpp. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c14 Marius Tomaschewski <mt@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO InfoProvider| |maintenance@opensuse.org --- Comment #14 from Marius Tomaschewski <mt@novell.com> 2010-12-06 14:40:28 UTC --- Mr Maintenance, do we need to provide update packages with a change of /etc/rsyslog.conf to use: "$RepeatedMsgReduction on" as default? Another Objections/Instructions? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c15 --- Comment #15 from Carlos Robinson <carlos.e.r@opensuse.org> 2010-12-06 17:29:03 UTC ---
It is neither a bug in rsyslog, nor in openSUSE, but as already written in comment 8, it is a configration option and openSUSE is using the rsyslogd default setting of "$RepeatedMsgReduction off".
Question: When a message is repeated many times and filtered out, doesn't rsyslog print a message of the type "skipped 100 identical messages", or something like that? I have seen that on openSUSE logs, time ago... If that is not so, I think it would be useful perhaps to leave messages on in factory, and filter them in the stable releases. As to the bug being solely the fault of hpijs/services.cpp, I mildly object: a faulty program should not crash a system (disk full). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c16 --- Comment #16 from Johannes Meixner <jsmeix@novell.com> 2010-12-07 09:25:16 UTC --- Only FYI: Some days ago at home our file system where /home/ is mounted was filled up by Gimp when it crashed while processing a huge image file which resulted an incredible huge file in Gimp's directory where Gimp stores its internal (temporary) stuff. Almost any (faulty) program can fill up the disk. E.g. like ---------------------------------------------------------------- # cat /dev/urandom >fillup_file ---------------------------------------------------------------- Of course some space is reserved on "/" for root and one could set up disk quotas and so on... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c17 --- Comment #17 from Marius Tomaschewski <mt@novell.com> 2010-12-07 09:54:31 UTC --- (In reply to comment #15)
It is neither a bug in rsyslog, nor in openSUSE, but as already written in comment 8, it is a configration option and openSUSE is using the rsyslogd default setting of "$RepeatedMsgReduction off".
Question:
When a message is repeated many times and filtered out, doesn't rsyslog print a message of the type "skipped 100 identical messages", or something like that? I have seen that on openSUSE logs, time ago...
As I already wrote: - syslogd does this by default. - for syslog-ng-1.6 I wrote a patch to do it by default as well. in syslog-ng-2.0.9, there is no suppress() option doing it as in syslog-ng-3.1. - in rsyslogd it is a option with built-in default 'off'. For openSUSE-11.4 I've enabled the RepeatedMsgReduction option and I'll enable the suppress() option for syslog-ng as well in the next submit to catch this special case. Because it is an option in rsyslog (=default), I've asked in comment 14, whether I have to provide an update package with new /etc/rsyslog.conf, that enables this option or not.
If that is not so, I think it would be useful perhaps to leave messages on in factory, and filter them in the stable releases.
As to the bug being solely the fault of hpijs/services.cpp, I mildly object: a faulty program should not crash a system (disk full).
It is not possible to prohibit this. The option catches a special case only: programs that log *exactly* same message. As soon as a message differs even one bit, the message gets logged. So a user or program just need to run e.g. the following to fill up the log-disk: i=0 ; while true ; do logger "======== $((i++)) ==========" ; done ==>> Best is to always put /var/log to a separate disk. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c18 --- Comment #18 from Marius Tomaschewski <mt@novell.com> 2010-12-07 09:58:34 UTC --- (In reply to comment #17)
- for syslog-ng-1.6 I wrote a patch to do it by default as well. in syslog-ng-2.0.9, there is no suppress() option doing it as in syslog-ng-3.1.
BTW: Also here: suppress() is an option in 3.1 and 'off' by default. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c19 Christian Dengler <cdengler@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cdengler@novell.com --- Comment #19 from Christian Dengler <cdengler@novell.com> 2010-12-08 16:36:01 UTC ---
Because it is an option in rsyslog (=default), I've asked in comment 14, whether I have to provide an update package with new /etc/rsyslog.conf, that enables this option or not.
IMHO we should only change the behavior in the next openSUSE release (-1). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c20 Christian Dengler <cdengler@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|maintenance@opensuse.org | --- Comment #20 from Christian Dengler <cdengler@novell.com> 2011-01-03 12:19:38 UTC --- So no update here. Be so kind and fix it with the next openSUSE release. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656197 https://bugzilla.novell.com/show_bug.cgi?id=656197#c21 Marius Tomaschewski <mt@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |FIXED --- Comment #21 from Marius Tomaschewski <mt@novell.com> 2011-01-07 09:47:53 UTC --- fixed in factory for 11.4. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com