Mailinglist Archive: opensuse (2912 mails)

[Randall R Schulz <rschulz@xxxxxxxxx>]

Chuck,

On Monday 07 February 2005 08:01, Rhugga wrote:
> I have a very large syslog file, over 1 million plus lines and I am
> trying to truncate duplicate entries, for example this filesystem
> message that spams every few seconds:
>
> Feb  1 00:17:00 db-0202 ufs: [ID 845546 kern.notice] NOTICE: alloc: /u70: file system full
> Feb  1 00:17:04 db-0202 ufs: [ID 845546 kern.notice] NOTICE: alloc: /u70: file system full
> Feb  1 00:17:20 db-0202 ufs: [ID 845546 kern.notice] NOTICE: alloc: /u70: file system full
> Feb  1 00:17:49 db-0202 ufs: [ID 845546 kern.notice] NOTICE: alloc: /u70: file system full
> 
> So I am doing this:
>
> cat syslog | sort | uniq -f3 > tempfile
>
> This manages to reduce about 103,894 messages like those above to
> only 3916, but it should be reducing them to 1. The only difference
> between these entries are the timestamp, which -f3 is telling uniq to
> ignore.
>
> So, I next started looking at the resulting 3916 messages and can't
> find any differences between them other than within the first 16
> characters of each line, which is the timestamp. I compared free
> space positions, I checked for non-printable characters using cat -a,
> etc....
>
> I also tried adding -c to uniq to have it prefix each outline with a
> count of the number of duplicates removed, and that seems to be
> working. (however, still getting 3916 identical messages in the
> outfile)

The "uniq" command only looks at adjacent lines, so other log
messages--those unrelated to the file system full are interrupting
uniq's deletion of repeated lines.

Also, the use of "sort" is really gratuitous, since the messages are
time-stamped and sort is only going to reorder lines that share a
timestamp.


> Ugh. Next step is to compile solaris's uniq on SLES 9 and see if that
> resolves the problem. Anyone else do any heavy duty work with uniq
> and notice anything similiar to what I am seeing?
>
> Thx
>
> =====
> Chuck Carson


Randall Schulz