Mailinglist Archive: opensuse (2912 mails)

[Jon Nelson <jnelson-suse@xxxxxxxxxxx>]

On Mon, 7 Feb 2005, Rhugga wrote:

> 
> I have a very large syslog file, over 1 million plus lines and I am trying to truncate duplicate
> entries, for example this filesystem message that spams every few seconds:
> 
> Feb  1 00:17:00 db-0202 ufs: [ID 845546 kern.notice] NOTICE: alloc: /u70: file system full
> Feb  1 00:17:04 db-0202 ufs: [ID 845546 kern.notice] NOTICE: alloc: /u70: file system full
> Feb  1 00:17:20 db-0202 ufs: [ID 845546 kern.notice] NOTICE: alloc: /u70: file system full
> Feb  1 00:17:49 db-0202 ufs: [ID 845546 kern.notice] NOTICE: alloc: /u70: file system full
> 
> So I am doing this:
> 
> cat syslog | sort | uniq -f3 > tempfile

Aha.  What you want here is this:

cat syslog | sort --key=5 | uniq --skip-fields=4 > tempfile

That will skip "Feb", "1", "AA:BB:CC" and "db-0202" in both sorting
and determining uniqueness.  You could change both of those to 1 less
than my example (4 and 3 respectively) if db-0202 is not going to be
unique. 

--
Carpe diem - Seize the day.
Carp in denim - There's a fish in my pants!

Jon Nelson <jnelson-suse@xxxxxxxxxxx>