[opensuse] logsanalysis: (tail -f) vs (perl File::Tail)
Hi! Anyone have any view over the best way to watch for changes in log files to do some analysis? I want to create some small scripts/software to watch for changes in log files (like /var/log/messages or var/log/mai) and register some data in a db and I found to possible approaches: * to use Perl File::Tail to listen on a file and process any text that arrives * to use tail -f and pipe the output to my software And I was thinking what would the performance implications be and the preferred way of doing such analysis. Kind regards, -- Marcin Floryan http://marcin.floryan.pl/ Please consider the environment before printing this email. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Marcin Floryan wrote:
Hi!
Anyone have any view over the best way to watch for changes in log files to do some analysis?
I want to create some small scripts/software to watch for changes in log files (like /var/log/messages or var/log/mai) and register some data in a db and I found to possible approaches:
* to use Perl File::Tail to listen on a file and process any text that arrives * to use tail -f and pipe the output to my software
And I was thinking what would the performance implications be and the preferred way of doing such analysis.
Kind regards,
Probably better to send this to the openSuSE programming list than this list. Personal view is that it would be best to use a Perl module if it exists in a script daemon, rather than use a command line call and pipe data to a perl script. Not on performance grounds, but more because one can design the script to handle unusual events and manage processing accordingly (especially if you are backending with a database that in itself may be adding to the logs you are monitoring). But that is a personal preference...., others may be more comfortable with the pipe approach. - -- ============================================================================== I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone. Bjarne Stroustrup ============================================================================== -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHghz7asN0sSnLmgIRAul0AJ9Og5nvnRfLiZW5j7V/XrkpEHe/uwCgsHfF exk2oAyrMwSSCYLOWcBjt4Y= =C/be -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hi, On Jan 7, 2008 2:37 PM, G T Smith <grahamsmith@gandalfsemporium.homelinux.com> wrote:
Marcin Floryan wrote:
Hi!
Anyone have any view over the best way to watch for changes in log files to do some analysis? ... * to use Perl File::Tail to listen on a file and process any text that arrives * to use tail -f and pipe the output to my software ...
Personal view is that it would be best to use a Perl module if it exists in a script daemon, rather than use a command line call and pipe data to a perl script.
Not on performance grounds, but more because one can design the script to handle unusual events and manage processing accordingly (especially if you are backending with a database that in itself may be adding to the logs you are monitoring).
I guess, it depends on what one wants to do with the logs. I used "tail" in a set of tests and found it very flexible and convenient. E.g. I did not want to know exactly in what log file the message I'm looking for should appear. I did something like "tail -n 0 -F /var/log/messages /var/log/secure ... | tee <some file>" and captured the result of tee within perl script with "expect". So when the message appeared I had the file <some file> with all log messages up to this moment and was able to grep there or to do anything else I wanted. (-F works even if the log file is not present when you start "tail" or is "logrotated"). Regards, -- Mark Goldstein -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mark Goldstein wrote:
Hi,
On Jan 7, 2008 2:37 PM, G T Smith <grahamsmith@gandalfsemporium.homelinux.com> wrote:
Marcin Floryan wrote:
Hi!
Anyone have any view over the best way to watch for changes in log files to do some analysis? ... * to use Perl File::Tail to listen on a file and process any text that arrives * to use tail -f and pipe the output to my software ...
Personal view is that it would be best to use a Perl module if it exists in a script daemon, rather than use a command line call and pipe data to a perl script.
Not on performance grounds, but more because one can design the script to handle unusual events and manage processing accordingly (especially if you are backending with a database that in itself may be adding to the logs you are monitoring).
I guess, it depends on what one wants to do with the logs. I used "tail" in a set of tests and found it very flexible and convenient. E.g. I did not want to know exactly in what log file the
<snip> It always does :-) If one is only filtering this is probably fine, as is the approach of using syslog suggested elsewhere ... but the thing I am picking up here is the perl script is intended to talk to something else as well (a database backend by the look of it). The Perl tail related modules offer a variety of options for performing this, Also without the need to call tail or any piping (e.g. File::Tail::App, Event::File::Tail, IO::Tail and File::Tail::FAM all seem to be possibilities here)....
Regards,
- -- ============================================================================== I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone. Bjarne Stroustrup ============================================================================== -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHgi9GasN0sSnLmgIRArY2AJ0cUhYCyKrWKWCUVa67CjH8V1eyOgCfSKfj qR/LASpXQ8BwAN1M/sJ4UaA= =4fGS -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Jan 7, 2008 3:55 PM, G T Smith <grahamsmith@gandalfsemporium.homelinux.com> wrote:
Mark Goldstein wrote:
Hi, ... I guess, it depends on what one wants to do with the logs. I used "tail" in a set of tests and found it very flexible and convenient. E.g. I did not want to know exactly in what log file the
<snip>
It always does :-)
If one is only filtering this is probably fine, as is the approach of using syslog suggested elsewhere ...
but the thing I am picking up here is the perl script is intended to talk to something else as well (a database backend by the look of it).
The Perl tail related modules offer a variety of options for performing this, Also without the need to call tail or any piping (e.g. File::Tail::App, Event::File::Tail, IO::Tail and File::Tail::FAM all seem to be possibilities here)....
Sure, I just gave one example where I thought "tail" was good enough and provided quick solution (not "quick and dirty"). I have to admit, I did not use "File::Tail" from pure ignorance and laziness :-). I'll definitely look into it next time I'll need to to something of this sort. Regards, -- Mark Goldstein -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Monday 2008-01-07 at 16:12 +0200, Mark Goldstein wrote:
Sure, I just gave one example where I thought "tail" was good enough and provided quick solution (not "quick and dirty"). I have to admit, I did not use "File::Tail" from pure ignorance and laziness :-). I'll definitely look into it next time I'll need to to something of this sort.
If you go the "tail -f" way, then you'd better use "tailf" instead. Its a different program that doesn't cause unnecesary disk activity. DESCRIPTION tailf will print out the last 10 lines of a file and then wait for the file to grow. It is similar to tail -f but does not access the file when it is not growing. This has the side effect of not updating the access time for the file, so a filesystem flush does not occur periodically when no log activity is happening. tailf is extremely useful for monitoring log files on a laptop when logging is infrequent and the user desires that the hard disk spin down to conserve battery life. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) iD8DBQFHgkZjtTMYHG2NR9URAsrvAJ0Sv2cUijoQVbb7p4mbij4W9MNL7ACgik+A s5uV9JrETNOBODlMGG2nTS4= =UJsx -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Mark Goldstein pecked at the keyboard and wrote:
On Jan 7, 2008 3:55 PM, G T Smith <grahamsmith@gandalfsemporium.homelinux.com> wrote: <snip>
Sure, I just gave one example where I thought "tail" was good enough and provided quick solution (not "quick and dirty"). I have to admit, I did not use "File::Tail" from pure ignorance and laziness :-). I'll definitely look into it next time I'll need to to something of this sort.
Regards,
Not a programmer just an observer: Wouldn't tailf (preferred over tail -f) possibly create a delay in receiving wanted data while waiting for the buffer to actually write data to the file? -- Ken Schneider SuSe since Version 5.2, June 1998 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Marcin Floryan wrote:
Anyone have any view over the best way to watch for changes in log files to do some analysis?
I would have syslog-ng select and write the lines you want to a named pipe. Then you just run a script reading from stdin and redirect stdin to that pipe. /Per Jessen, Zürich -- http://www.spamchek.com/ - your spam is our business. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (6)
-
Carlos E. R. -
G T Smith -
Ken Schneider -
Marcin Floryan -
Mark Goldstein -
Per Jessen