On 2014-06-04 14:33, Anton Aylward wrote:
On 06/04/2014 08:12 AM, Carlos E. R. wrote:
However, in that same control center, I often used plain zgrep (rather cgrep) on an alternative Linux server that also collected the same text logs, in order to quickly search for issues, mostly issues that were not clearly defined.
I did mention 'swatch'. Using perl to do the grepping.
Its useful in that it can deal with things in close proximity in the logs, but it is not a cross correlator of 'deep pattern' tool.
cgrep was an in house variation of grep that knew about correlation of our logs, so it was perfect.
However, the problem was that the central machine and database was very slow and difficult to handle.
I've observed that in some products I've tested, not just syslog tools. They look fine at the trades show/exhibition, but under real loads....
Exactly.
Yes, that sums it up well. Databases are intensive applications. Used in a bank to watch for all manner of attacks and frauds and other chicanery they are very, very different home or small network applications. Syslog databases run full out, non stop. If your database or if your h/w has any weaknesses or errors then they will be shown up.
And yes Java and Javascript simply won't cut it.
No, the database was in double Unix machine, Sun perhaps. The clients we used to interact with it were via web browsers, and thus operating system independent. Those clients used javascript or java, I'm not clear about it. The clients were terribly slow till doubled or quadrupled their ram, and the server was not fast, either. I believe that if the central log server failed, the machines we were supervising would simply cache the data, and send it later. After all, this system was designed for machines that previously printed logs in paper, with technicians permanently on site, 24/7. With actual big fire style ring bells to awake them when needed :-)
But that is implementation issues. The idea is sound.
And yes, I've seen poorly implemented Dbs do that. Try doing proper SQL queries with a dBase style database!
Heh. I take your word for that, I'm not a database expert :-)
I don't think its simply magnetic vs ssd. With ext4 you can preallocate and make sure the database - whatever database - is 'all in one place'.
And with xfs. But the devs said that systemd journal is very fast on their machines, and then we found out they were using SSD. I think they told bug reporters to use SSD, that magnetic media was obsolete, and things like that, in typical systemd team attitude.
Even granted that, there could be many reasons for heavy disk activity. One can't make sweeping assertions, which many contributors to this thread have been doing. I'd ask WHY there was all that disk activity. What is the layout on the disk of that log file? Sequential records? (Often a problem with databases that log time sequential records: hit index, hit file, hit index, hit file; and the records are stored for the convenience of space packing rather than speed of access.)
Yes, that would make sense. But none has explained it here, or posted a link that I noticed. One reason might be that a system log must be flushed to disk early, it can not remain in memory for long. So the design priority could be write speed and reliability, not reading/querying speed. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)