[heroes] Logfiles on pontifex - /var/log/ nearly full
Hello, the /var/log/ partition on pontifex is nearly full (will "explode" in about two days). Do you still need all the old logs, or can you delete some of them? Regards, Christian Boltz -- Eine kurze richtige Antwort (mancher mag sie als unfreundlich bezeichnen) ist besser als eine lange, freundliche, falsche. [Dirk H. Hohndel, SuSE] -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
On Tuesday, December 4, 2018 4:19:11 PM CST Christian Boltz wrote:
the /var/log/ partition on pontifex is nearly full (will "explode" in about two days).
Do you still need all the old logs, or can you delete some of them?
Not for the existing metrics on metrics.o.o. The previous and current day are all that is _needed_, but obviously if something goes wrong having a few days to resolve would be best. The only reason to keep older logs is that more details can be extracted in the future. If the active logs are to be pruned then it would seem logical to prune the old logs on langley as well. -- Jimmy -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Christian Boltz wrote:
Hello,
the /var/log/ partition on pontifex is nearly full (will "explode" in about two days).
There is still 12Gb left, surely we don't use up 6Gb a day?
Do you still need all the old logs, or can you delete some of them?
It is /var/log/apache2 that is taking up the space, 114Gb out of 139Gb. Of those, download.o.o takes 102Gb. 2017 is 18Gb, 2018 is 84Gb. I suggest we get rid of the 2017 logs? That'll give us some room to breathe. I wonder if logrotate used to do this housekeeping or not, but I see no current config that would do it. -- Per Jessen, Zürich (6.5°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Hello, Am Mittwoch, 5. Dezember 2018, 09:24:50 CET schrieb Per Jessen:
Christian Boltz wrote:
the /var/log/ partition on pontifex is nearly full (will "explode" in about two days).
There is still 12Gb left, surely we don't use up 6Gb a day?
When I noticed the nearly-full partition in the monitoring yesterday, there were 6 GB left - but the free disk space heavily depends at which time you look at it ;-) The server indeed writes about 6 GB of logs every day. One detail I overlooked yesterday [1] is that logrotation compresses these 6 GB quite well, so as long as logrotation doesn't break, You can also see all this on https://monitor.opensuse.org/pnp4nagios/graph? host=pontifex2.infra.opensuse.org&srv=fs__var_log&view=2 (especially in the "growth" graph)
Do you still need all the old logs, or can you delete some of them?
It is /var/log/apache2 that is taking up the space, 114Gb out of 139Gb. Of those, download.o.o takes 102Gb. 2017 is 18Gb, 2018 is 84Gb.
I suggest we get rid of the 2017 logs? That'll give us some room to breathe.
Sounds like a good idea, and Jimmy's and Ludwig's answers indicate that we can delete them without causing any harm. Ludwig and Jimmy - if I misunderstood you, please speak up *now*. Otherwise I will delete the 2017 logs tomorrow. (Jimmy, I'll leave it up to you if you want to keep or delete the 2017 logs on langley. However, Ludwig's answer looks like he'd prefer to keep them there.)
I wonder if logrotate used to do this housekeeping or not, but I see no current config that would do it.
Missing log rotation is our favorite reason for full disks, therefore I don't expect that we have that housekeeping in our logrotate config ;-)) Regards, Christian Boltz [1] I had only looked at trend displayed on the summary page, and at the time I looked there was a +3 GB/24h trend (the starting point of this calculation was probably a minute before log rotation ;-) -- in my experience it's safe to assume developers never test [Stephan Kulow in opensuse-factory] -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Christian Boltz schrieb:
Am Mittwoch, 5. Dezember 2018, 09:24:50 CET schrieb Per Jessen:
Christian Boltz wrote:
the /var/log/ partition on pontifex is nearly full (will "explode" in about two days).
There is still 12Gb left, surely we don't use up 6Gb a day?
When I noticed the nearly-full partition in the monitoring yesterday, there were 6 GB left - but the free disk space heavily depends at which time you look at it ;-)
Thanks for taking care btw!
The server indeed writes about 6 GB of logs every day. One detail I overlooked yesterday [1] is that logrotation compresses these 6 GB quite well, so as long as logrotation doesn't break,
You can also see all this on https://monitor.opensuse.org/pnp4nagios/graph? host=pontifex2.infra.opensuse.org&srv=fs__var_log&view=2 (especially in the "growth" graph)
Tells me "Disallowed key characters in global data."
Do you still need all the old logs, or can you delete some of them?
It is /var/log/apache2 that is taking up the space, 114Gb out of 139Gb. Of those, download.o.o takes 102Gb. 2017 is 18Gb, 2018 is 84Gb.
I suggest we get rid of the 2017 logs? That'll give us some room to breathe.
Sounds like a good idea, and Jimmy's and Ludwig's answers indicate that we can delete them without causing any harm.
Ludwig and Jimmy - if I misunderstood you, please speak up *now*. Otherwise I will delete the 2017 logs tomorrow.
Please verify that any logs you delete are actually on langley.
(Jimmy, I'll leave it up to you if you want to keep or delete the 2017 logs on langley. However, Ludwig's answer looks like he'd prefer to keep them there.)
Langley should keep all logs. Don't delete them there please. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Hello, Am Mittwoch, 5. Dezember 2018, 13:37:15 CET schrieb Ludwig Nussel:
Christian Boltz schrieb:
Am Mittwoch, 5. Dezember 2018, 09:24:50 CET schrieb Per Jessen:
Christian Boltz wrote:
[...]
Thanks for taking care btw!
You are welcome ;-)
The server indeed writes about 6 GB of logs every day. One detail I overlooked yesterday [1] is that logrotation compresses these 6 GB quite well, so as long as logrotation doesn't break,
You can also see all this on https://monitor.opensuse.org/pnp4nagios/graph? host=pontifex2.infra.opensuse.org&srv=fs__var_log&view=2 (especially in the "growth" graph)
Tells me "Disallowed key characters in global data."
Ah, the joys of the "evil" cookies from Provo :-( - since some hardware (load balancer?) in Provo was replaced some months ago, all *.opensuse.org services hosted in Provo send out a cookie with an interesting[tm] name, and that cookie breaks various services. Of course Theo opened a ticket, but it might take, well, some time until the provo admins work on it. (On the positive side, we were able to fix paste.o.o, so the cookie now "only" affects heroes-internal services.) Until this is fixed (or we moved everything to Nuremberg - not sure which one will happen first), there is a workaround: Use a private browser window to access the above link.
I suggest we get rid of the 2017 logs? That'll give us some room to breathe.
Sounds like a good idea, and Jimmy's and Ludwig's answers indicate that we can delete them without causing any harm.
Ludwig and Jimmy - if I misunderstood you, please speak up *now*. Otherwise I will delete the 2017 logs tomorrow.
Please verify that any logs you delete are actually on langley.
IIRC langley is not in the heroes network, which means you or Jimmy will have to do this. Regards, Christian Boltz -- Oder kannst du dir ein AUto vorstellen das erst mit einem Benzinmotor fabriziert wird und dann wenn du es mit Diesel betankst auch noch fährt. *lach* [Thomas Templin in suse-linux] -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Christian Boltz schrieb:
Am Mittwoch, 5. Dezember 2018, 13:37:15 CET schrieb Ludwig Nussel:
[...] Please verify that any logs you delete are actually on langley.
IIRC langley is not in the heroes network, which means you or Jimmy will have to do this.
We are not in infra and can only access langley read only. So that would be something for Theo then. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Christian Boltz schrieb:
the /var/log/ partition on pontifex is nearly full (will "explode" in about two days).
Do you still need all the old logs, or can you delete some of them?
Well, for consistency it would be great to store them as it was done before pointifex moved to a new machine. They don't need to stay on pontifex though. There is a system called langley for that purpose. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
On Tue, Dec 04, 2018 at 11:19:11PM +0100, Christian Boltz wrote:
Hello,
the /var/log/ partition on pontifex is nearly full (will "explode" in about two days).
Do you still need all the old logs, or can you delete some of them?
Regards,
Christian Boltz
Rudi and I added another 50GB to the partition Theo
participants (5)
-
Christian Boltz -
Jimmy Berry -
Ludwig Nussel -
Per Jessen -
Theo Chatzimichos