Hello, Am Mittwoch, 5. Dezember 2018, 09:24:50 CET schrieb Per Jessen:

Christian Boltz wrote:

...

the /var/log/ partition on pontifex is nearly full (will "explode" in about two days).

There is still 12Gb left, surely we don't use up 6Gb a day?

When I noticed the nearly-full partition in the monitoring yesterday, there were 6 GB left - but the free disk space heavily depends at which time you look at it ;-) The server indeed writes about 6 GB of logs every day. One detail I overlooked yesterday [1] is that logrotation compresses these 6 GB quite well, so as long as logrotation doesn't break, You can also see all this on https://monitor.opensuse.org/pnp4nagios/graph? host=pontifex2.infra.opensuse.org&srv=fs__var_log&view=2 (especially in the "growth" graph)

...

Do you still need all the old logs, or can you delete some of them?

It is /var/log/apache2 that is taking up the space, 114Gb out of 139Gb. Of those, download.o.o takes 102Gb. 2017 is 18Gb, 2018 is 84Gb.

I suggest we get rid of the 2017 logs? That'll give us some room to breathe.

Sounds like a good idea, and Jimmy's and Ludwig's answers indicate that we can delete them without causing any harm. Ludwig and Jimmy - if I misunderstood you, please speak up *now*. Otherwise I will delete the 2017 logs tomorrow. (Jimmy, I'll leave it up to you if you want to keep or delete the 2017 logs on langley. However, Ludwig's answer looks like he'd prefer to keep them there.)

I wonder if logrotate used to do this housekeeping or not, but I see no current config that would do it.

Missing log rotation is our favorite reason for full disks, therefore I don't expect that we have that housekeeping in our logrotate config ;-)) Regards, Christian Boltz [1] I had only looked at trend displayed on the summary page, and at the time I looked there was a +3 GB/24h trend (the starting point of this calculation was probably a minute before log rotation ;-) -- in my experience it's safe to assume developers never test [Stephan Kulow in opensuse-factory] -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Hello, Am Mittwoch, 5. Dezember 2018, 13:37:15 CET schrieb Ludwig Nussel:

Christian Boltz schrieb:

...

Am Mittwoch, 5. Dezember 2018, 09:24:50 CET schrieb Per Jessen:

...

Christian Boltz wrote:

[...]

Thanks for taking care btw!

You are welcome ;-)

...

The server indeed writes about 6 GB of logs every day. One detail I overlooked yesterday [1] is that logrotation compresses these 6 GB quite well, so as long as logrotation doesn't break,

You can also see all this on https://monitor.opensuse.org/pnp4nagios/graph? host=pontifex2.infra.opensuse.org&srv=fs__var_log&view=2 (especially in the "growth" graph)

Tells me "Disallowed key characters in global data."

Ah, the joys of the "evil" cookies from Provo :-( - since some hardware (load balancer?) in Provo was replaced some months ago, all *.opensuse.org services hosted in Provo send out a cookie with an interesting[tm] name, and that cookie breaks various services. Of course Theo opened a ticket, but it might take, well, some time until the provo admins work on it. (On the positive side, we were able to fix paste.o.o, so the cookie now "only" affects heroes-internal services.) Until this is fixed (or we moved everything to Nuremberg - not sure which one will happen first), there is a workaround: Use a private browser window to access the above link.

...

...

I suggest we get rid of the 2017 logs? That'll give us some room to breathe.

Sounds like a good idea, and Jimmy's and Ludwig's answers indicate that we can delete them without causing any harm.

Ludwig and Jimmy - if I misunderstood you, please speak up *now*. Otherwise I will delete the 2017 logs tomorrow.

Please verify that any logs you delete are actually on langley.

IIRC langley is not in the heroes network, which means you or Jimmy will have to do this. Regards, Christian Boltz -- Oder kannst du dir ein AUto vorstellen das erst mit einem Benzinmotor fabriziert wird und dann wenn du es mit Diesel betankst auch noch fährt. *lach* [Thomas Templin in suse-linux] -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Hello, Am Mittwoch, 5. Dezember 2018, 09:24:50 CET schrieb Per Jessen:

Christian Boltz wrote:

...

the /var/log/ partition on pontifex is nearly full (will "explode" in about two days).

There is still 12Gb left, surely we don't use up 6Gb a day?

When I noticed the nearly-full partition in the monitoring yesterday, there were 6 GB left - but the free disk space heavily depends at which time you look at it ;-) The server indeed writes about 6 GB of logs every day. One detail I overlooked yesterday [1] is that logrotation compresses these 6 GB quite well, so as long as logrotation doesn't break, You can also see all this on https://monitor.opensuse.org/pnp4nagios/graph? host=pontifex2.infra.opensuse.org&srv=fs__var_log&view=2 (especially in the "growth" graph)

...

Do you still need all the old logs, or can you delete some of them?

It is /var/log/apache2 that is taking up the space, 114Gb out of 139Gb. Of those, download.o.o takes 102Gb. 2017 is 18Gb, 2018 is 84Gb.

I suggest we get rid of the 2017 logs? That'll give us some room to breathe.

Sounds like a good idea, and Jimmy's and Ludwig's answers indicate that we can delete them without causing any harm. Ludwig and Jimmy - if I misunderstood you, please speak up *now*. Otherwise I will delete the 2017 logs tomorrow. (Jimmy, I'll leave it up to you if you want to keep or delete the 2017 logs on langley. However, Ludwig's answer looks like he'd prefer to keep them there.)

I wonder if logrotate used to do this housekeeping or not, but I see no current config that would do it.

Missing log rotation is our favorite reason for full disks, therefore I don't expect that we have that housekeeping in our logrotate config ;-)) Regards, Christian Boltz [1] I had only looked at trend displayed on the summary page, and at the time I looked there was a +3 GB/24h trend (the starting point of this calculation was probably a minute before log rotation ;-) -- in my experience it's safe to assume developers never test [Stephan Kulow in opensuse-factory] -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Hello, Am Mittwoch, 5. Dezember 2018, 13:37:15 CET schrieb Ludwig Nussel:

Christian Boltz schrieb:

...

Am Mittwoch, 5. Dezember 2018, 09:24:50 CET schrieb Per Jessen:

...

Christian Boltz wrote:

[...]

Thanks for taking care btw!

You are welcome ;-)

...

The server indeed writes about 6 GB of logs every day. One detail I overlooked yesterday [1] is that logrotation compresses these 6 GB quite well, so as long as logrotation doesn't break,

You can also see all this on https://monitor.opensuse.org/pnp4nagios/graph? host=pontifex2.infra.opensuse.org&srv=fs__var_log&view=2 (especially in the "growth" graph)

Tells me "Disallowed key characters in global data."

Ah, the joys of the "evil" cookies from Provo :-( - since some hardware (load balancer?) in Provo was replaced some months ago, all *.opensuse.org services hosted in Provo send out a cookie with an interesting[tm] name, and that cookie breaks various services. Of course Theo opened a ticket, but it might take, well, some time until the provo admins work on it. (On the positive side, we were able to fix paste.o.o, so the cookie now "only" affects heroes-internal services.) Until this is fixed (or we moved everything to Nuremberg - not sure which one will happen first), there is a workaround: Use a private browser window to access the above link.

...

...

I suggest we get rid of the 2017 logs? That'll give us some room to breathe.

Sounds like a good idea, and Jimmy's and Ludwig's answers indicate that we can delete them without causing any harm.

Ludwig and Jimmy - if I misunderstood you, please speak up *now*. Otherwise I will delete the 2017 logs tomorrow.

Please verify that any logs you delete are actually on langley.

IIRC langley is not in the heroes network, which means you or Jimmy will have to do this. Regards, Christian Boltz -- Oder kannst du dir ein AUto vorstellen das erst mit einem Benzinmotor fabriziert wird und dann wenn du es mit Diesel betankst auch noch fährt. *lach* [Thomas Templin in suse-linux] -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org