Hi all, Am 11.06.19 um 18:13 schrieb Martin Caj:
There is a disk /dev/vdb mount as /var/lib/pgsql. The disk is 100GB big and its full.
so, after a good-night hacking session by darix (CC) and me, we were able to resolve the issues and everything should be running again. Also the mirrors should be usable again and the background jobs are running, so new content will end up on the mirrors also shortly.
anyway I spoke with my colleague and they will extend the data disk Tomorrow. (if there is enough space felt on the storage).
Instead of extending the disks, we cleaned up somewhat. Most space was taken up by logfiles and the session table of the weblate database. We truncated this table and compressed the logfiles, which essentially cleaned up 50 GB, so we are all good for now. Also we did implement a cronjob to compress the logfiles on a daily basis. There is still some work to do, though: - Make sure that old / expired session data in weblate (django_sessions) is truncated regularly - Make sure that all data is migrated away from mirrordb3 & mirrodb4 - Decommission mirrordb3 & mirrordb4 - Reclaim storage from mirrordb3 & mirrordb4 (2x 350GB!) Don't forget to give some thanks to darix who provided a lot of help and assistance! He did this basically in his free time and this goes way beyond his commitments from his daily work. Without his dedication this might have taken way longer, as the official processes are somewhat broken right now. Best regards, Karol Babioch