is mirrordb1 down? # mb show xmission Traceback (most recent call last): File "/usr/bin/mb", line 1688, in <module> r = mirrordoctor.main() File "/usr/lib/python2.7/site-packages/cmdln.py", line 251, in main retval = self.postoptparse() File "/usr/bin/mb", line 83, in postoptparse self.conn = mb.conn.Conn(self.config.dbconfig, debug = self.options.debug) File "/usr/lib64/python2.7/site-packages/mb/conn.py", line 150, in __init__ class Version(SQLObject): File "/usr/lib/python2.7/site-packages/sqlobject/declarative.py", line 92, in __new__ cls.__classinit__(cls, new_attrs) File "/usr/lib/python2.7/site-packages/sqlobject/main.py", line 789, in __classinit__ sqlmeta.addColumnsFromDatabase() File "/usr/lib/python2.7/site-packages/sqlobject/main.py", line 442, in addColumnsFromDatabase for columnDef in conn.columnsFromSchema(sqlmeta.table, soClass): File "/usr/lib/python2.7/site-packages/sqlobject/postgres/pgconnection.py", line 279, in columnsFromSchema keyData = self.queryAll(keyQuery % self.sqlrepr(tableName)) File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 427, in queryAll return self._runWithConnection(self._queryAll, s) File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 325, in _runWithConnection conn = self.getConnection() File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 336, in getConnection conn = self.makeConnection() File "/usr/lib/python2.7/site-packages/sqlobject/postgres/pgconnection.py", line 142, in makeConnection raise OperationalError("%s; used connection string %r" % (e, self.dsn)) sqlobject.dberrors.OperationalError: could not connect to server: Connection refused Is the server running on host "mirrordb1.infra.opensuse.org" (192.168.47.26) and accepting TCP/IP connections on port 5432? ; used connection string 'dbname=mb_opensuse2 user=mb password=Kae4EemuHiequai7 host=mirrordb1.infra.opensuse.org port=5432' -- Per Jessen, Zürich (12.8°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Per Jessen wrote:
is mirrordb1 down?
https://progress.opensuse.org/issues/52724 I am unable to login, not sure why. -- Per Jessen, Zürich (15.1°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Per Jessen composed on 2019-06-07 11:04 (UTC+0200):
Per Jessen wrote:
is mirrordb1 down?
I am unable to login, not sure why.
I have no problem logging in on POO, but it's impossible any more on FOO and BOO. Please unprivatize issues 52772 & 52775. -- Evolution as taught in public schools is religion, not science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Felix Miata wrote:
Per Jessen composed on 2019-06-07 11:04 (UTC+0200):
Per Jessen wrote:
is mirrordb1 down?
I am unable to login, not sure why.
I have no problem logging in on POO, but it's impossible any more on FOO and BOO.
Different login issues. I'm trying to ssh to mirrordb1, but something's wrong with my ssh-agent.
Please unprivatize issues 52772 & 52775.
52772 was already public, I have updated 52775. -- Per Jessen, Zürich (15.6°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Hi All I did a quick check today and the problem is on mirrordb1 with disk space There is a disk /dev/vdb mount as /var/lib/pgsql. The disk is 100GB big and its full. on the PG-slave the disk /dev/vdb (the same size) has 15 GB free disk space.. anyway I spoke with my colleague and they will extend the data disk Tomorrow. (if there is enough space felt on the storage). I'm sorry for this, but since Theo left and Thorsten leaving now we have no replacement.... Managing the infra for openSUSE is harder and harder .... Martin On 09.06.19 09:55, Per Jessen wrote:
Felix Miata wrote:
Per Jessen composed on 2019-06-07 11:04 (UTC+0200):
Per Jessen wrote:
is mirrordb1 down?
I am unable to login, not sure why.
I have no problem logging in on POO, but it's impossible any more on FOO and BOO.
Different login issues. I'm trying to ssh to mirrordb1, but something's wrong with my ssh-agent.
Please unprivatize issues 52772 & 52775.
52772 was already public, I have updated 52775.
--
Martin Caj
Martin Caj wrote:
Hi All
I did a quick check today and the problem is on mirrordb1 with disk space
There is a disk /dev/vdb mount as /var/lib/pgsql. The disk is 100GB big and its full.
on the PG-slave
the disk /dev/vdb (the same size) has 15 GB free disk space..
anyway I spoke with my colleague and they will extend the data disk Tomorrow. (if there is enough space felt on the storage).
Do we need to run an optimize or reorg? ISTR something about pqsql needing that. -- Per Jessen, Zürich (15.9°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Hello, Am Dienstag, 11. Juni 2019, 18:13:34 CEST schrieb Martin Caj:
I did a quick check today and the problem is on mirrordb1 with disk space
There is a disk /dev/vdb mount as /var/lib/pgsql. The disk is 100GB big and its full.
Right, that's what the monitoring also says since Friday.
on the PG-slave
the disk /dev/vdb (the same size) has 15 GB free disk space..
... and, according to the monitoring, wastes another 12 GB: POSTGRES_BLOAT CRITICAL: DB "mb_opensuse2" (db mb_opensuse2) table public.filearr rows:29279714 pages:2273999 shouldbe:587601 (3.9X) wasted size:13814972416 (12 GB) so after fixing the full disk on the master, you should probably run the cleanup routines (vacuum?) on the slave.
anyway I spoke with my colleague and they will extend the data disk Tomorrow. (if there is enough space felt on the storage).
We had a similar situation (disk full) a while ago, and _IIRC_ the problem was caused by old replication logs not being deleted. Maybe you should check for those logs (and ensure they get dropped automatically once old enough) instead of blindly adding more disk space just to delay the problem to next week ;-) As a sidenote: The mirrordb*.infra.o.o machines are not in salt and not connected to the FreeIPA accounts, which means I can't login there. (That's not as bad as it sounds because I don't know much about Postgres and wouldn't touch it anyway ;-)
I'm sorry for this, but since Theo left and Thorsten leaving now we have no replacement....
Managing the infra for openSUSE is harder and harder ....
It's not really harder, we "only" lack some people ;-) Regards, Christian Boltz --
Surely a nice initiative - but maybe over-engineered? How do you define over-engineered? I just spent 1-2h on it, so there cannot be much engineering in it. [> Dominique Leuenberger and Bernhard M. Wiedemann in opensuse-factory]
-- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Christian Boltz wrote:
As a sidenote: The mirrordb*.infra.o.o machines are not in salt and not connected to the FreeIPA accounts, which means I can't login there.
Ditto. -- Per Jessen, Zürich (16.8°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org
Hi all, Am 11.06.19 um 18:13 schrieb Martin Caj:
There is a disk /dev/vdb mount as /var/lib/pgsql. The disk is 100GB big and its full.
so, after a good-night hacking session by darix (CC) and me, we were able to resolve the issues and everything should be running again. Also the mirrors should be usable again and the background jobs are running, so new content will end up on the mirrors also shortly.
anyway I spoke with my colleague and they will extend the data disk Tomorrow. (if there is enough space felt on the storage).
Instead of extending the disks, we cleaned up somewhat. Most space was taken up by logfiles and the session table of the weblate database. We truncated this table and compressed the logfiles, which essentially cleaned up 50 GB, so we are all good for now. Also we did implement a cronjob to compress the logfiles on a daily basis. There is still some work to do, though: - Make sure that old / expired session data in weblate (django_sessions) is truncated regularly - Make sure that all data is migrated away from mirrordb3 & mirrodb4 - Decommission mirrordb3 & mirrordb4 - Reclaim storage from mirrordb3 & mirrordb4 (2x 350GB!) Don't forget to give some thanks to darix who provided a lot of help and assistance! He did this basically in his free time and this goes way beyond his commitments from his daily work. Without his dedication this might have taken way longer, as the official processes are somewhat broken right now. Best regards, Karol Babioch
participants (5)
-
Christian Boltz -
Felix Miata -
Karol Babioch -
Martin Caj -
Per Jessen