- Heroes

[heroes] Testing Period for New Progress Redmine
by Estu Fardani 15 Apr '20

15 Apr '20

Dear All, New Progress will enter testing stage. We need help to testing this service. To use as much as old redmine. Testing Period will be around 1 weeks and will end at 15th April. Please give us feedback, report bug or broken feature as you can by reply this email. Here our new redmine: url: https://progress-test.opensuse.org/ user: use the same credential from old redmine. Thank you. Regards - Estu Fardani - +62 856 432 14898 https://tuanpembual.wordpress.com -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

2 5

[heroes] Reminder: Heroes meeting on tuesday
by Christian Boltz 05 Apr '20

05 Apr '20

Hello, the next heroes meeting will be on tuesday (2020-04-07) at 20:00 CEST / 18:00 UTC in the #opensuse-admin IRC channel. Note: Europe switches to summer time, which means the UTC time is now 18:00. See https://progress.opensuse.org/issues/64168 for the usual tickets, and feel free to add whatever you want to discuss Regards, Christian Boltz -- I strongly object that we spend any sleep or human resources attempting to cater to the "pentium 1 , 512 mb" use-case. A phone probably has more computing power than that these days. [Cristian Rodríguez in opensuse-factory] -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] Brief update
by Per Jessen 01 Apr '20

01 Apr '20

Instead of taking part in the meeting last night, I enabled TLS on anna and elsa for outbound mail. anna had a small queue of mails that could not be delivered due to the recipient server insisting on TLS. See ticket#40574 for more details. Forums update - no progress since last report. Still need to look at the authentication. -- Per Jessen, Zürich (2.1°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 1

[heroes] Mail setup
by Lars Vogdt 30 Mar '20

30 Mar '20

Hi @ll I finally want to start with the new Email setup for openSUSE - and I'm currently looking for volunteers... :-) Planned short term: * set SPF records for openSUSE domains * install mx1 and mx2.opensuse.org as incoming servers ** use postfix and rspamd ** integrate clamav (and reject messages seen as spam directly) ** integrate the alias table for members Planned mid term: * enable DCIM on all outgoing mail servers Something I missed? Regards, Lars -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

5 6

[heroes] Galera changes
by Lars Vogdt 27 Mar '20

27 Mar '20

Hi After the latest changes and new databases (hello, forums! ;-), I decided that it's time to check the current situation of our database cluster again. You might know mysqltuner.pl and tuning-primer.sh from the mysql-tuning-scripts package: the two tools analyze a running mysql/mariadb and galera cluster and print out statistics and recommendations (and even more, if you call them with the right options ;-). Both tools recommended to increase the "InnoDB buffer pool size". I cross checked this with an SQL statement stolen from stackexchange[1]: => result of all tools: Recommended_InnoDB_Buffer_Pool_Size = 32G As the nodes inside the cluster just had 16G RAM and already showed some lags, I decided to follow the recommendation - but also increase the amount of available RAM on the machines. The maximum amount of RAM configured/available for each node was 20G. Good enough to increase the 16G to 20G without downtime, but not enough for the planned 32G innodb_buffer_pool if it really gets used. So I decided to set the current amount of RAM for each node to 48G and added a buffer in the VM configuration up to 64G. This needed a cold restart of all nodes (and the backup host) - but everything went fine. Just remember that we already set the systemd timeout value "TimeoutSec=7200" for the mariadb service, as in worst case (more than initially 20, now 40min downtime of the mariadb service results in a SST restore via mariabackup for ~40G of database files), the mariadb service needs up to 1:30h to start. So we are now at 48G RAM for each node together with the following InnoDB settings in my.cnf: innodb_buffer_pool_size = 32G innodb_buffer_pool_instances = 12 innodb_doublewrite = 1 innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 0 innodb_flush_method = O_DIRECT innodb_log_buffer_size = 8M innodb_log_file_size = 4G innodb_lock_wait_timeout = 240 innodb_print_all_deadlocks = 1 innodb_io_capacity = 2000 innodb_read_io_threads = 64 innodb_thread_concurrency = 0 innodb_write_io_threads = 64 Thanks for reading so far :-) From my side, there are currently some todo's and questions left: * there is a database called "database" on each node, which does not contain any tables but instead a file called db.opt with: default-character-set=utf8mb4 default-collation=utf8mb4_general_ci as content. Does anyone know why this is there? * Some wiki databases have 3 different collations and 2 different engines defined now. Note: interestingly not all of them. Example: [!!] 3 different collations for database wiki_it [!!] 2 different engines for database wiki_it [!!] wiki_it table column(s) has several charsets defined for all text like column(s). [!!] wiki_it table column(s) has several collations defined for all text like column(s). [!!] There are 2 storage engines. Be careful. Affected wiki databases: wiki_es, wiki_it, wiki_nl, wiki_pt, wiki_old_de, wiki_zh, wiki_en and: webforums When I migrated the wiki databases, this was not the case. * Some (äh: 8944!) tables also have wrong types set for some fields. * Also very important: a lot of tables don't have primary keys. This slows down the synchronization between the nodes. I will see if there are some guidelines for the mediawiki databases to add indexes to these tables - but some webforums tables also don't have primary keys. @Per: maybe you can check this for vB? * On the opposite side, there are ~20 unused indexes. I will remove them next week. I will start with some "ALTER TABLE" operations during the next days, to fix some of the settings above. Depending on the size of the tables, this might lead to some (hopefully short) locks of the databases. With kind regards, Lars -- [1]: https://dba.stackexchange.com/questions/27328/how-large-should-be-mysql-inn… -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

3 3

Re: [heroes] Did someone/something touch gcc.opensuse.org?
by Michael Matz 23 Mar '20

23 Mar '20

Hello Lars, On Wed, 18 Mar 2020, Lars Vogdt wrote: > While... on the other hand... now that we have a prove of concept that > we can upgrade tumbleweed machines... If you don't mind, I want to put > this machine on a list of "run zypper dup once a month". If we check > right afterward that everything still works as expected (=> Monitoring: > just tell me what we should check) we should be ready to go. Worst > thing would be that we might loose a few hours, if we notice early > enough that something is broken and we need to revert to a > former snapshot state. The two important "services" on that would be a simple HTTPS check on some random URL, let's say https://gcc.opensuse.org/ (better would be https://gcc.opensuse.org/gcc-old/SPEC/CINT/sb-czerny-head-64-2006/recent.ht… but the 'gcc-old' part at least changed once over the last 15 years, so it's only semi-stable), and a check that would verify rsync via ssh access (that is what got broken, the webserver continued to work), from inside the SUSE network. I don't know if you have the capability to do such checking. (I could add a ssh key for that purpose). Ciao, Michael. -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

3 4

[heroes] Disabled central syslog server in favor of graylog.opensuse.org
by Lars Vogdt 22 Mar '20

22 Mar '20

JFYI: for the moment, I disabled the central syslog server in favor of the new graylog instance. No need to change anything on the host side: the syslog server is forwarding the traffic directly into the graylog queue. This means that the syslog server does NOT keep a copy of the remote logs on disk any longer. Instead, everything is pushed directly into a elasticsearch database for further processing. This in turn means that the old monitoring check for outdated remote logs is turned of. In the last weeks, I was working more on checking why a specific host was not sending logs any longer - and often enough, these hosts just had nothing to say (as they are planned replacements that are not active, yet). In the end, I was becoming too lazy to reconfigure that check again and again - so at least I am not sad about loosing this check. For those, who want to get a look at the new frontend for log files: https://graylog.opensuse.org/ Note: a openSUSE heroes LDAP account (FreeIPA) is needed to log in. A nice starting point might be the good documentation at: https://docs.graylog.org/en/3.2/pages/queries.html Regards, Lars -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] Status report: matomo
by Lars Vogdt 22 Mar '20

22 Mar '20

Friday/Saturday, I worked a bit on the matomo package in network:utilities and switched our productive matomo instance from pure mod_php to php-fpm. This gave some performance boost for the analytic section (I did not measure the time for the pure matomo.php script execution - so this is just an improvement for the guys looking at the statistics). For those with a log in, please use https://beans.opensuse.org/matomo/ in the future. I will keep the old https://beans.opensuse.org/piwik/ URL for some time, but depending on the usage, this might get deprecated/removed in the future. (It's just an alias anyway, but, hey: keep your environment clean starts with these little details ;-) Regards, Lars -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] http(s)://opensuse.org redirect moved
by Christian Boltz 22 Mar '20

22 Mar '20

Hello, FYI: http://opensuse.org and https://opensuse.org redirect to www.o.o. Until now, this redirect was done by a server (or cluster of servers) in Provo, which sadly broke today: https://progress.opensuse.org/issues/64722 Needless to say that we have some experience with opening tickets in Provo, therefore I just took the quick route and changed the DNS entry - the opensuse.org redirect now gets done by anna/elsa in Nuremberg. Possible risk: /openid - but according to the IRC discussion, that risk should be quite small. And even if I broke it, it would still be better than having http://opensuse.org broken ;-) Before: # host opensuse.org opensuse.org has address 130.57.66.19 opensuse.org has IPv6 address 2620:113:80c0:8::19 opensuse.org mail is handled by 42 mx1.suse.de. opensuse.org mail is handled by 42 mx2.suse.de. After: host opensuse.org opensuse.org has address 195.135.221.140 opensuse.org has IPv6 address 2620:113:80c0:8::16 opensuse.org mail is handled by 42 mx2.suse.de. opensuse.org mail is handled by 42 mx1.suse.de. The IPs are those of proxy.o.o/redirector.o.o - CNAME doesn't allow other entries, and we need the MX entries. Regards, Christian Boltz -- Not mentioned in the `known features'-list so far... [found on https://bugzilla.novell.com/show_bug.cgi?id=152068] -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] forum.infra.opensuse.org
by Per Jessen 21 Mar '20

21 Mar '20

For those with VPN access: http://forum.infra.opensuse.org Running on SLES12SP5 - Lars installed a VM for me yesterday. This is still vBulletin 4.2.2p4, raw copy from Provo. It should also be available at, may or may not quite work: https://forums-nbg.opensuse.org My own view of the above is screwed up due to a baseurl = http://forums-nbg.opensuse.org - yours might work. There is some caching going on, somewhere. I have not done anything to the database except import it, including whatever character set problems it contains. Next thing to sort out - authentication. iChain anyone? -- Per Jessen, Zürich (9.4°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

3 5