March 2020 - Heroes - openSUSE Mailing Lists

[heroes] Chameleon dev branch at static.opensuse.org
by Guo Yunhe 18 Apr '20

18 Apr '20

Hi, We are changing the development model of Chameleon so it is more reliable. https://github.com/openSUSE/chameleon/issues/18 The master branch (https://static.opensuse.org/chameleon/) is the stable branch, well tested and won't be updated frequently. Used by all production sites. The dev branch, which we would like to deploy at https://static.opensuse.org/chameleon-dev/ , is the unstable version for testing and can break now and then. Used for testing sites and local development environment. Need your help to set up the cron job to pull dev branch. Thanks! -- Guo Yunhe / @guoyunhe / guoyunhe.me -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

2 3

[heroes] Question about download.o.o stats
by Marco Varlese 17 Apr '20

17 Apr '20

Hello everybody, I have a question about download.o.o statistics. So the question is: Is there anyway for our current environment to know how many times *any* file is being downloaded? I am asking because I have created some ad-hoc virtual-machine appliances and it would be very beneficial to know how many times an appliance gets downloaded (and if possible dissect "new comers" vs "previous users"). The appliances live in one of the following folders, depending whether it is for Leap15.2 or Tumbleweed: 1) https://download.opensuse.org/repositories/Virtualization:/Appliances:/Imag… 2) https://download.opensuse.org/repositories/Virtualization:/Appliances:/Imag… and the file names follow the following convention: machinelearning-appliance.x86_64-* Beside, in general I would think that even knowing how many times RPMs are being downloaded (via zypper) could be beneficial to understand better end-users needs/trends. Is there anything that could be extrapolated from logs/metrics for statistics purposes as per my initial question? Thank you for your consideration. Best regards, -- Marco Varlese Architect Developer Technologies & AI/ML SUSE Labs SUSE Software Solutions Italy S.r.l. (T) +39 02.947.570.06 (M) +39 345.591.51.01 (E) marco.varlese(a)suse.com (W) https://gotomeet.me/marcovarlese -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

3 10

[heroes] Mail setup
by Lars Vogdt 30 Mar '20

30 Mar '20

Hi @ll I finally want to start with the new Email setup for openSUSE - and I'm currently looking for volunteers... :-) Planned short term: * set SPF records for openSUSE domains * install mx1 and mx2.opensuse.org as incoming servers ** use postfix and rspamd ** integrate clamav (and reject messages seen as spam directly) ** integrate the alias table for members Planned mid term: * enable DCIM on all outgoing mail servers Something I missed? Regards, Lars -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

5 6

[heroes] Galera changes
by Lars Vogdt 27 Mar '20

27 Mar '20

Hi After the latest changes and new databases (hello, forums! ;-), I decided that it's time to check the current situation of our database cluster again. You might know mysqltuner.pl and tuning-primer.sh from the mysql-tuning-scripts package: the two tools analyze a running mysql/mariadb and galera cluster and print out statistics and recommendations (and even more, if you call them with the right options ;-). Both tools recommended to increase the "InnoDB buffer pool size". I cross checked this with an SQL statement stolen from stackexchange[1]: => result of all tools: Recommended_InnoDB_Buffer_Pool_Size = 32G As the nodes inside the cluster just had 16G RAM and already showed some lags, I decided to follow the recommendation - but also increase the amount of available RAM on the machines. The maximum amount of RAM configured/available for each node was 20G. Good enough to increase the 16G to 20G without downtime, but not enough for the planned 32G innodb_buffer_pool if it really gets used. So I decided to set the current amount of RAM for each node to 48G and added a buffer in the VM configuration up to 64G. This needed a cold restart of all nodes (and the backup host) - but everything went fine. Just remember that we already set the systemd timeout value "TimeoutSec=7200" for the mariadb service, as in worst case (more than initially 20, now 40min downtime of the mariadb service results in a SST restore via mariabackup for ~40G of database files), the mariadb service needs up to 1:30h to start. So we are now at 48G RAM for each node together with the following InnoDB settings in my.cnf: innodb_buffer_pool_size = 32G innodb_buffer_pool_instances = 12 innodb_doublewrite = 1 innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 0 innodb_flush_method = O_DIRECT innodb_log_buffer_size = 8M innodb_log_file_size = 4G innodb_lock_wait_timeout = 240 innodb_print_all_deadlocks = 1 innodb_io_capacity = 2000 innodb_read_io_threads = 64 innodb_thread_concurrency = 0 innodb_write_io_threads = 64 Thanks for reading so far :-) From my side, there are currently some todo's and questions left: * there is a database called "database" on each node, which does not contain any tables but instead a file called db.opt with: default-character-set=utf8mb4 default-collation=utf8mb4_general_ci as content. Does anyone know why this is there? * Some wiki databases have 3 different collations and 2 different engines defined now. Note: interestingly not all of them. Example: [!!] 3 different collations for database wiki_it [!!] 2 different engines for database wiki_it [!!] wiki_it table column(s) has several charsets defined for all text like column(s). [!!] wiki_it table column(s) has several collations defined for all text like column(s). [!!] There are 2 storage engines. Be careful. Affected wiki databases: wiki_es, wiki_it, wiki_nl, wiki_pt, wiki_old_de, wiki_zh, wiki_en and: webforums When I migrated the wiki databases, this was not the case. * Some (äh: 8944!) tables also have wrong types set for some fields. * Also very important: a lot of tables don't have primary keys. This slows down the synchronization between the nodes. I will see if there are some guidelines for the mediawiki databases to add indexes to these tables - but some webforums tables also don't have primary keys. @Per: maybe you can check this for vB? * On the opposite side, there are ~20 unused indexes. I will remove them next week. I will start with some "ALTER TABLE" operations during the next days, to fix some of the settings above. Depending on the size of the tables, this might lead to some (hopefully short) locks of the databases. With kind regards, Lars -- [1]: https://dba.stackexchange.com/questions/27328/how-large-should-be-mysql-inn… -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

3 3

Re: [heroes] Did someone/something touch gcc.opensuse.org?
by Michael Matz 23 Mar '20

23 Mar '20

Hello Lars, On Wed, 18 Mar 2020, Lars Vogdt wrote: > While... on the other hand... now that we have a prove of concept that > we can upgrade tumbleweed machines... If you don't mind, I want to put > this machine on a list of "run zypper dup once a month". If we check > right afterward that everything still works as expected (=> Monitoring: > just tell me what we should check) we should be ready to go. Worst > thing would be that we might loose a few hours, if we notice early > enough that something is broken and we need to revert to a > former snapshot state. The two important "services" on that would be a simple HTTPS check on some random URL, let's say https://gcc.opensuse.org/ (better would be https://gcc.opensuse.org/gcc-old/SPEC/CINT/sb-czerny-head-64-2006/recent.ht… but the 'gcc-old' part at least changed once over the last 15 years, so it's only semi-stable), and a check that would verify rsync via ssh access (that is what got broken, the webserver continued to work), from inside the SUSE network. I don't know if you have the capability to do such checking. (I could add a ssh key for that purpose). Ciao, Michael. -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

3 4

[heroes] Disabled central syslog server in favor of graylog.opensuse.org
by Lars Vogdt 22 Mar '20

22 Mar '20

JFYI: for the moment, I disabled the central syslog server in favor of the new graylog instance. No need to change anything on the host side: the syslog server is forwarding the traffic directly into the graylog queue. This means that the syslog server does NOT keep a copy of the remote logs on disk any longer. Instead, everything is pushed directly into a elasticsearch database for further processing. This in turn means that the old monitoring check for outdated remote logs is turned of. In the last weeks, I was working more on checking why a specific host was not sending logs any longer - and often enough, these hosts just had nothing to say (as they are planned replacements that are not active, yet). In the end, I was becoming too lazy to reconfigure that check again and again - so at least I am not sad about loosing this check. For those, who want to get a look at the new frontend for log files: https://graylog.opensuse.org/ Note: a openSUSE heroes LDAP account (FreeIPA) is needed to log in. A nice starting point might be the good documentation at: https://docs.graylog.org/en/3.2/pages/queries.html Regards, Lars -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] Status report: matomo
by Lars Vogdt 22 Mar '20

22 Mar '20

Friday/Saturday, I worked a bit on the matomo package in network:utilities and switched our productive matomo instance from pure mod_php to php-fpm. This gave some performance boost for the analytic section (I did not measure the time for the pure matomo.php script execution - so this is just an improvement for the guys looking at the statistics). For those with a log in, please use https://beans.opensuse.org/matomo/ in the future. I will keep the old https://beans.opensuse.org/piwik/ URL for some time, but depending on the usage, this might get deprecated/removed in the future. (It's just an alias anyway, but, hey: keep your environment clean starts with these little details ;-) Regards, Lars -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] http(s)://opensuse.org redirect moved
by Christian Boltz 22 Mar '20

22 Mar '20

Hello, FYI: http://opensuse.org and https://opensuse.org redirect to www.o.o. Until now, this redirect was done by a server (or cluster of servers) in Provo, which sadly broke today: https://progress.opensuse.org/issues/64722 Needless to say that we have some experience with opening tickets in Provo, therefore I just took the quick route and changed the DNS entry - the opensuse.org redirect now gets done by anna/elsa in Nuremberg. Possible risk: /openid - but according to the IRC discussion, that risk should be quite small. And even if I broke it, it would still be better than having http://opensuse.org broken ;-) Before: # host opensuse.org opensuse.org has address 130.57.66.19 opensuse.org has IPv6 address 2620:113:80c0:8::19 opensuse.org mail is handled by 42 mx1.suse.de. opensuse.org mail is handled by 42 mx2.suse.de. After: host opensuse.org opensuse.org has address 195.135.221.140 opensuse.org has IPv6 address 2620:113:80c0:8::16 opensuse.org mail is handled by 42 mx2.suse.de. opensuse.org mail is handled by 42 mx1.suse.de. The IPs are those of proxy.o.o/redirector.o.o - CNAME doesn't allow other entries, and we need the MX entries. Regards, Christian Boltz -- Not mentioned in the `known features'-list so far... [found on https://bugzilla.novell.com/show_bug.cgi?id=152068] -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] forum.infra.opensuse.org
by Per Jessen 21 Mar '20

21 Mar '20

For those with VPN access: http://forum.infra.opensuse.org Running on SLES12SP5 - Lars installed a VM for me yesterday. This is still vBulletin 4.2.2p4, raw copy from Provo. It should also be available at, may or may not quite work: https://forums-nbg.opensuse.org My own view of the above is screwed up due to a baseurl = http://forums-nbg.opensuse.org - yours might work. There is some caching going on, somewhere. I have not done anything to the database except import it, including whatever character set problems it contains. Next thing to sort out - authentication. iChain anyone? -- Per Jessen, Zürich (9.4°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

3 5

[heroes] Short status update
by Lars Vogdt 18 Mar '20

18 Mar '20

Hi Just if you wonder what happened during the last weeks.... 1) DNS Our FreeIPA instance is finally the one and only official instance for * opensuse.org * opensuse.de * opensuse.fr The connection to the MF-IT network and servers, as mentioned at https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/DNS is history. Only the DNS servers inside the heroes network are left and do what good DNS servers do: just work. Each server is answering ~15 requests per second, according to the statistics. ==> I just want to wait until next month, before I create an announcement that "openSUSE is now in Heroes hand" - might be that I need some help with the text and merge request... ;-) 2) Monitoring I started with https://monitor.opensuse.org/icingaweb2/ and connected it directly with the LDAP server. Means: people in the "monitoring-admins" group have full access, people in "monitoring-user" have normal rights. But please note that there are currently just 2 hosts monitored. The reason: the old icinga setup uses check_mk for autodiscovery and autogeneration of most checks. This is not possible any more with icinga2. I want to use Salt as replacement instead. Instead of "pnp4nagios", the new icinga2 uses Grafana: https://monitor.opensuse.org/grafana/d/YyV2BduWk/base-metrics?orgId=1&refre… It's the same instance which is also providing some graphs for: * our elasticsearch cluster for the wikis * the Galera cluster * the PostgreSQL cluster => https://monitor.opensuse.org/grafana/ This instance is also connected to our LDAP. Data sources are currently InfluxDB (for icinga2), PostgreSQL and Prometheus (for Galera). Instead of just storing the logs of our hosts on a hard drive and waiting if someone wants to have a look at them (which honestly more or less never happens), I decided to move forward and installed Graylog here: https://graylog.opensuse.org/ As - for example - https://graylog.opensuse.org/dashboards/5e6ea77657c155111a8fbd37 shows, this makes it much easier to get an overview of "what's currently going on" on our machines. The filtering and search functionality is IMHO way easier than the alternative ELK stack (which is also hardware to maintain as package). I did not setup big dashboards, yet, in the hope that some other volunteer steps in :-) Ah: obvious to say that Graylog also uses our LDAP and normally all monitoring-admins are also Graylog admins. But - as with Grafana - if you want more than your original rights, just ping. 3) Support I hope I did not forget/overlooked too many requests for additional resources or machines. Open (incl. "feedback") progress tickets are down to 142[1]. 4) Mirrors I tried to reduce the amount of messages sent to the admin-auto mailing list from provo-mirror.opensuse.org, olaf (our scanner) and pontifex (aka download.o.o) over the last weeks. This included some (small) fixes in some packages (did I tell anyone that I don't know python?). Overall just minor stuff. But if you wondered, you might know whom to ask now. 5) CaaSP As nobody was actively maintaining the CaaSP cluster any longer (since ~ half a year), I asked in https://progress.opensuse.org/issues/54977 for someone who wants to step in - but got no feedback. As nobody was loggin in to the machines as well and their content (with one exception below) was already migrated to other machines, I shut down the machines at 1st of March. The only real issue since then is that we lost our docker containers with the gitlab runners. I'm sorry for that, but these runners were outdated (42.3, anyone?) anyway. I discussed with Ricardo (our Gitlab Guru) and we decided to setup two independent machines just to host the runners (via docker containers) in the future (gitlab-runner{1,2}.infra.opensuse.org) This is the next TODO on the list - I'm currently just waiting for a time slot from Ricardo to finalize the setup. --- There is probably a bit more - but my brain is currently blocking my memory. I just want to give you a short status summary of topics I still remember. Stay healthy! Lars -- [1]: https://monitor.opensuse.org/pnp4nagios/graph?host=redmine.infra.opensuse.o… -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] Did someone/something touch gcc.opensuse.org?
by Michael Matz 18 Mar '20

18 Mar '20

Hello, because something changed on Friday, and now the libraries are in an inconsistent state, such that rsync isn't working anymore leading to a cron job of mine breaking. % ssh -p2271 gcc(a)gate.opensuse.org Last login: Wed Jun 27 13:30:30 2018 from 192.168.47.252 Have a lot of fun... sed: relocation error: /lib64/libacl.so.1: symbol getxattr, version ATTR_1.0 not defined in file libattr.so.1 with link time reference sed: relocation error: /lib64/libacl.so.1: symbol getxattr, version ATTR_1.0 not defined in file libattr.so.1 with link time reference gcc@gcc-stats:~> rsync --help rsync: relocation error: /lib64/libacl.so.1: symbol getxattr, version ATTR_1.0 not defined in file libattr.so.1 with link time reference gcc@gcc-stats:~> last gcc pts/0 192.168.47.252 Mon Mar 16 13:58 still logged in root pts/0 192.168.253.202 Fri Mar 13 19:28 - 19:31 (00:02) root pts/0 192.168.47.7 Fri Mar 13 19:25 - 19:27 (00:01) root pts/0 192.168.253.202 Thu Jan 2 23:22 - 23:24 (00:02) I will fix it by hand eventually (by hand because also rpm isn't working anymore), but I'd like to find out the cause for this. From 'last' it would seem something (semi-)automatic, and whatever it was it's failure mode isn't graceful. Ciao, Michael. -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

2 5

[heroes] (pontifex) Empty changes file for openSUSE Leap 15.2 - new snapshot notification is not being sent
by Lubos Kocman 17 Mar '20

17 Mar '20

Hello team, seems like that notification for last openSUSE 15.2 Beta snapshot was not sent to opensuse-factory@ Reason behind seems to be the empty changes file in http://download.opensuse.org/distribution/leap/15.2/iso/ This file is supposed to be populated by the OBS publisher script. Can you please have a look? Autobuild told me that they can't touch this as it's on the pontifex server. Thank you -- Best regards Luboš Kocman Release Manager openSUSE Leap SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nuremberg Germany (HRB 36809, AG Nürnberg) Managing Director: Felix Imendörffer -- Best regards Luboš Kocman Release Manager openSUSE Leap SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nuremberg Germany (HRB 36809, AG Nürnberg) Managing Director: Felix Imendörffer

2 2

[heroes] another attempt
by Per Jessen 15 Mar '20

15 Mar '20

All, with no real solution for the character set issue within reach, as a sort of last desperate way out, I have created a copy of the forums with the code+data from Provo. It took a little fiddling, but it seems to be running: http://osforums.jessen.ch (it was quicker for me to do it locally, and no need to bother Lars). It's running on Leap 42.3 with php5 and mariadb 10.2.31, login+register do not yet work, but it looks like everything is being displayed correctly. Which is a major step forward. I would suggest you take a look, see if you can spot any issues - I think some links may still refer to forums.o.o, I'm not sure. I think we should be able to run this on an SLES12 machine in Nürnberg, import a new database and then switch the forums over. -- Per Jessen, Zürich (15.6°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

5 6

[heroes] forums migration status
by Per Jessen 10 Mar '20

10 Mar '20

Good morning, I suppose I have _some_ progress to report - after a lot of trail&error and help and suggestions, at least the front page can be made to look correct. Unfortunately that is only half the problem. Here is what I did - Yesterday I ran a last ditch import, with no conversions, just the raw data from Provo in Latin1. I figured that had to work when it works in Provo. Maybe we can consider a correct move/conversion to UTF8 later on. Then I ran the upgrade - with the stand-alone database, this can now be done in less than three hours. With practice, it is also easily repeated. The process warned me "that I should probably set the client charset to utf8mb4", so I did. (mysqli property set in php, and also the default in mysql anyway). I started up the forums this morning, corrected a couple of settings and moved it to https://forums-nbg.o.o. First attempt - it was still showing what is clearly UTF8 characters on the front page. Easily spotted in some German text, and the links to external forums in Español and Français. I decided to try removing the client charset and just let it default (to whatever). The pages now look fine, but the links don't work. Switch back to client charset = utf8, and the links work, but the text is wrong. Change the client charset to "latin1" and the page looks fine, the links don't. I have been staring at this for three-four days, I am having trouble seeing the forest for the trees. I think the next step is to try out the scripts that Malcolm sent a link to. -- Per Jessen, Zürich (5.9°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

5 7

[heroes] Re: forums migration update [2]
by Per Jessen 09 Mar '20

09 Mar '20

Per Jessen wrote: > http://forum.infra.opensuse.org/ > Unfortunately you will only have access if you are on our vpn, I apologize. I have just this morning switched over to the publicly accessible address: https://forums-nbg.opensuse.org/ I have enabled a setting to support multiple domains, but I am not sure how well vBulletin really works with that, it seems a bit mixed up. -- Per Jessen, Zürich (6.7°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

3 4

[heroes] Wikis are down
by Guo Yunhe 06 Mar '20

06 Mar '20

All wiki sites show database errors. -- Guo Yunhe / @guoyunhe / guoyunhe.me -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

2 1

[heroes] forums migration - 504 gateway time-out ??
by Per Jessen 05 Mar '20

05 Mar '20

All, at one of the upgrade steps, I am getting stuck on the same one with this error - 504 Gateway Time-out Server didn't respond in time The upgrades steps are lots of database operations - is there a time-out somewhere ? Would the error above come from the proxy (mysql.infra) ? -- Per Jessen, Zürich (6.7°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 1

[heroes] Wildcard DNS entry *.opensuse.org
by Christian Boltz 05 Mar '20

05 Mar '20

Hello, it looks like someone added a wildcard DNS entry *.opensuse.org pointing to 130.57.66.19 Just curious - what's the reason for this, and where does this IP live? BTW: this IP has an interesting list of reverse DNS entries ;-) Regards, Christian Boltz -- I'm not out to destroy Microsoft. That will just be a completely unintentional side effect. [Linus Torvalds] -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

2 1

[heroes] Status report
by Per Jessen 03 Mar '20

03 Mar '20

As I will not make it on time for the meeting, maybe not at all - a quick update on the forums migration: Last week we finally managed to get a database export. Part of the delay was really my fault, not enough space on the upload filesystem (although a compressed export would have easily fitted ...). I have also agreed with Jared that he will make a new export whenever needed, so we can have the most current data for the finished migration. I have the newest vBulletin software, it is installed on http://forum.infra.o.o. The database is loaded on the galera cluster (thanks for your help Lars). I used innodb for all tables, I saw no complaints. So far for the good news, now the less good: - I am not yet able to actually run the vBulletin code - currently I get an HTTP 500, which I'm still trying to figure out. - charset. vBulletin recommends is UTF8mb4, but our database is in Latin1. I think(!) this should be simple to convert with ALTER. - stylesheets etc. Apparently this version of vBulletin uses a new setup, not compatible with the current one. I'd like some help here. I'm wondering if some of the current forum admins might want to lend a hand? - authentication. The 'vb_user' table holds 40501 rows. The current system is tied in to the SSO setup in Provo, I don't know how or if this table is used. - according to Marta, the contract with Provo expired on Feb29. MF-IT have agreed to keep running the forums for another week. In summary - if I can solve the HTTP 500 and the database conversion/upgrade works "just like that", the only issue is really authentication. -- Per Jessen, Zürich (6.8°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

4 12

[heroes] Need to skip the meeting tomorrow
by Lars Vogdt 02 Mar '20

02 Mar '20

I'm sorry, but I've a conflicting meeting AND a sick wife at home, so I have no chance to attend the meeting tomorrow. I will try to write down a work report until the meeting, but I can not really promise to get it done in time. Sorry for that. Lars -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] Reminder: heroes meeting on tuesday
by Christian Boltz 01 Mar '20

01 Mar '20

Hello, the next heroes meeting will be on tuesday (2020-03-03) at 20:00 CET / 19:00 UTC in the #opensuse-admin IRC channel. See https://progress.opensuse.org/issues/63172 for the usual topics, and feel free to add whatever you want to discuss. Regards, Christian Boltz -- <jjohansen> I just need to stop responding to cboltz <cboltz> no, but you should come up with things I don't notice ;-) <jjohansen> something you don't notice? Isn't that asking a bit much :) <cboltz> no, that's called "high quality code" or at least "high quality well-hidden bug" *g,d&r* * jjohansen fails to believe there is such a thing as a well-hidden bug for cboltz [from #apparmor] -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0