Uyuni starvation, not checking registered client
Hi all, Our Uyuni infra is quite little with 90 registered clients and increasing, however since some days we notice like a starvation of taskomatic which don't seem to be able to launch SSHPush tasks and check Uyuni client status. All the registered servers are consequently shown as inactive ("System not checking in with Uyuni"). Important perhaps to mention that "push via SSH" is the registration method used for all the registered servers ( = no Salt minion installed). The only one servers not affected by this issue are the Uyuni proxies which are registered with salt-minion. We can repetitively see in the taskomatik logs this: 2020-11-22 10:21:00,500 [DefaultQuartzScheduler_Worker-88] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping. 2020-11-22 10:22:00,327 [DefaultQuartzScheduler_Worker-22] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping. 2020-11-22 10:23:00,173 [DefaultQuartzScheduler_Worker-92] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping. No more info in any logs, and repositories sync is still working without issue though. Seems so that the SSHPush tasks queue is blocked at some point ? A restart of taskomatic temporarily solve the issue. I have tried to follow this https://www.uyuni-project.org/uyuni-docs/uyuni/large-deployments/tuning.html, but well, from what mentioned the doc, it should not be necessary in my case ("Tuning is not required on installations of fewer than 1000 clients. Do not perform these instructions on small or medium scale installations."), and no luck with it anyway. Any ideas ? Somebody who experienced this same issue ? Philippe. Philippe Bidault | Unix Engineer | Getronics ________________________________ M. 34617301667 | E. Philippe.Bidault@Getronics.com | W. www.getronics.comhttps://www.getronics.com [cid:3-lines_7512e235-163a-4853-8e4f-b0d4fc62ce73.png]https://getronics.com/ Getronics CMC Service Desk Iberia S.L - VAT No:S.L.: B66686262. Registered Office - Getronics CMC Service Desk Iberia S.L, C/Rosselloi, Porcel, 21 planta 11, 08016 Barcelona, Spain. The information transmitted is intended only for use by the addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. Thank you. Legal disclaimer: http://www.getronics.com/legal/
Looks strange indeed. 90 clients is not a big deployment. Are you on 2020.09 + the patch for the CVEs that we announced one week ago? Can you provide the output of `zypper info salt-master` and `rpm -q -- changelog salt|head -n 50`? On domingo, 22 de noviembre de 2020 12:04:31 (CET) Bidault, Philippe wrote:
Hi all,
Our Uyuni infra is quite little with 90 registered clients and increasing, however since some days we notice like a starvation of taskomatic which don't seem to be able to launch SSHPush tasks and check Uyuni client status. All the registered servers are consequently shown as inactive ("System not checking in with Uyuni"). Important perhaps to mention that "push via SSH" is the registration method used for all the registered servers ( = no Salt minion installed). The only one servers not affected by this issue are the Uyuni proxies which are registered with salt-minion.
We can repetitively see in the taskomatik logs this:
2020-11-22 10:21:00,500 [DefaultQuartzScheduler_Worker-88] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping. 2020-11-22 10:22:00,327 [DefaultQuartzScheduler_Worker-22] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping. 2020-11-22 10:23:00,173 [DefaultQuartzScheduler_Worker-92] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping.
No more info in any logs, and repositories sync is still working without issue though. Seems so that the SSHPush tasks queue is blocked at some point ? A restart of taskomatic temporarily solve the issue.
I have tried to follow this https://www.uyuni-project.org/uyuni-docs/uyuni/large-deployments/tuning.htm l, but well, from what mentioned the doc, it should not be necessary in my case ("Tuning is not required on installations of fewer than 1000 clients. Do not perform these instructions on small or medium scale installations."), and no luck with it anyway.
Any ideas ? Somebody who experienced this same issue ?
Philippe.
Philippe Bidault | Unix Engineer | Getronics ________________________________ M. 34617301667 | E. Philippe.Bidault@Getronics.com | W. www.getronics.comhttps://www.getronics.com
[cid:3-lines_7512e235-163a-4853-8e4f-b0d4fc62ce73.png]<https://getronics.com />
Getronics CMC Service Desk Iberia S.L - VAT No:S.L.: B66686262. Registered Office - Getronics CMC Service Desk Iberia S.L, C/Rosselloi, Porcel, 21 planta 11, 08016 Barcelona, Spain.
The information transmitted is intended only for use by the addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. Thank you. Legal disclaimer: http://www.getronics.com/legal/
-- Julio González Gil Release Engineer, SUSE Manager and Uyuni jgonzalez@suse.com
Uyuni 2020.09 installed, but without the last CVE patch:
# zypper info salt-master
Loading repository data...
Warning: Repository 'Main Update Repository' appears to be outdated. Consider using a different mirror or server.
Reading installed packages...
Information for package salt-master:
------------------------------------
Repository : Main Update Repository
Name : salt-master
Version : 3000-lp152.3.9.1
Arch : x86_64
Vendor : openSUSE
Installed Size : 3.0 MiB
Installed : Yes (automatically)
Status : up-to-date
Source package : salt-3000-lp152.3.9.1.src
Summary : The management component of Saltstack with zmq protocol supported
Description :
The Salt master is the central server to which all minions connect.
Enabled commands to remote systems to be called in parallel rather
than serially.
uyuni:~ # rpm -q --changelog salt | head -50
* Wed Aug 12 2020 Pablo Suárez Hernández
Hi all,
Our Uyuni infra is quite little with 90 registered clients and increasing, however since some days we notice like a starvation of taskomatic which don't seem to be able to launch SSHPush tasks and check Uyuni client status. All the registered servers are consequently shown as inactive ("System not checking in with Uyuni"). Important perhaps to mention that "push via SSH" is the registration method used for all the registered servers ( = no Salt minion installed). The only one servers not affected by this issue are the Uyuni proxies which are registered with salt-minion.
We can repetitively see in the taskomatik logs this:
2020-11-22 10:21:00,500 [DefaultQuartzScheduler_Worker-88] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping. 2020-11-22 10:22:00,327 [DefaultQuartzScheduler_Worker-22] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping. 2020-11-22 10:23:00,173 [DefaultQuartzScheduler_Worker-92] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping.
No more info in any logs, and repositories sync is still working without issue though. Seems so that the SSHPush tasks queue is blocked at some point ? A restart of taskomatic temporarily solve the issue.
I have tried to follow this https://www.uyuni-project.org/uyuni-docs/uyuni/large-deployments/tunin g.htm l, but well, from what mentioned the doc, it should not be necessary in my case ("Tuning is not required on installations of fewer than 1000 clients. Do not perform these instructions on small or medium scale installations."), and no luck with it anyway.
Any ideas ? Somebody who experienced this same issue ?
Philippe.
Philippe Bidault | Unix Engineer | Getronics ________________________________ M. 34617301667 | E. Philippe.Bidault@Getronics.com | W. www.getronics.comhttps://www.getronics.com
[cid:3-lines_7512e235-163a-4853-8e4f-b0d4fc62ce73.png]<https://getroni cs.com />
Getronics CMC Service Desk Iberia S.L - VAT No:S.L.: B66686262. Registered Office - Getronics CMC Service Desk Iberia S.L, C/Rosselloi, Porcel, 21 planta 11, 08016 Barcelona, Spain.
The information transmitted is intended only for use by the addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. Thank you. Legal disclaimer: http://www.getronics.com/legal/
-- Julio González Gil Release Engineer, SUSE Manager and Uyuni jgonzalez@suse.com
Hi all,
Not sure if I am alone with this, but I am still stuck with this and have opened an issue:
https://github.com/uyuni-project/uyuni/issues/3182
Philippe.
-----Original Message-----
From: Bidault, Philippe
Hi all,
Our Uyuni infra is quite little with 90 registered clients and increasing, however since some days we notice like a starvation of taskomatic which don't seem to be able to launch SSHPush tasks and check Uyuni client status. All the registered servers are consequently shown as inactive ("System not checking in with Uyuni"). Important perhaps to mention that "push via SSH" is the registration method used for all the registered servers ( = no Salt minion installed). The only one servers not affected by this issue are the Uyuni proxies which are registered with salt-minion.
We can repetitively see in the taskomatik logs this:
2020-11-22 10:21:00,500 [DefaultQuartzScheduler_Worker-88] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping. 2020-11-22 10:22:00,327 [DefaultQuartzScheduler_Worker-22] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping. 2020-11-22 10:23:00,173 [DefaultQuartzScheduler_Worker-92] WARN com.redhat.rhn.taskomatic.task.SSHPush - Maximum number of workers already put ... skipping.
No more info in any logs, and repositories sync is still working without issue though. Seems so that the SSHPush tasks queue is blocked at some point ? A restart of taskomatic temporarily solve the issue.
I have tried to follow this https://www.uyuni-project.org/uyuni-docs/uyuni/large-deployments/tunin g.htm l, but well, from what mentioned the doc, it should not be necessary in my case ("Tuning is not required on installations of fewer than 1000 clients. Do not perform these instructions on small or medium scale installations."), and no luck with it anyway.
Any ideas ? Somebody who experienced this same issue ?
Philippe.
Philippe Bidault | Unix Engineer | Getronics ________________________________ M. 34617301667 | E. Philippe.Bidault@Getronics.com | W. www.getronics.comhttps://www.getronics.com
[cid:3-lines_7512e235-163a-4853-8e4f-b0d4fc62ce73.png]<https://getroni cs.com />
Getronics CMC Service Desk Iberia S.L - VAT No:S.L.: B66686262. Registered Office - Getronics CMC Service Desk Iberia S.L, C/Rosselloi, Porcel, 21 planta 11, 08016 Barcelona, Spain.
The information transmitted is intended only for use by the addressee and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of it, or the taking of any action in reliance upon this information by persons and/or entities other than the intended recipient is prohibited. If you received this in error, please inform the sender and/or addressee immediately and delete the material. Thank you. Legal disclaimer: http://www.getronics.com/legal/
-- Julio González Gil Release Engineer, SUSE Manager and Uyuni jgonzalez@suse.com _______________________________________________ Uyuni Users mailing list -- users@lists.uyuni-project.org To unsubscribe, email users-leave@lists.uyuni-project.org List Netiquette: https://en.opensuse.org/openSUSE:Mailing_list_netiquette List Archives: https://lists.opensuse.org/archives/list/users@lists.uyuni-project.org
participants (2)
-
Bidault, Philippe -
Julio González Gil