I think, this was the secret. https://github.com/uyuni-project/uyuni/blob/3f25c01103407b6b048987bf1460ca03... web_session_secret_1 Robert Am 9. Apr. 2022, 07:44, um 07:44, Robert Paschedag <robert.paschedag@web.de> schrieb:

Hi Paul,

I apologize....I just switched job and do not have access to any uyuni installation anymore. So trying to get it from my mind.

I also had 2 times those strange issues with cobbler.

I somehow expect, that you also might get "server error" when you try to view "kickstart profiles"?

First time, cobbler issue was caused by an SSL verification issue. While using company PKI certs, I forgot to add the root or intermediate cert into the "trusted certs" on the uyuni server itself. So when the uyuni server tried to do a SSL connection to "itself" (to the cobbler service by using the fqdn from the cobbler config), the verification failed... Silently!!

The second issue with cobbler, was that the password for cobbler was somehow wrong. The password in the database did not match the one within the rhn.conf.

I hope I remember correctly... the "password" from something like "web_service_1"? (in rhn.conf) is used by the code to access cobbler.

Once I changed the string in rhn.conf to match the one within the db (and restart of all services), the issue was gone.

Don't know the table within the DB anymore. Sorry.

Hope, this might help you to find and fix the issue.

Robert

...

P.S.

We’re seeing similar errors in the taskomatic log

2022-04-08 12:44:00,035 [DefaultQuartzScheduler_Worker-12] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - re-throwing exception since we havent yet. 2022-04-08 12:44:00,035 [DefaultQuartzScheduler_Worker-12] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - Executing a task

Am 8. Apr. 2022, 21:49, um 21:49, Paul-Andre Panon <ppanon@sierrawireless.com> schrieb: threw

...

an exception: com.redhat.rhn.manager.kickstart.cobbler.NoCobblerTokenException 2022-04-08 12:44:00,035 [DefaultQuartzScheduler_Worker-12] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - Message: We had an error trying to login. 2022-04-08 12:44:00,035 [DefaultQuartzScheduler_Worker-12] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - Cause: redstone.xmlrpc.XmlRpcFault: <class 'cobbler.cexceptions.CX'>:'login failed (taskomatic_user)' 2022-04-08 12:44:00,035 [DefaultQuartzScheduler_Worker-12] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - Stack trace:com.redhat.rhn.manager.kickstart.cobbler.NoCobblerTokenException: We had an error trying to login. at com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:56) at com.redhat.rhn.frontend.integration.IntegrationService.authorize(IntegrationService.java:113) at com.redhat.rhn.frontend.integration.IntegrationService.getAuthToken(IntegrationService.java:66) at com.redhat.rhn.manager.kickstart.cobbler.CobblerCommand.<init>(CobblerCommand.java:60) at com.redhat.rhn.manager.kickstart.cobbler.CobblerCommand.<init>(CobblerCommand.java:81) at com.redhat.rhn.manager.kickstart.cobbler.CobblerDistroSyncCommand.<init>(CobblerDistroSyncCommand.java:47) at com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:85) at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:89) at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:158) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) Caused by: redstone.xmlrpc.XmlRpcFault: <class 'cobbler.cexceptions.CX'>:'login failed (taskomatic_user)' at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:444) at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165) at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:69) at com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:51) ... 10 more

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

Thanks for your suggestions Robert. It turns out that my co-worker had changed the cobbler configuration and I wasn't informed. When he rolled back his changes it fixed things. I am quite surprised that cobbler would be involved in deleting a system. Cheers, Paul-Andre Panon From: Robert Paschedag <robert.paschedag@web.de> Sent: Saturday, April 9, 2022 5:48 AM To: Victor Zhestkov via Uyuni Users <users@lists.uyuni-project.org> Subject: Re: Problem deleting systems from Uyuni 2022.03 (and probably 2022.02 as well before upgrading) See this message (and following) on gitter https://gitter.im/uyuni-project/users?at=604b5f4dd1aee44e2de020c1<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitter.im%2Fuyuni-project%2Fusers%3Fat%3D604b5f4dd1aee44e2de020c1&data=04%7C01%7Cppanon%40sierrawireless.com%7Cb8807a730f424f53397908da1a27287c%7C08059a4c248643dd89e33a747e0dcbe8%7C0%7C0%7C637851052767086469%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fpyvk8hpxM1kGba5oPcstqbyeia%2FtkjTqNp0UlTC9OQ%3D&reserved=0> Robert Am 9. Apr. 2022, um 08:03, Robert Paschedag <robert.paschedag@web.de<mailto:robert.paschedag@web.de>> schrieb: I think, this was the secret. https://github.com/uyuni-project/uyuni/blob/3f25c01103407b6b048987bf1460ca031f6f2b66/python/spacewalk/server/rhn.conf.sample#L74<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fuyuni-project%2Fuyuni%2Fblob%2F3f25c01103407b6b048987bf1460ca031f6f2b66%2Fpython%2Fspacewalk%2Fserver%2Frhn.conf.sample%23L74&data=04%7C01%7Cppanon%40sierrawireless.com%7Cb8807a730f424f53397908da1a27287c%7C08059a4c248643dd89e33a747e0dcbe8%7C0%7C0%7C637851052767086469%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qAfBRz9748c1z%2Bvk10OPho%2Fs39OGTMFJah9GTBxW8AI%3D&reserved=0> web_session_secret_1 Robert Am 9. Apr. 2022, um 07:44, Robert Paschedag < robert.paschedag@web.de<mailto:robert.paschedag@web.de>> schrieb: Hi Paul, I apologize....I just switched job and do not have access to any uyuni installation anymore. So trying to get it from my mind. I also had 2 times those strange issues with cobbler. I somehow expect, that you also might get "server error" when you try to view "kickstart profiles"? First time, cobbler issue was caused by an SSL verification issue. While using company PKI certs, I forgot to add the root or intermediate cert into the "trusted certs" on the uyuni server itself. So when the uyuni server tried to do a SSL connection to "itself" (to the cobbler service by using the fqdn from the cobbler config), the verification failed... Silently!! The second issue with cobbler, was that the password for cobbler was somehow wrong. The password in the database did not match the one within the rhn.conf. I hope I remember correctly... the "password" from something like "web_service_1"? (in rhn.conf) is used by the code to access cobbler. Once I changed the string in rhn.conf to match the one within the db (and restart of all services), the issue was gone. Don't know the table within the DB anymore. Sorry. Hope, this might help you to find and fix the issue. Robert Am 8. Apr. 2022, um 21:49, Paul-Andre Panon < ppanon@sierrawireless.com<mailto:ppanon@sierrawireless.com>> schrieb: P.S. We're seeing similar errors in the taskomatic log 2022-04-08 12:44:00,035 [DefaultQuartzScheduler_Worker-12] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - re-throwing exception since we havent yet. 2022-04-08 12:44:00,035 [DefaultQuartzScheduler_Worker-12] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - Executing a task threw an exception: com.redhat.rhn.manager.kickstart.cobbler.NoCobblerTokenException 2022-04-08 12:44:00,035 [DefaultQuartzScheduler_Worker-12] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - Message: We had an error trying to login. 2022-04-08 12:44:00,035 [DefaultQuartzScheduler_Worker-12] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - Cause: redstone.xmlrpc.XmlRpcFault: <class 'cobbler.cexceptions.CX'>:'login failed (taskomatic_user)' 2022-04-08 12:44:00,035 [DefaultQuartzScheduler_Worker-12] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - Stack trace:com.redhat.rhn.manager.kickstart.cobbler.NoCobblerTokenException: We had an error trying to login. at com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:56) at com.redhat.rhn.frontend.integration.IntegrationService.authorize(IntegrationService.java:113) at com.redhat.rhn.frontend.integration.IntegrationService.getAuthToken(IntegrationService.java:66) at com.redhat.rhn.manager.kickstart.cobbler.CobblerCommand.<init>(CobblerCommand.java:60) at com.redhat.rhn.manager.kickstart.cobbler.CobblerCommand.<init>(CobblerCommand.java:81) at com.redhat.rhn.manager.kickstart.cobbler.CobblerDistroSyncCommand.<init>(CobblerDistroSyncCommand.java:47) at com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:85) at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:89) at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:158) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) Caused by: redstone.xmlrpc.XmlRpcFault: <class 'cobbler.cexceptions.CX'>:'login failed (taskomatic_user)' at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:444) at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165) at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:69) at com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:51) ... 10 more Sent from Mail<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986&data=04%7C01%7Cppanon%40sierrawireless.com%7Cb8807a730f424f53397908da1a27287c%7C08059a4c248643dd89e33a747e0dcbe8%7C0%7C0%7C637851052767086469%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0vF7p5ya5UfLpkTfs5ae%2FxlqVlI2pb3hLKO9mS3KzP8%3D&reserved=0> for Windows