Hello All, can you assist me by guiding me in the correct direction? Yesterday I found out that all my clients (4 openSUSE 15.1) are not reporting for the past 3 days.I tried to refresh , but it didn't do anything .

Then I removed the VMs and tried to add them, but after accepting the salt key - the VM is not added as a system. I have checked the logs and I see a lot of errors , one of them is : 2019-07-09 16:26:24,719 [salt-event-thread-1] ERROR com.suse.manager.reactor.PGEventListener - Unexpected exception while executing a MessageAction com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at

Latest update:  removing the system, cleaning up system id, salt minion keys & id + removal of minion cache allowed me to bootstrap the machine again without this side effect. I will keep an eye on it. Best Regards, Strahil Nikolov В вторник, 9 юли 2019 г., 19:49:06 ч. Гринуич+3, Strahil Nikolov написа: Hi Chris, thanks for your hint. I have tested adding a fresh openSUSE 15.1 and everything is fine until I have noticed that every minute the Uyuni is doing package refresh (8 refresh per minute) and never stops. Running it manually passes in 3.22s . Now I have to find the reason behind this constant running of package refresh. Adding uyuni-devel list. Best Regards, Strahil Nikolov В вторник, 9 юли 2019 г., 17:47:16 ч. Гринуич+3, Cristian Gherman написа: I had to change the id of minions in something: fe-1.blabla.com  => fe-1-prod.blabla,com If I use previous names , they are not added to Uyuni any more. Cristian Gherman -------- Forwarded Message -------- Subject:     Re: [uyuni-users] need assistance with debugging bootstrap issues Date:     Tue, 9 Jul 2019 13:55:59 +0000 (UTC) From:     Strahil Nikolov To:     Cristian Gherman Hi Christian, thanks for the fast reply. Already done all of these. The new key is accepted , but the machines are not added. I even used "snapper" to rollback one of the VMs to openSUSE 15.0 in order to check bootstraping. The results are the same. Rebooting Uyuni doesn't solve the issue :) Any ideas appreciated. Best Regards, Strahil Nikolov В вторник, 9 юли 2019 г., 9:52:05 ч. Гринуич-4, Cristian Gherman написа: Hello, Can you try this: - remove minion key from master - stop the salt-minion on VM's - remove minion key from /etc/salt/pki/ - change the minion id in /etc/minion_id - change the machine id in /etc/machine-id with command systemd-machine-id-setup - start the salt-minion on VM's - accept new keys, check if minons is added on Uyuni This steps helped me with some stoborn minions some time ago. Cristian Gherman On 7/9/19 4:39 PM, Strahil Nikolov wrote: path $

at com.google.gson.Gson.fromJson(Gson.java:900) at com.google.gson.Gson.fromJson(Gson.java:953) at com.suse.manager.reactor.messaging.JobReturnEventMessageAction.lambda$execute$7(JobReturnEventMessageAction.java:128) at java.util.Optional.ifPresent(Optional.java:159) at com.suse.manager.reactor.messaging.JobReturnEventMessageAction.execute(JobReturnEventMessageAction.java:114) at com.suse.manager.reactor.PGEventListener.lambda$null$1(PGEventListener.java:74) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at com.suse.manager.reactor.PGEventListener.lambda$notify$2(PGEventListener.java:72) at java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:419) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at com.suse.manager.reactor.PGEventListener.notify(PGEventListener.java:69) at com.suse.salt.netapi.event.AbstractEventStream.notifyListeners(AbstractEventStream.java:64) at com.suse.manager.reactor.PGEventStream.lambda$processEvents$3(PGEventStream.java:147) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:352) at java.util.stream.Sink$ChainedReference.end(Sink.java:258) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at com.suse.manager.reactor.PGEventStream.processEvents(PGEventStream.java:141) at com.suse.manager.reactor.PGEventStream.lambda$null$0(PGEventStream.java:127) at com.redhat.rhn.frontend.events.TransactionHelper.run(TransactionHelper.java:62) at com.redhat.rhn.frontend.events.TransactionHelper.handlingTransaction(TransactionHelper.java:46) at com.suse.manager.reactor.PGEventStream.lambda$notification$2(PGEventStream.java:126) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at path $ at com.google.gson.internal.bind.JsonTreeReader.expect(JsonTreeReader.java:161) at com.google.gson.internal.bind.JsonTreeReader.beginObject(JsonTreeReader.java:87) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:183) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145) at com.google.gson.Gson.fromJson(Gson.java:888) ... 30 more 2019-07-09 16:26:24,719 [salt-event-thread-4] ERROR com.redhat.rhn.frontend.events.TransactionHelper - java.lang.ClassCastException 2019-07-09 16:26:24,719 [salt-event-thread-4] ERROR com.redhat.rhn.frontend.events.TransactionHelper - Additional Exception during Exception handling java.lang.ClassCastException 2019-07-09 16:26:24,720 [salt-event-thread-3] ERROR com.redhat.rhn.frontend.events.TransactionHelper - java.lang.ClassCastException 2019-07-09 16:26:24,719 [salt-event-thread-1] ERROR com.redhat.rhn.frontend.events.TransactionHelper - com.suse.manager.reactor.PGEventListenerException 2019-07-09 16:26:24,720 [salt-event-thread-5] ERROR com.redhat.rhn.frontend.events.TransactionHelper - com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected NUMBER but was BEGIN_OBJECT at path $.retcode

Any ideas how to debug this one? Which logs should I check ? Maybe someone has already had a similar situation...

Best Regards, Strahil Nikolov

-- To unsubscribe, e-mail: uyuni-users+unsubscribe@opensuse.org To contact the owner, e-mail: uyuni-users+owner@opensuse.org -- To unsubscribe, e-mail: uyuni-users+unsubscribe@opensuse.org To contact the owner, e-mail: uyuni-users+owner@opensuse.org

On July 13, 2019 1:27:25 PM GMT+03:00, Strahil Nikolov wrote:

I think I have narrowed the problem.

So far, I have reinstalled uyuni from scratch and I think the whole issue is related to the uyuni tools for openSUSE 15.1 . Bootstrapping openSUSE 15.0 is a piece of a cake, and while upgrading works I have noticed that: openSUSE 15.1 doesn't work nice in salt with the tools for 15.1  After upgrading the 15.0 to 15.1 (dev tools for 15.1) I have noticed a nasty bug where the "Package List Refresh scheduled by (none)" keeps repeating for ever (noticed the same on the old installation - more than 32000 pages of such events). The workaround is to schedule a package refresh from the web. Once it succeeds - the events stop. Fresh bootstraping 15.1 with the regular 15.0 tools causes problems with salt highstate. 

My guess is that the uyuni tools for 15.1 are either incompatible for 4.0.1 or there is a bug in another component in uyuni.

Note: I have noticed also conflicts between packages in epel and uyuni tools (we got the same issue for RHEL 7 + epel on our SUMA 3.2) for CentOS 7.

Best Regards, Strahil Nikolov

В вторник, 9 юли 2019 г., 20:03:03 ч. Гринуич+3, Strahil Nikolov написа:

Latest update:  removing the system, cleaning up system id, salt minion keys & id + removal of minion cache allowed me to bootstrap the machine again without this side effect.

I will keep an eye on it.

Best Regards, Strahil Nikolov

В вторник, 9 юли 2019 г., 19:49:06 ч. Гринуич+3, Strahil Nikolov написа:

Hi Chris,

thanks for your hint. I have tested adding a fresh openSUSE 15.1 and everything is fine until I have noticed that every minute the Uyuni is doing package refresh (8 refresh per minute) and never stops.

Running it manually passes in 3.22s . Now I have to find the reason behind this constant running of package refresh.

Adding uyuni-devel list.

Best Regards, Strahil Nikolov

В вторник, 9 юли 2019 г., 17:47:16 ч. Гринуич+3, Cristian Gherman написа:

I had to change the id of minions in something:

fe-1.blabla.com  => fe-1-prod.blabla,com

If I use previous names , they are not added to Uyuni any more.

Cristian Gherman

-------- Forwarded Message -------- Subject:     Re: [uyuni-users] need assistance with debugging bootstrap issues Date:     Tue, 9 Jul 2019 13:55:59 +0000 (UTC) From:     Strahil Nikolov To:     Cristian Gherman

Hi Christian,

thanks for the fast reply.

Already done all of these. The new key is accepted , but the machines are not added.

I even used "snapper" to rollback one of the VMs to openSUSE 15.0 in order to check bootstraping. The results are the same. Rebooting Uyuni doesn't solve the issue :)

Any ideas appreciated.

Best Regards, Strahil Nikolov

В вторник, 9 юли 2019 г., 9:52:05 ч. Гринуич-4, Cristian Gherman написа:

Hello,

Can you try this:

- remove minion key from master

- stop the salt-minion on VM's

- remove minion key from /etc/salt/pki/

- change the minion id in /etc/minion_id

- change the machine id in /etc/machine-id with command systemd-machine-id-setup

- start the salt-minion on VM's

- accept new keys, check if minons is added on Uyuni

This steps helped me with some stoborn minions some time ago.

Cristian Gherman

...

Hello All, can you assist me by guiding me in the correct direction? Yesterday I found out that all my clients (4 openSUSE 15.1) are not reporting for the past 3 days.I tried to refresh , but it didn't do anything .

Then I removed the VMs and tried to add them, but after accepting the salt key - the VM is not added as a system. I have checked the logs and I see a lot of errors , one of them is : 2019-07-09 16:26:24,719 [salt-event-thread-1] ERROR com.suse.manager.reactor.PGEventListener - Unexpected exception while executing a MessageAction com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at

On 7/9/19 4:39 PM, Strahil Nikolov wrote: path $

...

at com.google.gson.Gson.fromJson(Gson.java:900) at com.google.gson.Gson.fromJson(Gson.java:953) at com.suse.manager.reactor.messaging.JobReturnEventMessageAction.lambda$execute$7(JobReturnEventMessageAction.java:128) at java.util.Optional.ifPresent(Optional.java:159) at com.suse.manager.reactor.messaging.JobReturnEventMessageAction.execute(JobReturnEventMessageAction.java:114) at com.suse.manager.reactor.PGEventListener.lambda$null$1(PGEventListener.java:74) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at com.suse.manager.reactor.PGEventListener.lambda$notify$2(PGEventListener.java:72) at java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:419) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at com.suse.manager.reactor.PGEventListener.notify(PGEventListener.java:69) at com.suse.salt.netapi.event.AbstractEventStream.notifyListeners(AbstractEventStream.java:64) at com.suse.manager.reactor.PGEventStream.lambda$processEvents$3(PGEventStream.java:147) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:352) at java.util.stream.Sink$ChainedReference.end(Sink.java:258) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at com.suse.manager.reactor.PGEventStream.processEvents(PGEventStream.java:141) at com.suse.manager.reactor.PGEventStream.lambda$null$0(PGEventStream.java:127) at com.redhat.rhn.frontend.events.TransactionHelper.run(TransactionHelper.java:62) at com.redhat.rhn.frontend.events.TransactionHelper.handlingTransaction(TransactionHelper.java:46) at com.suse.manager.reactor.PGEventStream.lambda$notification$2(PGEventStream.java:126) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at path $ at com.google.gson.internal.bind.JsonTreeReader.expect(JsonTreeReader.java:161) at com.google.gson.internal.bind.JsonTreeReader.beginObject(JsonTreeReader.java:87) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:183) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145) at com.google.gson.Gson.fromJson(Gson.java:888) ... 30 more 2019-07-09 16:26:24,719 [salt-event-thread-4] ERROR com.redhat.rhn.frontend.events.TransactionHelper - java.lang.ClassCastException 2019-07-09 16:26:24,719 [salt-event-thread-4] ERROR com.redhat.rhn.frontend.events.TransactionHelper - Additional Exception during Exception handling java.lang.ClassCastException 2019-07-09 16:26:24,720 [salt-event-thread-3] ERROR com.redhat.rhn.frontend.events.TransactionHelper - java.lang.ClassCastException 2019-07-09 16:26:24,719 [salt-event-thread-1] ERROR com.redhat.rhn.frontend.events.TransactionHelper - com.suse.manager.reactor.PGEventListenerException 2019-07-09 16:26:24,720 [salt-event-thread-5] ERROR com.redhat.rhn.frontend.events.TransactionHelper - com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected NUMBER but was BEGIN_OBJECT at path $.retcode

Any ideas how to debug this one? Which logs should I check ? Maybe someone has already had a similar situation...

Best Regards, Strahil Nikolov

Hello Community, I thound the problem, which in fact is not into the uyuni tools as I thought. I've downgraded my openSUSE 15.0 systems to use latest salt-2018.X.Y.Z (and all dependencies) and the issue with the constant package refresh stopped . It seems that openSUSE 15.0 started to use salt with version v.2018 (like uyuni) while later updates provide v.2019. Best Regards, Strahil Nikolov -- To unsubscribe, e-mail: uyuni-users+unsubscribe@opensuse.org To contact the owner, e-mail: uyuni-users+owner@opensuse.org