[uyuni-users] need assistance with debugging bootstrap issues
Hello All, can you assist me by guiding me in the correct direction? Yesterday I found out that all my clients (4 openSUSE 15.1) are not reporting for the past 3 days.I tried to refresh , but it didn't do anything . Then I removed the VMs and tried to add them, but after accepting the salt key - the VM is not added as a system. I have checked the logs and I see a lot of errors , one of them is : 2019-07-09 16:26:24,719 [salt-event-thread-1] ERROR com.suse.manager.reactor.PGEventListener - Unexpected exception while executing a MessageAction com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at path $ at com.google.gson.Gson.fromJson(Gson.java:900) at com.google.gson.Gson.fromJson(Gson.java:953) at com.suse.manager.reactor.messaging.JobReturnEventMessageAction.lambda$execute$7(JobReturnEventMessageAction.java:128) at java.util.Optional.ifPresent(Optional.java:159) at com.suse.manager.reactor.messaging.JobReturnEventMessageAction.execute(JobReturnEventMessageAction.java:114) at com.suse.manager.reactor.PGEventListener.lambda$null$1(PGEventListener.java:74) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at com.suse.manager.reactor.PGEventListener.lambda$notify$2(PGEventListener.java:72) at java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:419) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at com.suse.manager.reactor.PGEventListener.notify(PGEventListener.java:69) at com.suse.salt.netapi.event.AbstractEventStream.notifyListeners(AbstractEventStream.java:64) at com.suse.manager.reactor.PGEventStream.lambda$processEvents$3(PGEventStream.java:147) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:352) at java.util.stream.Sink$ChainedReference.end(Sink.java:258) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at com.suse.manager.reactor.PGEventStream.processEvents(PGEventStream.java:141) at com.suse.manager.reactor.PGEventStream.lambda$null$0(PGEventStream.java:127) at com.redhat.rhn.frontend.events.TransactionHelper.run(TransactionHelper.java:62) at com.redhat.rhn.frontend.events.TransactionHelper.handlingTransaction(TransactionHelper.java:46) at com.suse.manager.reactor.PGEventStream.lambda$notification$2(PGEventStream.java:126) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at path $ at com.google.gson.internal.bind.JsonTreeReader.expect(JsonTreeReader.java:161) at com.google.gson.internal.bind.JsonTreeReader.beginObject(JsonTreeReader.java:87) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:183) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145) at com.google.gson.Gson.fromJson(Gson.java:888) ... 30 more 2019-07-09 16:26:24,719 [salt-event-thread-4] ERROR com.redhat.rhn.frontend.events.TransactionHelper - java.lang.ClassCastException 2019-07-09 16:26:24,719 [salt-event-thread-4] ERROR com.redhat.rhn.frontend.events.TransactionHelper - Additional Exception during Exception handling java.lang.ClassCastException 2019-07-09 16:26:24,720 [salt-event-thread-3] ERROR com.redhat.rhn.frontend.events.TransactionHelper - java.lang.ClassCastException 2019-07-09 16:26:24,719 [salt-event-thread-1] ERROR com.redhat.rhn.frontend.events.TransactionHelper - com.suse.manager.reactor.PGEventListenerException 2019-07-09 16:26:24,720 [salt-event-thread-5] ERROR com.redhat.rhn.frontend.events.TransactionHelper - com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected NUMBER but was BEGIN_OBJECT at path $.retcode Any ideas how to debug this one? Which logs should I check ? Maybe someone has already had a similar situation... Best Regards, Strahil Nikolov -- To unsubscribe, e-mail: uyuni-users+unsubscribe@opensuse.org To contact the owner, e-mail: uyuni-users+owner@opensuse.org
Hello, Can you try this: - remove minion key from master - stop the salt-minion on VM's - remove minion key from /etc/salt/pki/ - change the minion id in /etc/minion_id - change the machine id in /etc/machine-id with command systemd-machine-id-setup - start the salt-minion on VM's - accept new keys, check if minons is added on Uyuni This steps helped me with some stoborn minions some time ago. Cristian Gherman On 7/9/19 4:39 PM, Strahil Nikolov wrote:
Hello All, can you assist me by guiding me in the correct direction? Yesterday I found out that all my clients (4 openSUSE 15.1) are not reporting for the past 3 days.I tried to refresh , but it didn't do anything .
Then I removed the VMs and tried to add them, but after accepting the salt key - the VM is not added as a system. I have checked the logs and I see a lot of errors , one of them is : 2019-07-09 16:26:24,719 [salt-event-thread-1] ERROR com.suse.manager.reactor.PGEventListener - Unexpected exception while executing a MessageAction com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at path $ at com.google.gson.Gson.fromJson(Gson.java:900) at com.google.gson.Gson.fromJson(Gson.java:953) at com.suse.manager.reactor.messaging.JobReturnEventMessageAction.lambda$execute$7(JobReturnEventMessageAction.java:128) at java.util.Optional.ifPresent(Optional.java:159) at com.suse.manager.reactor.messaging.JobReturnEventMessageAction.execute(JobReturnEventMessageAction.java:114) at com.suse.manager.reactor.PGEventListener.lambda$null$1(PGEventListener.java:74) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at com.suse.manager.reactor.PGEventListener.lambda$notify$2(PGEventListener.java:72) at java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:419) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at com.suse.manager.reactor.PGEventListener.notify(PGEventListener.java:69) at com.suse.salt.netapi.event.AbstractEventStream.notifyListeners(AbstractEventStream.java:64) at com.suse.manager.reactor.PGEventStream.lambda$processEvents$3(PGEventStream.java:147) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:352) at java.util.stream.Sink$ChainedReference.end(Sink.java:258) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at com.suse.manager.reactor.PGEventStream.processEvents(PGEventStream.java:141) at com.suse.manager.reactor.PGEventStream.lambda$null$0(PGEventStream.java:127) at com.redhat.rhn.frontend.events.TransactionHelper.run(TransactionHelper.java:62) at com.redhat.rhn.frontend.events.TransactionHelper.handlingTransaction(TransactionHelper.java:46) at com.suse.manager.reactor.PGEventStream.lambda$notification$2(PGEventStream.java:126) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at path $ at com.google.gson.internal.bind.JsonTreeReader.expect(JsonTreeReader.java:161) at com.google.gson.internal.bind.JsonTreeReader.beginObject(JsonTreeReader.java:87) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:183) at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145) at com.google.gson.Gson.fromJson(Gson.java:888) ... 30 more 2019-07-09 16:26:24,719 [salt-event-thread-4] ERROR com.redhat.rhn.frontend.events.TransactionHelper - java.lang.ClassCastException 2019-07-09 16:26:24,719 [salt-event-thread-4] ERROR com.redhat.rhn.frontend.events.TransactionHelper - Additional Exception during Exception handling java.lang.ClassCastException 2019-07-09 16:26:24,720 [salt-event-thread-3] ERROR com.redhat.rhn.frontend.events.TransactionHelper - java.lang.ClassCastException 2019-07-09 16:26:24,719 [salt-event-thread-1] ERROR com.redhat.rhn.frontend.events.TransactionHelper - com.suse.manager.reactor.PGEventListenerException 2019-07-09 16:26:24,720 [salt-event-thread-5] ERROR com.redhat.rhn.frontend.events.TransactionHelper - com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected NUMBER but was BEGIN_OBJECT at path $.retcode
Any ideas how to debug this one? Which logs should I check ? Maybe someone has already had a similar situation...
Best Regards, Strahil Nikolov -- To unsubscribe, e-mail: uyuni-users+unsubscribe@opensuse.org To contact the owner, e-mail: uyuni-users+owner@opensuse.org
participants (2)
-
Cristian Gherman -
Strahil Nikolov