Kickstart profile creation fails (on Enterprise Linux 9)
Hello, can anyone supply me an idea what to look for/at to solve the below issue? (keep in mind, that I am not using the Leap Java bundles/versions) Steps taken: 1. Go to https://<server>/rhn/kickstart/CreateProfileWizard.do 2. Follow the wizard to the end past the password screen 3. Click on Finish Expectation: New profile shows up and can be Uyuni-managed Reality: "Internal Server Error" shows up. On loading the profile screen, the profile is there but cannot be Uyuni-managed. /var/log/cobbler/cobbler.log does not seem to show anything releavant (errors). /var/log/rhn/web* did not get any new entries or have not been created after deletion and restart. /var/log/tomcat/catalina.2023-02-16.log 16-Feb-2023 09:31:33.473 SEVERE [http-nio-127.0.0.1-8080-exec-1] org.apache.tomcat.websocket.pojo.PojoEndpointBase.onError Failed to call onError method of POJO end point for POJO of type [com.suse.manager.webui.websocket.Notification] java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.tomcat.websocket.pojo.PojoEndpointBase.onError(PojoEndpointBase.java:134) at org.apache.tomcat.websocket.WsFrameBase.handleThrowableOnSend(WsFrameBase.java:579) at org.apache.tomcat.websocket.WsFrameBase.sendMessageText(WsFrameBase.java:418) at org.apache.tomcat.websocket.server.WsFrameServer.sendMessageText(WsFrameServer.java:129) at org.apache.tomcat.websocket.WsFrameBase.processDataText(WsFrameBase.java:515) at org.apache.tomcat.websocket.WsFrameBase.processData(WsFrameBase.java:301) at org.apache.tomcat.websocket.WsFrameBase.processInputBuffer(WsFrameBase.java:133) at org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:85) at org.apache.tomcat.websocket.server.WsFrameServer.doOnDataAvailable(WsFrameServer.java:183) at org.apache.tomcat.websocket.server.WsFrameServer.notifyDataAvailable(WsFrameServer.java:162) at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:157) at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:60) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.NullPointerException at com.suse.manager.webui.websocket.Notification.onError(Notification.java:150) ... 24 more 16-Feb-2023 09:32:01.557 WARNING [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-10] org.apache.struts.action.RequestProcessor.processException Unhandled Exception thrown: class java.lang.reflect.InvocationTargetException /var/log/tomcat/localhost.2023-02-16.log 16-Feb-2023 09:32:01.578 SEVERE [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-10] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [action] in context with path [/rhn] threw exception [jav a.lang.reflect.InvocationTargetException] with root cause redstone.xmlrpc.XmlRpcFault: <class 'TypeError'>:cannot marshal None unless allow_none is enabled at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:444) at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165) at org.cobbler.CobblerConnection.invokeMethod(CobblerConnection.java:146) at org.cobbler.CobblerConnection.invokeMethod(CobblerConnection.java:162) at org.cobbler.CobblerObject.refreshResolved(CobblerObject.java:360) at org.cobbler.CobblerObject.modify(CobblerObject.java:341) at org.cobbler.CobblerObject.modify(CobblerObject.java:321) at org.cobbler.Profile.setDistro(Profile.java:724) at org.cobbler.Profile.setDistro(Profile.java:715) at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileCommand.updateCobblerFields(CobblerProfileCommand.java:75) at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileCreateCommand.store(CobblerProfileCreateCommand.java:107) at com.redhat.rhn.manager.kickstart.KickstartWizardHelper.store(KickstartWizardHelper.java:187) at com.redhat.rhn.domain.kickstart.builder.KickstartBuilder.create(KickstartBuilder.java:585) at com.redhat.rhn.frontend.action.kickstart.CreateProfileWizardAction.runComplete(CreateProfileWizardAction.java:297) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.redhat.rhn.frontend.struts.wizard.WizardStep.invoke(WizardStep.java:115) at com.redhat.rhn.frontend.struts.wizard.RhnWizardAction.dispatch(RhnWizardAction.java:104) at com.redhat.rhn.frontend.struts.wizard.RhnWizardAction.execute(RhnWizardAction.java:90) at com.redhat.rhn.frontend.action.kickstart.CreateProfileWizardAction.execute(CreateProfileWizardAction.java:104) at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:237) at com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:104) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196) at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432) at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at com.redhat.rhn.frontend.servlets.AuthFilter.doFilter(AuthFilter.java:99) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142) at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:71) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:102) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:59) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:98) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:829) Thank you and best wishes, Stefan
Hi Stefan, This stacktrace seems to be the most useful in those you provided. You probably should deploy the code from git, put a breakpoint in CobblerProfileCommand.java:75 and debug from here. It seems there is a null value somewhere that doesn't please the cobbler XML-RPC API... but what is null? If you need help to setup your server to connect to your IDE's debugger, just ask. I though we had that documented on the wiki but can't find it anymore. -- Cedric On Thu, 2023-02-16 at 09:51 +0100, Stefan Bluhm wrote:
16-Feb-2023 09:32:01.578 SEVERE [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-10] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [action] in context with path [/rhn] threw exception [jav a.lang.reflect.InvocationTargetException] with root cause redstone.xmlrpc.XmlRpcFault: <class 'TypeError'>:cannot marshal None unless allow_none is enabled at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:444) at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165) at org.cobbler.CobblerConnection.invokeMethod(CobblerConnection.java:146) at org.cobbler.CobblerConnection.invokeMethod(CobblerConnection.java:162) at org.cobbler.CobblerObject.refreshResolved(CobblerObject.java:360) at org.cobbler.CobblerObject.modify(CobblerObject.java:341) at org.cobbler.CobblerObject.modify(CobblerObject.java:321) at org.cobbler.Profile.setDistro(Profile.java:724) at org.cobbler.Profile.setDistro(Profile.java:715) at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileCommand.updateCobblerFields(CobblerProfileCommand.java:75) at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileCreateCommand.store(CobblerProfileCreateCommand.java:107) at com.redhat.rhn.manager.kickstart.KickstartWizardHelper.store(KickstartWizardHelper.java:187) at com.redhat.rhn.domain.kickstart.builder.KickstartBuilder.create(KickstartBuilder.java:585) at com.redhat.rhn.frontend.action.kickstart.CreateProfileWizardAction.runComplete(CreateProfileWizardAction.java:297)
Hello Cedric, I managed to setup the remote debugging connection, triggered the breakpoint and stepped through the code a bit. So next would be to identify the issue. Your instructions are in the git wiki (https://github.com/uyuni-project/uyuni/wiki/Development-Environment-Alternat...). Thank you for motivating me to get the debugging set up. That will certainly help me in future. (so far successfully avoided it for the last few years now...) Best wishes, Stefan ----- Ursprüngliche Mail ----- Von: "Cedric Bosdonnat via Uyuni Development" <devel@lists.uyuni-project.org> An: "devel" <devel@lists.uyuni-project.org>, "uyuni-devel" <uyuni-devel@opensuse.org> CC: "Cedric Bosdonnat" Gesendet: Sonntag, 26. Februar 2023 20:47:11 Betreff: Re: Kickstart profile creation fails (on Enterprise Linux 9) Hi Stefan, This stacktrace seems to be the most useful in those you provided. You probably should deploy the code from git, put a breakpoint in CobblerProfileCommand.java:75 and debug from here. It seems there is a null value somewhere that doesn't please the cobbler XML-RPC API... but what is null? If you need help to setup your server to connect to your IDE's debugger, just ask. I though we had that documented on the wiki but can't find it anymore. -- Cedric On Thu, 2023-02-16 at 09:51 +0100, Stefan Bluhm wrote:
16-Feb-2023 09:32:01.578 SEVERE [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-10] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [action] in context with path [/rhn] threw exception [jav a.lang.reflect.InvocationTargetException] with root cause redstone.xmlrpc.XmlRpcFault: <class 'TypeError'>:cannot marshal None unless allow_none is enabled at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:444) at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165) at org.cobbler.CobblerConnection.invokeMethod(CobblerConnection.java:146) at org.cobbler.CobblerConnection.invokeMethod(CobblerConnection.java:162) at org.cobbler.CobblerObject.refreshResolved(CobblerObject.java:360) at org.cobbler.CobblerObject.modify(CobblerObject.java:341) at org.cobbler.CobblerObject.modify(CobblerObject.java:321) at org.cobbler.Profile.setDistro(Profile.java:724) at org.cobbler.Profile.setDistro(Profile.java:715) at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileCommand.updateCobblerFields(CobblerProfileCommand.java:75) at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileCreateCommand.store(CobblerProfileCreateCommand.java:107) at com.redhat.rhn.manager.kickstart.KickstartWizardHelper.store(KickstartWizardHelper.java:187) at com.redhat.rhn.domain.kickstart.builder.KickstartBuilder.create(KickstartBuilder.java:585) at com.redhat.rhn.frontend.action.kickstart.CreateProfileWizardAction.runComplete(CreateProfileWizardAction.java:297)
Thanks again Cedric, I was able to trace the request up until the point when it was passed on to Cobbler and also compared it to a working Leap installation. Despite the Enterprise Linux and Leap version of Cobbler both being the same, it seems there are still differences (probably due to the interface patches in OBS). Installing the OBS version on Enterprise Linux also removed the exception. Best wishes, Stefan ----- Ursprüngliche Mail ----- Von: "Cedric Bosdonnat via Uyuni Development" <devel@lists.uyuni-project.org> An: "devel" <devel@lists.uyuni-project.org>, "uyuni-devel" <uyuni-devel@opensuse.org> CC: "Cedric Bosdonnat" Gesendet: Sonntag, 26. Februar 2023 20:47:11 Betreff: Re: Kickstart profile creation fails (on Enterprise Linux 9) Hi Stefan, This stacktrace seems to be the most useful in those you provided. You probably should deploy the code from git, put a breakpoint in CobblerProfileCommand.java:75 and debug from here. It seems there is a null value somewhere that doesn't please the cobbler XML-RPC API... but what is null? If you need help to setup your server to connect to your IDE's debugger, just ask. I though we had that documented on the wiki but can't find it anymore. -- Cedric On Thu, 2023-02-16 at 09:51 +0100, Stefan Bluhm wrote:
16-Feb-2023 09:32:01.578 SEVERE [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-10] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [action] in context with path [/rhn] threw exception [jav a.lang.reflect.InvocationTargetException] with root cause redstone.xmlrpc.XmlRpcFault: <class 'TypeError'>:cannot marshal None unless allow_none is enabled at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:444) at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165) at org.cobbler.CobblerConnection.invokeMethod(CobblerConnection.java:146) at org.cobbler.CobblerConnection.invokeMethod(CobblerConnection.java:162) at org.cobbler.CobblerObject.refreshResolved(CobblerObject.java:360) at org.cobbler.CobblerObject.modify(CobblerObject.java:341) at org.cobbler.CobblerObject.modify(CobblerObject.java:321) at org.cobbler.Profile.setDistro(Profile.java:724) at org.cobbler.Profile.setDistro(Profile.java:715) at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileCommand.updateCobblerFields(CobblerProfileCommand.java:75) at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileCreateCommand.store(CobblerProfileCreateCommand.java:107) at com.redhat.rhn.manager.kickstart.KickstartWizardHelper.store(KickstartWizardHelper.java:187) at com.redhat.rhn.domain.kickstart.builder.KickstartBuilder.create(KickstartBuilder.java:585) at com.redhat.rhn.frontend.action.kickstart.CreateProfileWizardAction.runComplete(CreateProfileWizardAction.java:297)
participants (2)
-
Cedric Bosdonnat -
Stefan Bluhm