Re: Where to find bootstrap logs
Thank you both. the trace took me further. See some errors on accessing the CA. So some permission errors. I will look at them later. file_|-/etc/pki/ca-trust/source/anchors/RHN-ORG-TRUSTED-SSL-CERT_|-/etc/pki/ca-trust/source/anchors/RHN-ORG-TRUSTED-SSL-CERT_|-managed(retcode=2): Unable to manage file: none of the specified sources were found cmd_|-update-ca-certificates_|-/usr/bin/update-ca-trust extract_|-run(retcode=2): One or more requisite failed: certs./etc/pki/ca-trust/source/anchors/RHN-ORG-TRUSTED-SSL-CERT cmd_|-trust_res_gpg_key_|-rpm --import https://uyuni.example.com:443/pub/res-gpg-pubkey-0182b964.key_|-run(retcode=2): Command "rpm --import https://uyuni.example.com:443/pub/res-gpg-pubkey-0182b964.key" run cmd_|-trust_suse_manager_tools_rhel_gpg_key_|-rpm --import https://uyuni.example.com:443/pub/sle12-gpg-pubkey-39db7c82.key_|-run(retcode=2): Command "rpm --import https://uyuni.example.com:443/pub/sle12-gpg-pubkey-39db7c82.key" run Also in /var/logs/salt/api 2021-07-12 14:34:33,890 [salt.roster :106 ][ERROR ][1895] Can't access roster for backend flat: Roster file "susemanager-roster-17133542272910374645.tmp" not found The "py26-compat-salt" package will need some fixing and testing for EL8 for different namings and impacts.... Tomcat is also empty. Seems like I have to revisit some file locations and permissions. I am now moving this thread over to uyuni-devel as this is not a usage question anymore. Best wishes, Stefan Von: "Victor Zhestkov" <Victor.Zhestkov@suse.com> An: "opensuse org" <opensuse.org@bluhm-de.com>, "Julio Gonzalez" <jgonzalez@suse.com> CC: "Victor Zhestkov" <Victor.Zhestkov@suse.com>, "users" <users@lists.uyuni-project.org>, "uyuni-users" <uyuni-users@opensuse.org> Gesendet: Montag, 12. Juli 2021 14:11:45 Betreff: Re: Where to find bootstrap logs Hi Stefan. Bootstrapping with web UI is performed with salt-ssh (even if you are bootstraping ordinary minion, not ssh one) through salt-api service. To find out exact error it's better to set logging level for salt master to trace: /etc/salt/master.d/logging.conf: log_level_logfile: trace Could be done with any conf file under /etc/salt/master.d/ no need to restart salt-master, but salt-api only. Unfortunately most of the messages required to debugging are visible it trace logging level only, but it's too noisy and log file is growing to fast. Regards, Victor On Mon, 2021-07-12 at 13:42 +0200, Stefan Bluhm wrote: Hello Julio, Server and client are both AlmaLinux 8. The py26 packages were not installed so I guess this is a missing dependency that needs fixing. I installed them, now the salt api log is empty and I am back to square 1: No log entries at all. a) What OS is installed at the client? Since py26 is mentioned, I guess CentOS6? AlmaLinux 8 b) How are you bootstrapping? WebUI? Script? WebUI (Systems --> Bootstrapping) c) What kind of client do you intend to bootstrap? SSH or regular salt minion? I actually don't know. I guess SSH as it mentions SSH on the page? Maybe here is the user error. I don't know anything about boot strapping. This is what I did: 1. Installed Uyuni. 2. Loaded the channels and created an activation key 3. Kickstarted an Alma Client. Now I am trying add the Alma Server to the Uyuni system management. I did nothing further on the client assuming the WebUI boot strapping is doing everything. Best wishes, Stefan ----- Ursprüngliche Mail ----- Von: "Julio Gonzalez" < [ mailto:jgonzalez@suse.com | jgonzalez@suse.com ] > An: "Victor Zhestkov" < [ mailto:Victor.Zhestkov@suse.com | Victor.Zhestkov@suse.com ] >, "Stefan Bluhm" < [ mailto:opensuse.org@bluhm-de.com | opensuse.org@bluhm-de.com ] > CC: "users" < [ mailto:users@lists.uyuni-project.org | users@lists.uyuni-project.org ] >, "uyuni-users" < [ mailto:uyuni-users@opensuse.org | uyuni-users@opensuse.org ] > Gesendet: Montag, 12. Juli 2021 13:05:09 Betreff: Re: Where to find bootstrap logs On lunes, 12 de julio de 2021 10:07:13 (CEST) Stefan Bluhm wrote: BQ_BEGIN Thank you both! Julio, I added you to my linux god of the month list. I didn't realise "tail -f *" was possible. It would have saved me so much time in the past. there were no entries for "tail -f /var/log/rhn/*" But "tail -f /var/log/salt/*" gave in api ==> api <== 2021-07-12 10:02:17,312 [salt.utils.thin :289 ][WARNING ][1899] Module tornado is not a Python importable module with 2021-07-12 10:02:17,313 [salt.utils.thin :289 ][WARNING ][1899] Module msgpack is not a Python importable module with /usr/lib64/susemanager/py26-compat/msgpack 2021-07-12 10:02:17,364 [salt.utils.thin :282 ][WARNING ][1899] Module backports_abc configured with not a file or does not exist: /usr/lib/python2.7/site-packages/salt/ext/backports_abc.py 2021-07-12 10:02:17,365 [salt.utils.thin :303 ][ERROR ][1899] Missing dependencies for the alternative version in the external configuration: tornado, msgpack Our salt experts should have a look, but I wonder: a) What OS is installed at the client? Since py26 is mentioned, I guess CentOS6? b) How are you bootstrapping? WebUI? Script? c) What kind of client do you intend to bootstrap? SSH or regular salt minion? In any case, both directories are provided by the compat packages: m233:~ # rpm -qf /usr/lib64/susemanager/py26-compat/tornado py26-compat-tornado-4.2.1-53.2.uyuni1.x86_64 m233:~ # rpm -qf /usr/lib64/susemanager/py26-compat/msgpack py26-compat-msgpack-python-0.4.6-53.2.uyuni1.x86_64 Both should be installed, as dependencies for py26-compat-salt, which is part of the uyuni serve pattern. Do you have them installed? BQ_BEGIN So will chase up on those modules above. Best wishes, Stefan Von: "Victor Zhestkov" < [ mailto:Victor.Zhestkov@suse.com | Victor.Zhestkov@suse.com ] > An: "opensuse org" < [ mailto:opensuse.org@bluhm-de.com | opensuse.org@bluhm-de.com ] >, "Julio Gonzalez" < [ mailto:jgonzalez@suse.com | jgonzalez@suse.com ] > CC: "users" < [ mailto:users@lists.uyuni-project.org | users@lists.uyuni-project.org ] >, "uyuni-users" < [ mailto:uyuni-users@opensuse.org | uyuni-users@opensuse.org ] > Gesendet: Montag, 12. Juli 2021 09:58:14 Betreff: Re: Where to find bootstrap logs Hi Stefan. Please also check /var/log/salt/api. Regards, Victor On Mon, 2021-07-12 at 09:21 +0200, Stefan Bluhm wrote: Thank you Julio, I also expected a server error. But I do not see any relevant errors in the logs (or so I assume). find . -type f -newer /tmp/$$ # all files changed today ./tasko/sat/notifications-cleanup-bunch/notifications-cleanup_15359_out ./tasko/sat/mgr-sync-refresh-bunch/mgr-sync-refresh_16848_err ./tasko/sat/mgr-sync-refresh-bunch/mgr-sync-refresh_16848_out ./tasko/sat/channel-repodata-bunch/channel-repodata_17057_out ./search/rhn_search.log ./search/rhn_search_daemon.log ./rhn_taskomatic_daemon.log tomcat/local_access_log.2021-07-12.txt shows these consecutive lines: 192.168.0.152 - - [12/Jul/2021:07:33:06 +0200] "POST /rhn/manager/frontend-log HTTP/1.1" 200 16 192.168.0.152 - - [12/Jul/2021:07:44:53 +0200] "GET /rhn/errors/500.jsp HTTP/1.1" 500 - /var/lib/pgsql/data/log/postgresql-Mon.log is empty. So I guess, I have to up the log level, right? Best wishes, Stefan ----- Ursprüngliche Mail ----- Von: "Julio Gonzalez" < [ mailto: [ mailto:jgonzalez@suse.com | jgonzalez@suse.com ] | [ mailto:jgonzalez@suse.com | jgonzalez@suse.com ] ] > An: "uyuni-users" < [ mailto: [ mailto:uyuni-users@opensuse.org | uyuni-users@opensuse.org ] | [ mailto:uyuni-users@opensuse.org | uyuni-users@opensuse.org ] ] >, "users" < [ mailto: [ mailto:users@lists.uyuni-project.org | users@lists.uyuni-project.org ] | [ mailto:users@lists.uyuni-project.org | users@lists.uyuni-project.org ] ] > CC: "Stefan Bluhm" < [ mailto: [ mailto:opensuse.org@bluhm-de.com | opensuse.org@bluhm-de.com ] | [ mailto:opensuse.org@bluhm-de.com | opensuse.org@bluhm-de.com ] ] > Gesendet: Montag, 12. Juli 2021 08:50:08 Betreff: Re: Where to find bootstrap logs On lunes, 12 de julio de 2021 7:48:43 (CEST) Stefan Bluhm wrote: BQ_BEGIN Hello, I seem to be blind or have lost my patience. When entering "host" (tried IP/hostname), "user"/"password", "activation key" and "Disable SSH.." under "Systems --> Bootstrapping" a blue info message appears "Your system is bootstrapping: waiting for a response..". After some long time, the bar changes to the red error bar saying: "Server error, please check log files.". Where are the logs for the bootstrap attempts? I claim, I searched "everywhere". In this case, this seems to be problem at the server, not at the client. /var/log/rhn at the server is the place where you need to start looking. Most likely there will be an ISE (Internal Server error) there. In some cases it could be interesting having a look at the tomcat log and PostgreSQL logs. BQ_BEGIN Thank you and best wishes, Stefan BQ_END BQ_END BQ_END BQ_END
For the CA, check if the following file exists on the server: /usr/share/susemanager/salt/certs/RHN-ORG-TRUSTED-SSL-CERT It should be a link to /srv/www/htdocs/pub/RHN-ORG-TRUSTED-SSL-CERT For `rpm --import ..` errors check if the URLs are accessible and if so what returns rpm --import URL on running it manually On Mon, 2021-07-12 at 15:03 +0200, Stefan Bluhm wrote: Thank you both. the trace took me further. See some errors on accessing the CA. So some permission errors. I will look at them later. file_|-/etc/pki/ca-trust/source/anchors/RHN-ORG-TRUSTED-SSL-CERT_|-/etc/pki/ca-trust/source/anchors/RHN-ORG-TRUSTED-SSL-CERT_|-managed(retcode=2): Unable to manage file: none of the specified sources were found cmd_|-update-ca-certificates_|-/usr/bin/update-ca-trust extract_|-run(retcode=2): One or more requisite failed: certs./etc/pki/ca-trust/source/anchors/RHN-ORG-TRUSTED-SSL-CERT cmd_|-trust_res_gpg_key_|-rpm --import https://uyuni.example.com:443/pub/res-gpg-pubkey-0182b964.key_|-run(retcode=2): Command "rpm --import https://uyuni.example.com:443/pub/res-gpg-pubkey-0182b964.key" run cmd_|-trust_suse_manager_tools_rhel_gpg_key_|-rpm --import https://uyuni.example.com:443/pub/sle12-gpg-pubkey-39db7c82.key_|-run(retcode=2): Command "rpm --import https://uyuni.example.com:443/pub/sle12-gpg-pubkey-39db7c82.key" run Also in /var/logs/salt/api 2021-07-12 14:34:33,890 [salt.roster :106 ][ERROR ][1895] Can't access roster for backend flat: Roster file "susemanager-roster-17133542272910374645.tmp" not found The "py26-compat-salt" package will need some fixing and testing for EL8 for different namings and impacts.... Tomcat is also empty. Seems like I have to revisit some file locations and permissions. I am now moving this thread over to uyuni-devel as this is not a usage question anymore. Best wishes, Stefan ________________________________ Von: "Victor Zhestkov" <Victor.Zhestkov@suse.com> An: "opensuse org" <opensuse.org@bluhm-de.com>, "Julio Gonzalez" <jgonzalez@suse.com> CC: "Victor Zhestkov" <Victor.Zhestkov@suse.com>, "users" <users@lists.uyuni-project.org>, "uyuni-users" <uyuni-users@opensuse.org> Gesendet: Montag, 12. Juli 2021 14:11:45 Betreff: Re: Where to find bootstrap logs Hi Stefan. Bootstrapping with web UI is performed with salt-ssh (even if you are bootstraping ordinary minion, not ssh one) through salt-api service. To find out exact error it's better to set logging level for salt master to trace: /etc/salt/master.d/logging.conf: log_level_logfile: trace Could be done with any conf file under /etc/salt/master.d/ no need to restart salt-master, but salt-api only. Unfortunately most of the messages required to debugging are visible it trace logging level only, but it's too noisy and log file is growing to fast. Regards, Victor On Mon, 2021-07-12 at 13:42 +0200, Stefan Bluhm wrote: Hello Julio, Server and client are both AlmaLinux 8. The py26 packages were not installed so I guess this is a missing dependency that needs fixing. I installed them, now the salt api log is empty and I am back to square 1: No log entries at all. a) What OS is installed at the client? Since py26 is mentioned, I guess CentOS6? AlmaLinux 8 b) How are you bootstrapping? WebUI? Script? WebUI (Systems --> Bootstrapping) c) What kind of client do you intend to bootstrap? SSH or regular salt minion? I actually don't know. I guess SSH as it mentions SSH on the page? Maybe here is the user error. I don't know anything about boot strapping. This is what I did: 1. Installed Uyuni. 2. Loaded the channels and created an activation key 3. Kickstarted an Alma Client. Now I am trying add the Alma Server to the Uyuni system management. I did nothing further on the client assuming the WebUI boot strapping is doing everything. Best wishes, Stefan ----- Ursprüngliche Mail ----- Von: "Julio Gonzalez" <jgonzalez@suse.com<mailto:jgonzalez@suse.com>> An: "Victor Zhestkov" <Victor.Zhestkov@suse.com<mailto:Victor.Zhestkov@suse.com>>, "Stefan Bluhm" <opensuse.org@bluhm-de.com<mailto:opensuse.org@bluhm-de.com>> CC: "users" <users@lists.uyuni-project.org<mailto:users@lists.uyuni-project.org>>, "uyuni-users" <uyuni-users@opensuse.org<mailto:uyuni-users@opensuse.org>> Gesendet: Montag, 12. Juli 2021 13:05:09 Betreff: Re: Where to find bootstrap logs On lunes, 12 de julio de 2021 10:07:13 (CEST) Stefan Bluhm wrote: Thank you both! Julio, I added you to my linux god of the month list. I didn't realise "tail -f *" was possible. It would have saved me so much time in the past. there were no entries for "tail -f /var/log/rhn/*" But "tail -f /var/log/salt/*" gave in api ==> api <== 2021-07-12 10:02:17,312 [salt.utils.thin :289 ][WARNING ][1899] Module tornado is not a Python importable module with 2021-07-12 10:02:17,313 [salt.utils.thin :289 ][WARNING ][1899] Module msgpack is not a Python importable module with /usr/lib64/susemanager/py26-compat/msgpack 2021-07-12 10:02:17,364 [salt.utils.thin :282 ][WARNING ][1899] Module backports_abc configured with not a file or does not exist: /usr/lib/python2.7/site-packages/salt/ext/backports_abc.py 2021-07-12 10:02:17,365 [salt.utils.thin :303 ][ERROR ][1899] Missing dependencies for the alternative version in the external configuration: tornado, msgpack Our salt experts should have a look, but I wonder: a) What OS is installed at the client? Since py26 is mentioned, I guess CentOS6? b) How are you bootstrapping? WebUI? Script? c) What kind of client do you intend to bootstrap? SSH or regular salt minion? In any case, both directories are provided by the compat packages: m233:~ # rpm -qf /usr/lib64/susemanager/py26-compat/tornado py26-compat-tornado-4.2.1-53.2.uyuni1.x86_64 m233:~ # rpm -qf /usr/lib64/susemanager/py26-compat/msgpack py26-compat-msgpack-python-0.4.6-53.2.uyuni1.x86_64 Both should be installed, as dependencies for py26-compat-salt, which is part of the uyuni serve pattern. Do you have them installed? So will chase up on those modules above. Best wishes, Stefan Von: "Victor Zhestkov" <Victor.Zhestkov@suse.com<mailto:Victor.Zhestkov@suse.com>> An: "opensuse org" <opensuse.org@bluhm-de.com<mailto:opensuse.org@bluhm-de.com>>, "Julio Gonzalez" <jgonzalez@suse.com<mailto:jgonzalez@suse.com>> CC: "users" <users@lists.uyuni-project.org<mailto:users@lists.uyuni-project.org>>, "uyuni-users" <uyuni-users@opensuse.org<mailto:uyuni-users@opensuse.org>> Gesendet: Montag, 12. Juli 2021 09:58:14 Betreff: Re: Where to find bootstrap logs Hi Stefan. Please also check /var/log/salt/api. Regards, Victor On Mon, 2021-07-12 at 09:21 +0200, Stefan Bluhm wrote: Thank you Julio, I also expected a server error. But I do not see any relevant errors in the logs (or so I assume). find . -type f -newer /tmp/$$ # all files changed today ./tasko/sat/notifications-cleanup-bunch/notifications-cleanup_15359_out ./tasko/sat/mgr-sync-refresh-bunch/mgr-sync-refresh_16848_err ./tasko/sat/mgr-sync-refresh-bunch/mgr-sync-refresh_16848_out ./tasko/sat/channel-repodata-bunch/channel-repodata_17057_out ./search/rhn_search.log ./search/rhn_search_daemon.log ./rhn_taskomatic_daemon.log tomcat/local_access_log.2021-07-12.txt shows these consecutive lines: 192.168.0.152 - - [12/Jul/2021:07:33:06 +0200] "POST /rhn/manager/frontend-log HTTP/1.1" 200 16 192.168.0.152 - - [12/Jul/2021:07:44:53 +0200] "GET /rhn/errors/500.jsp HTTP/1.1" 500 - /var/lib/pgsql/data/log/postgresql-Mon.log is empty. So I guess, I have to up the log level, right? Best wishes, Stefan ----- Ursprüngliche Mail ----- Von: "Julio Gonzalez" < [ mailto:jgonzalez@suse.com<mailto:jgonzalez@suse.com> |jgonzalez@suse.com<mailto:jgonzalez@suse.com> ] > An: "uyuni-users" < [ mailto:uyuni-users@opensuse.org<mailto:uyuni-users@opensuse.org> | uyuni-users@opensuse.org<mailto:uyuni-users@opensuse.org> ] >, "users" < [ mailto:users@lists.uyuni-project.org<mailto:users@lists.uyuni-project.org> |users@lists.uyuni-project.org<mailto:users@lists.uyuni-project.org> ] > CC: "Stefan Bluhm" < [ mailto:opensuse.org@bluhm-de.com<mailto:opensuse.org@bluhm-de.com> | opensuse.org@bluhm-de.com<mailto:opensuse.org@bluhm-de.com> ] > Gesendet: Montag, 12. Juli 2021 08:50:08 Betreff: Re: Where to find bootstrap logs On lunes, 12 de julio de 2021 7:48:43 (CEST) Stefan Bluhm wrote: BQ_BEGIN Hello, I seem to be blind or have lost my patience. When entering "host" (tried IP/hostname), "user"/"password", "activation key" and "Disable SSH.." under "Systems --> Bootstrapping" a blue info message appears "Your system is bootstrapping: waiting for a response..". After some long time, the bar changes to the red error bar saying: "Server error, please check log files.". Where are the logs for the bootstrap attempts? I claim, I searched "everywhere". In this case, this seems to be problem at the server, not at the client. /var/log/rhn at the server is the place where you need to start looking. Most likely there will be an ISE (Internal Server error) there. In some cases it could be interesting having a look at the tomcat log and PostgreSQL logs. BQ_BEGIN Thank you and best wishes, Stefan BQ_END BQ_END
For the CA, check if the following file exists on the server: /usr/share/susemanager/salt/certs/RHN-ORG-TRUSTED-SSL-CERT It should be a link to /srv/www/htdocs/pub/RHN-ORG-TRUSTED-SSL-CERT For `rpm --import ..` errors check if the URLs are accessible and if so what returns rpm --import URL on running it manually On Mon, 2021-07-12 at 15:03 +0200, Stefan Bluhm wrote: Thank you both. the trace took me further. See some errors on accessing the CA. So some permission errors. I will look at them later. file_|-/etc/pki/ca-trust/source/anchors/RHN-ORG-TRUSTED-SSL-CERT_|-/etc/pki/ca-trust/source/anchors/RHN-ORG-TRUSTED-SSL-CERT_|-managed(retcode=2): Unable to manage file: none of the specified sources were found cmd_|-update-ca-certificates_|-/usr/bin/update-ca-trust extract_|-run(retcode=2): One or more requisite failed: certs./etc/pki/ca-trust/source/anchors/RHN-ORG-TRUSTED-SSL-CERT cmd_|-trust_res_gpg_key_|-rpm --import https://uyuni.example.com:443/pub/res-gpg-pubkey-0182b964.key_|-run(retcode=2): Command "rpm --import https://uyuni.example.com:443/pub/res-gpg-pubkey-0182b964.key" run cmd_|-trust_suse_manager_tools_rhel_gpg_key_|-rpm --import https://uyuni.example.com:443/pub/sle12-gpg-pubkey-39db7c82.key_|-run(retcode=2): Command "rpm --import https://uyuni.example.com:443/pub/sle12-gpg-pubkey-39db7c82.key" run Also in /var/logs/salt/api 2021-07-12 14:34:33,890 [salt.roster :106 ][ERROR ][1895] Can't access roster for backend flat: Roster file "susemanager-roster-17133542272910374645.tmp" not found The "py26-compat-salt" package will need some fixing and testing for EL8 for different namings and impacts.... Tomcat is also empty. Seems like I have to revisit some file locations and permissions. I am now moving this thread over to uyuni-devel as this is not a usage question anymore. Best wishes, Stefan
participants (2)
-
Stefan Bluhm -
Victor Zhestkov