Leap 15.2+ ipmievd times out on start
Hello, I upgraded one of my servers from Leap 15.1 to 15.4 (first to 15.2, then 15.3, eventually 15.4). Since the update to 15.2 it fails to start: # systemctl status ipmievd.service × ipmievd.service - Ipmievd Daemon Loaded: loaded (/usr/lib/systemd/system/ipmievd.service; enabled; vendor preset: disabled) Active: failed (Result: timeout) since Thu 2023-06-01 08:30:05 CEST; 40min ago Process: 2369 ExecStart=/usr/sbin/ipmievd $IPMIEVD_OPTIONS (code=exited, status=0/SUCCESS) Jun 01 08:28:35 theta systemd[1]: Starting Ipmievd Daemon... Jun 01 08:28:35 theta systemd[1]: ipmievd.service: Can't open PID file /run/ipmievd.pid (yet?) after start: Operation not permitted Jun 01 08:28:35 theta ipmievd[2370]: Reading sensors... Jun 01 08:28:36 theta ipmievd[2370]: Waiting for events... Jun 01 08:30:05 theta systemd[1]: ipmievd.service: start operation timed out. Terminating. Jun 01 08:30:05 theta systemd[1]: ipmievd.service: Failed with result 'timeout'. Jun 01 08:30:05 theta systemd[1]: Failed to start Ipmievd Daemon. # cat /etc/ipmievd IPMIEVD_OPTIONS="sel daemon pidfile=/run/ipmievd.pid" # zypper if ipmitool Loading repository data... Reading installed packages... Information for package ipmitool: --------------------------------- Repository : openSUSE_main Name : ipmitool Version : 1.8.18.238.gb7adc1d-150400.1.14 Arch : x86_64 Vendor : SUSE LLC <https://www.suse.com/> Installed Size : 5.8 MiB Installed : Yes Status : up-to-date Source package : ipmitool-1.8.18.238.gb7adc1d-150400.1.14.src Upstream URL : https://github.com/ipmitool/ipmitool Summary : Utility for IPMI Control Description : ipmitool itself runs fine. # ipmitool sel SEL Information Version : 1.5 (v1.5, v2 compliant) Entries : 5 Free Space : 8112 bytes Percent Used : 0% Last Add Time : 05/31/2023 11:46:21 AM CEST Last Del Time : 08/27/2019 04:52:20 PM CEST Overflow : false Supported Cmds : 'Reserve' what may the problem be? Thanks and regards Paul Neuwirth
On 2023-06-01 09:22, Paul Neuwirth via openSUSE Users wrote:
Hello,
I upgraded one of my servers from Leap 15.1 to 15.4 (first to 15.2, then 15.3, eventually 15.4). Since the update to 15.2 it fails to start:
# systemctl status ipmievd.service × ipmievd.service - Ipmievd Daemon Loaded: loaded (/usr/lib/systemd/system/ipmievd.service; enabled; vendor preset: disabled) Active: failed (Result: timeout) since Thu 2023-06-01 08:30:05 CEST; 40min ago Process: 2369 ExecStart=/usr/sbin/ipmievd $IPMIEVD_OPTIONS (code=exited, status=0/SUCCESS)
Jun 01 08:28:35 theta systemd[1]: Starting Ipmievd Daemon... Jun 01 08:28:35 theta systemd[1]: ipmievd.service: Can't open PID file /run/ipmievd.pid (yet?) after start: Operation not permitted
When I see that message, the first guess is to check on apparmor. in a terminal as root, run aa-logprof. Next, try ls -l /run/ipmievd.pid ... -- Cheers / Saludos, Carlos E. R. (from 15.4 x86_64 at Telcontar)
On Thursday 2023-06-01 10:28, Carlos E. R. wrote:
Date: Thu, 1 Jun 2023 10:28:30 From: Carlos E. R. <robin.listas@telefonica.net> To: oS-EN <users@lists.opensuse.org> Subject: Re: Leap 15.2+ ipmievd times out on start
On 2023-06-01 09:22, Paul Neuwirth via openSUSE Users wrote:
Hello,
I upgraded one of my servers from Leap 15.1 to 15.4 (first to 15.2, then 15.3, eventually 15.4). Since the update to 15.2 it fails to start:
# systemctl status ipmievd.service × ipmievd.service - Ipmievd Daemon Loaded: loaded (/usr/lib/systemd/system/ipmievd.service; enabled; vendor preset: disabled) Active: failed (Result: timeout) since Thu 2023-06-01 08:30:05 CEST; 40min ago Process: 2369 ExecStart=/usr/sbin/ipmievd $IPMIEVD_OPTIONS (code=exited, status=0/SUCCESS)
Jun 01 08:28:35 theta systemd[1]: Starting Ipmievd Daemon... Jun 01 08:28:35 theta systemd[1]: ipmievd.service: Can't open PID file /run/ipmievd.pid (yet?) after start: Operation not permitted
When I see that message, the first guess is to check on apparmor.
in a terminal as root, run aa-logprof. nothing there: # aa-logprof Updating AppArmor profiles in /etc/apparmor.d. Reading log entries from /var/log/audit/audit.log.
i also tried to stop apparmor, and restart ipmievd, same result.
Next, try
ls -l /run/ipmievd.pid
as I would have expected: # ls -l /run/ipmievd.pid ls: cannot access '/run/ipmievd.pid': No such file or directory
...
-- Cheers / Saludos,
Carlos E. R. (from 15.4 x86_64 at Telcontar)
Paul Neuwirth
On Thu, Jun 1, 2023 at 2:12 PM Paul Neuwirth via openSUSE Users <users@lists.opensuse.org> wrote:
as I would have expected: # ls -l /run/ipmievd.pid ls: cannot access '/run/ipmievd.pid': No such file or directory
So run it manually, see what it does, whether there are any errors, use options to increase verbose/debug, run it under strace ... or wait for someone intimately familiar with this specific program. Service startup fails because systemd does not see the PID file. You need to debug why this file is not created.
On Thursday 2023-06-01 13:33, Andrei Borzenkov wrote:
Date: Thu, 1 Jun 2023 13:33:42 From: Andrei Borzenkov <arvidjaar@gmail.com> To: oS-EN <users@lists.opensuse.org> Subject: Re: Leap 15.2+ ipmievd times out on start
On Thu, Jun 1, 2023 at 2:12 PM Paul Neuwirth via openSUSE Users <users@lists.opensuse.org> wrote:
as I would have expected: # ls -l /run/ipmievd.pid ls: cannot access '/run/ipmievd.pid': No such file or directory
So run it manually, see what it does, whether there are any errors, use options to increase verbose/debug, run it under strace ... or wait for someone intimately familiar with this specific program.
Service startup fails because systemd does not see the PID file. You need to debug why this file is not created.
This came to my mind just a few moments earlier ;-) running it manually works fine. It seems systemd times out, only because of the missing pid file, as process is running, until the timeout. a pid file named /run/ipmievd.pid0 is being created. I then deleted the config file and re-installed ipmitool. config file kept the same. running with -vv I get following line: ipmievd: using pidfile /run/ipmievd.pid0 no matter how much I increase verbosity, nothing is displayed, why the pidfile is named that way, although pidfile is set in the arguments explicitely: ipmievd -vv sel daemon pidfile=/run/ipmievd.pid manual says pidfile=<filename> Save process ID to this file when in daemon mode. Defaults to /run/ipmievd.pidN (where N is the ipmi device number -- defaults to 0). so the argument is not used at all. I can set it to whatever, file keeps to be /run/ipmievd.pid0 appears to be a bug in ipmitool ? Paul Neuwirth
On Thu, Jun 1, 2023 at 2:59 PM Paul Neuwirth <mail@paul-neuwirth.nl> wrote:
so the argument is not used at all. I can set it to whatever, file keeps to be /run/ipmievd.pid0
appears to be a bug in ipmitool ?
Yes. else if (strcasecmp(argv[i], "pidfile=") == 0) { which of course never matches. Some ancient source that can be found is using str*n*casecmp as it should. You may consider a bug report.
On 2023-06-01 13:59, Paul Neuwirth via openSUSE Users wrote:
On Thursday 2023-06-01 13:33, Andrei Borzenkov wrote:
Date: Thu, 1 Jun 2023 13:33:42 From: Andrei Borzenkov <arvidjaar@gmail.com> To: oS-EN <users@lists.opensuse.org> Subject: Re: Leap 15.2+ ipmievd times out on start
On Thu, Jun 1, 2023 at 2:12 PM Paul Neuwirth via openSUSE Users <users@lists.opensuse.org> wrote:
as I would have expected: # ls -l /run/ipmievd.pid ls: cannot access '/run/ipmievd.pid': No such file or directory
So run it manually, see what it does, whether there are any errors, use options to increase verbose/debug, run it under strace ... or wait for someone intimately familiar with this specific program.
Service startup fails because systemd does not see the PID file. You need to debug why this file is not created.
This came to my mind just a few moments earlier ;-) running it manually works fine. It seems systemd times out, only because of the missing pid file, as process is running, until the timeout. a pid file named /run/ipmievd.pid0 is being created. I then deleted the config file and re-installed ipmitool. config file kept the same.
running with -vv I get following line: ipmievd: using pidfile /run/ipmievd.pid0 no matter how much I increase verbosity, nothing is displayed, why the pidfile is named that way, although pidfile is set in the arguments explicitely: ipmievd -vv sel daemon pidfile=/run/ipmievd.pid
manual says pidfile=<filename> Save process ID to this file when in daemon mode. Defaults to /run/ipmievd.pidN (where N is the ipmi device number -- defaults to 0).
so the argument is not used at all. I can set it to whatever, file keeps to be /run/ipmievd.pid0
appears to be a bug in ipmitool ?
The manual says that it will use /run/ipmievd.pid0, it is doing what it says. It wants a number. -- Cheers / Saludos, Carlos E. R. (from 15.4 x86_64 at Telcontar)
ROn Thursday 2023-06-01 21:56, Carlos E. R. wrote:
Date: Thu, 1 Jun 2023 21:56:55 From: Carlos E. R. <robin.listas@telefonica.net> To: oS-EN <users@lists.opensuse.org> Subject: Re: Leap 15.2+ ipmievd times out on start
On 2023-06-01 13:59, Paul Neuwirth via openSUSE Users wrote:
On Thursday 2023-06-01 13:33, Andrei Borzenkov wrote:
Date: Thu, 1 Jun 2023 13:33:42 From: Andrei Borzenkov <arvidjaar@gmail.com> To: oS-EN <users@lists.opensuse.org> Subject: Re: Leap 15.2+ ipmievd times out on start
On Thu, Jun 1, 2023 at 2:12 PM Paul Neuwirth via openSUSE Users <users@lists.opensuse.org> wrote:
as I would have expected: # ls -l /run/ipmievd.pid ls: cannot access '/run/ipmievd.pid': No such file or directory
So run it manually, see what it does, whether there are any errors, use options to increase verbose/debug, run it under strace ... or wait for someone intimately familiar with this specific program.
Service startup fails because systemd does not see the PID file. You need to debug why this file is not created.
This came to my mind just a few moments earlier ;-) running it manually works fine. It seems systemd times out, only because of the missing pid file, as process is running, until the timeout. a pid file named /run/ipmievd.pid0 is being created. I then deleted the config file and re-installed ipmitool. config file kept the same.
running with -vv I get following line: ipmievd: using pidfile /run/ipmievd.pid0 no matter how much I increase verbosity, nothing is displayed, why the pidfile is named that way, although pidfile is set in the arguments explicitely: ipmievd -vv sel daemon pidfile=/run/ipmievd.pid
manual says pidfile=<filename> Save process ID to this file when in daemon mode. Defaults to /run/ipmievd.pidN (where N is the ipmi device number -- defaults to 0).
so the argument is not used at all. I can set it to whatever, file keeps to be /run/ipmievd.pid0
appears to be a bug in ipmitool ?
The manual says that it will use /run/ipmievd.pid0, it is doing what it says. It wants a number.
I don't see it
pidfile=<filename> Save process ID to this file when in daemon mode. Defaults to /run/ipmievd.pidN (where N is the ipmi device number -- defaults to 0).
that's all it says, not mentioning it requires a number (or N) in argument. also no change in providing an argument like this: # ipmievd -vv sel daemon pidfile=/run/custompidfile.pidN ipmievd version 1.8.18 Loading IANA PEN Registry... Using ipmi device 0 Set IPMB address to 0x20 Iana: 674 Running Get PICMG Properties my_addr 0x20, transit 0, target 0 Error response 0xc1 from Get PICMG Properties Running Get VSO Capabilities my_addr 0x20, transit 0, target 0 Invalid completion code received: Invalid command Acquire IPMB address Discovered IPMB address 0x0 Interface address: my_addr 0x20 transit 0:0 target 0x20:0 ipmb_target 0 ipmievd: using pidfile /run/ipmievd.pid0 Using ipmi device 0 Set IPMB address to 0x20
-- Cheers / Saludos,
Carlos E. R. (from 15.4 x86_64 at Telcontar)
Regards Paul
On 2023-06-02 07:47, Paul Neuwirth via openSUSE Users wrote:
ROn Thursday 2023-06-01 21:56, Carlos E. R. wrote:
...
running with -vv I get following line: ipmievd: using pidfile /run/ipmievd.pid0 no matter how much I increase verbosity, nothing is displayed, why the pidfile is named that way, although pidfile is set in the arguments explicitely: ipmievd -vv sel daemon pidfile=/run/ipmievd.pid
manual says pidfile=<filename> Save process ID to this file when in daemon mode. Defaults to /run/ipmievd.pidN (where N is the ipmi device number -- defaults to 0).
so the argument is not used at all. I can set it to whatever, file keeps to be /run/ipmievd.pid0
appears to be a bug in ipmitool ?
The manual says that it will use /run/ipmievd.pid0, it is doing what it says. It wants a number.
I don't see it
pidfile=<filename> Save process ID to this file when in daemon mode. Defaults to /run/ipmievd.pidN (where N is the ipmi device number -- defaults to 0).
that's all it says, not mentioning it requires a number (or N) in argument. also no change in providing an argument like this:
"N is the number". So it will use a number. I don't know how to specify a different file in the command line, maybe that's a bug. It will use "/run/ipmievd.pid0", so you just have to tell systemd about it. The systemd service file is: [Unit] Description=Ipmievd Daemon After=syslog.target After=ipmi.service [Service] EnvironmentFile=-/etc/ipmievd ExecStart=/usr/sbin/ipmievd $IPMIEVD_OPTIONS Type=forking PIDFile=/run/ipmievd.pid [Install] WantedBy=multi-user.target So you have to run in a terminal as root (create an override): # systemctl edit ipmievd.service you will get a commented out copy of the service file for edit in "vi" (you can cheange that). So you write: [Service] PIDFile=/run/ipmievd.pid0 and save it, then try to start the service. If this works, add that information to the bug. -- Cheers / Saludos, Carlos E. R. (from 15.4 x86_64 at Telcontar)
On Friday 2023-06-02 11:37, Carlos E. R. wrote:
Date: Fri, 2 Jun 2023 11:37:25 From: Carlos E. R. <robin.listas@telefonica.net> To: oS-EN <users@lists.opensuse.org> Subject: Re: Leap 15.2+ ipmievd times out on start
On 2023-06-02 07:47, Paul Neuwirth via openSUSE Users wrote:
ROn Thursday 2023-06-01 21:56, Carlos E. R. wrote:
...
running with -vv I get following line: ipmievd: using pidfile /run/ipmievd.pid0 no matter how much I increase verbosity, nothing is displayed, why the pidfile is named that way, although pidfile is set in the arguments explicitely: ipmievd -vv sel daemon pidfile=/run/ipmievd.pid
manual says pidfile=<filename> Save process ID to this file when in daemon mode. Defaults to /run/ipmievd.pidN (where N is the ipmi device number -- defaults to 0).
so the argument is not used at all. I can set it to whatever, file keeps to be /run/ipmievd.pid0
appears to be a bug in ipmitool ?
The manual says that it will use /run/ipmievd.pid0, it is doing what it says. It wants a number.
I don't see it
pidfile=<filename> Save process ID to this file when in daemon mode. Defaults to /run/ipmievd.pidN (where N is the ipmi device number -- defaults to 0).
that's all it says, not mentioning it requires a number (or N) in argument. also no change in providing an argument like this:
"N is the number". So it will use a number. I don't know how to specify a different file in the command line, maybe that's a bug. It will use "/run/ipmievd.pid0", so you just have to tell systemd about it.
The systemd service file is:
[Unit] Description=Ipmievd Daemon After=syslog.target After=ipmi.service
[Service] EnvironmentFile=-/etc/ipmievd ExecStart=/usr/sbin/ipmievd $IPMIEVD_OPTIONS Type=forking PIDFile=/run/ipmievd.pid
[Install] WantedBy=multi-user.target
So you have to run in a terminal as root (create an override):
# systemctl edit ipmievd.service nice command, I always created/edited files manually. neat.
you will get a commented out copy of the service file for edit in "vi" (you can cheange that). So you write:
[Service] PIDFile=/run/ipmievd.pid0
and save it, then try to start the service.
If this works, add that information to the bug.
thank you it works now. But I also started a thread in the ipmitool-devel list. it appears to me to be a bug somewhere.
-- Cheers / Saludos,
Carlos E. R. (from 15.4 x86_64 at Telcontar)
On 2023-06-02 12:27, Paul Neuwirth via openSUSE Users wrote:
On Friday 2023-06-02 11:37, Carlos E. R. wrote:
So you have to run in a terminal as root (create an override):
# systemctl edit ipmievd.service
nice command, I always created/edited files manually. neat.
Yes, it creates a modification of the existing service file (one single line affected), saved as an override file. The original is left intact. -- Cheers / Saludos, Carlos E. R. (from 15.4 x86_64 at Telcontar)
participants (3)
-
Andrei Borzenkov -
Carlos E. R. -
Paul Neuwirth