Problem with my new OBS instance
Hi experts, I just set up a new private OBS instance and I've run into few frustrating problems. Could you give me some advice on how can I debug this problems, such as what logs to look at, how to reset things? My setup is simple, on a single server running openSUSE Leap 15.3, with OBS 2.10 installed from the repository. Then setup by the /usr/lib/obs/server/setup-appliance.sh script. Then I noticed these issues. Firstly, in the configuration UI, I tried to uncheck the i586 and armv7l architectures. But they keep coming back. I also tried removing them from /configuration API, still the same. This is really confusing. Secondly, I connected my instance with the boo instance, and tried to start building some test packages. I soon noticed that my kernel and inird setup was not correct so the worker VM cannot boot into the guest OS. So I edited /etc/sysconfig/obs-server to use a different initrd. But I made a mistake here, I put the wrong path name there thus the VM cannot find the initrd. I can see build logs about the VM cannot start, as expected. The problem is OBS didn't think the build failed, instead it kept restarting the VM. Thus I have to abort the build. The abort succeeded. But later after I corrected the initrd path (and restarted the worker service), the package stuck in the "scheduled" state, never started building, even after I reboot the server. The message I got was "waiting for 8 compliant workers (8 of them bad) ". The package was able to start building before I corrected the initrd path problem so there is no unmet constraint (I didn't put any in place either). I left it overnight and it still stuck there. Then I thought I should delete this package to start from scratch again. This caused more problem... The package I deleted is no longer visible in the package list, but it still showed as "scheduled" in the monitor. And I have "scheduled: 1" on the web UI build result section... It seems my instance is not in a consistent state now. But I can't find enough documentation to help me on debugging it. Any suggestions or hints or link to useful docs are greatly appreciated. Thanks, Kai
On Sep 15 2021, Kai Liu wrote:
The message I got was "waiting for 8 compliant workers (8 of them bad) ".
Use obs_admin --drop-badhosts to clear the badhosts markers. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
On 2021/09/15 Wed 14:29, Andreas Schwab wrote:
On Sep 15 2021, Kai Liu wrote:
The message I got was "waiting for 8 compliant workers (8 of them bad) ".
Use obs_admin --drop-badhosts to clear the badhosts markers.
Thanks. Tried that, obs_admin --list-badhosts gave me "no badhosts found" so I guess it did what it was supposed to do. Waited for a few minutes, the deleted package is still there in scheduled state :( Regards, Kai
On Mittwoch, 15. September 2021, 14:29:13 CEST Andreas Schwab wrote:
On Sep 15 2021, Kai Liu wrote:
The message I got was "waiting for 8 compliant workers (8 of them bad) ".
Use obs_admin --drop-badhosts to clear the badhosts markers.
well, yes, but the jobs will just fail then due to missing workers most likely. or your dispatcher has a problem and is not running. Check it's log files... -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer
On 2021/09/15 Wed 15:02, Adrian Schröter wrote:
On Mittwoch, 15. September 2021, 14:29:13 CEST Andreas Schwab wrote:
On Sep 15 2021, Kai Liu wrote:
The message I got was "waiting for 8 compliant workers (8 of them bad) ".
Use obs_admin --drop-badhosts to clear the badhosts markers.
well, yes, but the jobs will just fail then due to missing workers most likely.
or your dispatcher has a problem and is not running. Check it's log files...
The dispatcher is running - the Status Monitor shows good status, the bs_dispatch process is running. Here are the latest logs from /srv/obs/log/dispatcher.log. Could you let me know if you see anything uncommon? 2021-09-15 00:06:14: [2210] starting build service dispatcher 2021-09-15 00:06:14: [2210] reading old state... 2021-09-15 00:06:18: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 00:11:23: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 00:16:24: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 00:21:25: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 00:26:30: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 00:31:31: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 00:36:32: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 00:41:37: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 00:46:38: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 00:51:43: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 00:56:44: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:01:45: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:06:50: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:11:51: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:16:52: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:21:57: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:26:58: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:32:03: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:37:04: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:42:05: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:47:10: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:52:11: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 01:57:16: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:02:17: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:07:18: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:12:23: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:17:24: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:22:29: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:27:30: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:32:31: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:37:36: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:42:37: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:47:42: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:52:43: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 02:57:44: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:02:49: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:07:50: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:12:51: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:17:56: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:22:57: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:28:02: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:33:03: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:38:04: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:43:09: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:48:10: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:53:15: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 03:58:16: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:03:17: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:08:22: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:13:23: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:18:28: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:23:29: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:28:30: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:33:35: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:38:36: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:43:41: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:48:42: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:53:43: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 04:58:48: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:03:49: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:08:54: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:13:55: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:18:56: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:24:01: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:29:02: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:34:03: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:39:08: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:44:09: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:49:14: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:54:15: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 05:59:16: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:04:21: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:09:22: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:14:27: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:19:28: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:24:29: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:29:34: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:34:35: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:39:36: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:44:41: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:49:42: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:54:47: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 06:59:48: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 07:04:49: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 07:09:54: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 07:14:55: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 07:20:00: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 07:25:01: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 07:30:02: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 07:35:07: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 07:40:08: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 07:45:13: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 07:50:14: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 07:55:15: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:00:20: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:05:21: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:10:26: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:15:27: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:20:28: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:25:33: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:30:34: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:35:39: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:40:40: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:45:41: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:50:46: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 08:55:47: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:00:48: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:05:53: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:10:54: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:15:59: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:21:00: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:26:01: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:31:06: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:36:07: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:41:12: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:46:13: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:51:14: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 09:56:19: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 10:01:20: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 10:06:25: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 10:11:26: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 10:16:27: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 10:21:32: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 10:25:16: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 setting dispatch details to: waiting for 8 compliant workers (8 of them bad) 2021-09-15 10:30:08: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 setting dispatch details to: waiting for 8 compliant workers (8 of them bad) 2021-09-15 10:35:09: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 10:40:14: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 10:45:15: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 10:50:16: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 10:55:21: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:00:22: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:05:27: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:10:28: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:15:29: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:20:34: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:25:35: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:30:36: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:35:41: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:40:42: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:45:47: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:50:48: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 11:55:49: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:00:54: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:05:55: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:11:00: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:16:01: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:21:02: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:26:07: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:31:08: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:36:13: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:41:14: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:46:15: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:51:20: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 12:56:21: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:01:26: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:06:27: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:11:28: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:16:33: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:21:34: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:26:40: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:31:45: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:36:46: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:41:51: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:46:52: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:51:53: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 13:56:58: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:01:59: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:07:04: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:12:05: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:17:06: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:22:11: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:27:12: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:32:17: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:37:18: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:42:19: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:47:24: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:52:25: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 14:57:30: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:02:31: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:07:32: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:12:37: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:17:38: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:22:43: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:27:44: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:31:04: [2210] assignjob x86_64/home:kai::openEuler_20.03::zlib-3ded052870d4e970ab5c53a5ac8c7164 -> x86_64:build:8 2021-09-15 15:31:05: [2210] rpc error: 400 remote error: rebooting... 2021-09-15 15:31:05: [2210] assignjob x86_64/home:kai::openEuler_20.03::zlib-3ded052870d4e970ab5c53a5ac8c7164 -> x86_64:build:6 2021-09-15 15:31:05: [2210] rpc error: 400 remote error: rebooting... 2021-09-15 15:31:05: [2210] assignjob x86_64/home:kai::openEuler_20.03::zlib-3ded052870d4e970ab5c53a5ac8c7164 -> x86_64:build:7 2021-09-15 15:31:05: [2210] rpc error: 400 remote error: rebooting... 2021-09-15 15:31:05: [2210] assignjob x86_64/home:kai::openEuler_20.03::zlib-3ded052870d4e970ab5c53a5ac8c7164 -> x86_64:build:3 2021-09-15 15:31:06: [2210] rpc error: 400 remote error: rebooting... 2021-09-15 15:31:06: [2210] assignjob x86_64/home:kai::openEuler_20.03::zlib-3ded052870d4e970ab5c53a5ac8c7164 -> x86_64:build:2 2021-09-15 15:31:06: [2210] rpc error: 400 remote error: rebooting... 2021-09-15 15:31:06: [2210] assignjob x86_64/home:kai::openEuler_20.03::zlib-3ded052870d4e970ab5c53a5ac8c7164 -> x86_64:build:1 2021-09-15 15:31:06: [2210] rpc error: 400 remote error: rebooting... 2021-09-15 15:31:06: [2210] assignjob x86_64/home:kai::openEuler_20.03::zlib-3ded052870d4e970ab5c53a5ac8c7164 -> x86_64:build:4 2021-09-15 15:31:07: [2210] rpc error: 400 remote error: rebooting... 2021-09-15 15:31:07: [2210] assignjob x86_64/home:kai::openEuler_20.03::zlib-3ded052870d4e970ab5c53a5ac8c7164 -> x86_64:build:5 2021-09-15 15:31:07: [2210] rpc error: 400 remote error: rebooting... 2021-09-15 15:31:08: [2210] assignjob x86_64/home:kai::openEuler_20.03::zlib-3ded052870d4e970ab5c53a5ac8c7164 -> x86_64:build:3 2021-09-15 15:31:08: [2210] assigned 1 jobs 2021-09-15 15:32:48: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:37:49: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:41:09: [2210] assignjob x86_64/home:kai::openEuler_20.03::gcc-08f83c778c3d5deb3ac4d5ff65305264 -> x86_64:build:7 2021-09-15 15:41:09: [2210] assigned 1 jobs 2021-09-15 15:42:50: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:47:52: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:52:53: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 15:57:58: [2210] checkconstraints x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891 2021-09-15 17:07:26: [2210] deleting badhost x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891/x86_64:build:1 2021-09-15 17:07:26: [2210] deleting badhost home:kai/kernel/x86_64/x86_64:build:1 2021-09-15 17:16:39: [2210] exiting... 2021-09-15 17:19:51: [2133] starting build service dispatcher 2021-09-15 17:19:51: [2133] reading old state... 2021-09-15 17:37:21: [2133] deleting badhost home:kai/kernel/x86_64/x86_64:build:5 2021-09-15 17:37:21: [2133] deleting badhost x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891/x86_64:build:5 2021-09-15 17:43:46: [2133] deleting badhost home:kai/kernel/x86_64/x86_64:build:2 2021-09-15 17:43:46: [2133] deleting badhost x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891/x86_64:build:2 2021-09-15 17:56:39: [2133] deleting badhost x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891/x86_64:build:3 2021-09-15 17:56:39: [2133] deleting badhost home:kai/kernel/x86_64/x86_64:build:3 2021-09-15 18:03:02: [2133] deleting badhost home:kai/kernel/x86_64/x86_64:build:4 2021-09-15 18:03:02: [2133] deleting badhost x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891/x86_64:build:4 2021-09-15 18:09:22: [2133] deleting badhost home:kai/kernel/x86_64/x86_64:build:8 2021-09-15 18:09:22: [2133] deleting badhost x86_64/home:kai::openEuler_20.03::kernel-e3432bbe2af1598c30f72e4752442891/x86_64:build:8 2021-09-15 20:43:23: [2133] removing all badhost entries...
On Mittwoch, 15. September 2021, 15:37:25 CEST Kai Liu wrote:
On 2021/09/15 Wed 15:02, Adrian Schröter wrote:
On Mittwoch, 15. September 2021, 14:29:13 CEST Andreas Schwab wrote:
On Sep 15 2021, Kai Liu wrote:
The message I got was "waiting for 8 compliant workers (8 of them bad) ".
Use obs_admin --drop-badhosts to clear the badhosts markers.
well, yes, but the jobs will just fail then due to missing workers most likely.
or your dispatcher has a problem and is not running. Check it's log files...
The dispatcher is running - the Status Monitor shows good status, the bs_dispatch process is running.
Here are the latest logs from /srv/obs/log/dispatcher.log. Could you let me know if you see anything uncommon?
not really, check if your workers are still registered and in which state: ls /srv/obs/workers/*/*$hostname* you see them sorted in the matching directories based on your state. -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer
On 2021/09/15 Wed 17:21, Adrian Schröter wrote:
On Mittwoch, 15. September 2021, 15:37:25 CEST Kai Liu wrote:
On 2021/09/15 Wed 15:02, Adrian Schröter wrote:
On Mittwoch, 15. September 2021, 14:29:13 CEST Andreas Schwab wrote:
On Sep 15 2021, Kai Liu wrote:
The message I got was "waiting for 8 compliant workers (8 of them bad) ".
Use obs_admin --drop-badhosts to clear the badhosts markers.
well, yes, but the jobs will just fail then due to missing workers most likely.
or your dispatcher has a problem and is not running. Check it's log files...
The dispatcher is running - the Status Monitor shows good status, the bs_dispatch process is running.
Here are the latest logs from /srv/obs/log/dispatcher.log. Could you let me know if you see anything uncommon?
not really, check if your workers are still registered and in which state:
ls /srv/obs/workers/*/*$hostname*
you see them sorted in the matching directories based on your state.
The are all registered and idle, the same as showed in the Status Monitor. build:/srv/obs # tree /srv/obs/workers/ /srv/obs/workers/ ├── building ├── down └── idle ├── x86_64:build:1 ├── x86_64:build:2 ├── x86_64:build:3 ├── x86_64:build:4 ├── x86_64:build:5 ├── x86_64:build:6 ├── x86_64:build:7 └── x86_64:build:8 3 directories, 8 files 30mins ago I removed the repository I set on the project that holding the stale package. Now the Monitor page is empty, the package was gone. However, I don't know if it still exist somewhere like a ghost. Any way that I can check for "internal state" to make sure it's really gone, or cleanup the system manually?
--
Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer
On Mittwoch, 15. September 2021, 14:13:09 CEST Kai Liu wrote:
Hi experts,
I just set up a new private OBS instance and I've run into few frustrating problems. Could you give me some advice on how can I debug this problems, such as what logs to look at, how to reset things?
My setup is simple, on a single server running openSUSE Leap 15.3, with OBS 2.10 installed from the repository. Then setup by the /usr/lib/obs/server/setup-appliance.sh script. Then I noticed these issues.
Firstly, in the configuration UI, I tried to uncheck the i586 and armv7l architectures. But they keep coming back. I also tried removing them from /configuration API, still the same. This is really confusing.
that should be enough. However, only the next start will be influenced. A current running one will not be stopped.
Secondly, I connected my instance with the boo instance, and tried to start building some test packages. I soon noticed that my kernel and inird setup was not correct so the worker VM cannot boot into the guest OS. So I edited /etc/sysconfig/obs-server to use a different initrd. But I made a mistake here, I put the wrong path name there thus the VM cannot find the initrd. I can see build logs about the VM cannot start, as expected. The problem is OBS didn't think the build failed, instead it kept restarting the VM. Thus I have to abort the build. The abort succeeded. But later after I corrected the initrd path (and restarted the worker service), the package stuck in the "scheduled" state, never started building, even after I reboot the server.
The message I got was "waiting for 8 compliant workers (8 of them bad) ". The package was able to start building before I corrected the initrd path problem so there is no unmet constraint (I didn't put any in place either). I left it overnight and it still stuck there.
Then I thought I should delete this package to start from scratch again. This caused more problem... The package I deleted is no longer visible in the package list, but it still showed as "scheduled" in the monitor. And I have "scheduled: 1" on the web UI build result section...
It seems my instance is not in a consistent state now. But I can't find enough documentation to help me on debugging it. Any suggestions or hints or link to useful docs are greatly appreciated.
Thanks, Kai
-- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer
participants (3)
-
Adrian Schröter -
Andreas Schwab -
Kai Liu