Hi,

22.04 seems to suggest its still unreleased and under development?

You are right 22.04 is not yet released. The release date is set to April 21st, roughly 6 weeks from now and we wanted to be prepared as good as we can. We had the DoD import running for some time and were wondering why it broke so fundamentally for aarch64. The publicly available mirrors for jammy (22.04) can be used (I do tests with the day-to-day snapshots provided on AWS). Thus I think the repos are in an "ok" state to be consumable by obs

...

Doing s/5.32/5.34/g didn't help, only breaking x86_64, too. Do you have any tip/hint how to proceed here? Or where to look at for deeper debugging? Are there any binary packages cached for DoD?

As we are a little bit out of ideas what the root cause of the issue is, Jan and I would be interested how to debug the DoD import. Looking at the project which contains the Download on demand sources the only error I can see is: "The repository setup is broken, build or publish not possible" Do you have an idea how to get more details on why it is considered broken ? As Jan mentioned the dod log did not expose any obvious error and the projconf setup (osc buildconfig) did not expose a difference that would explain the error. Would be great to get some idea where to look at to understand the "why" for the broken state independently on what repo gets imported through DoD Thanks much Regards, Marcus -- Public Key available via: https://keybase.io/marcus_schaefer/key.asc keybase search marcus_schaefer ------------------------------------------------------- Marcus Schäfer Am Unterösch 9 Tel: +49 7562 905437 D-88316 Isny / Rohrdorf Germany -------------------------------------------------------

Hi,

the first place to look is if the dod update has any problems getting the meta data, check

/srv/obs/log/dodup.log

yep, thanks all information there looks good. All metadata and signatures could be fetched and read

...

"The repository setup is broken, build or publish not possible"

most likely no meta data, so OBS can not even detect the binary type of it.

After debugging I have more information which I hope helps. The project of the DoD consists out of the typical four Debian/Ubuntu repos: standard update universe universe-update The standard import looks clean but all others sends the "brokencode" which results in that "The repository setup is broken..." on the API The metadata set for a broken universe import as example looks like this: tree universe universe ├── aarch64 │   ├── :bininfo │   ├── :full │   │   ├── doddata │   │   └── doddata.cookie │   ├── :full.solv │   └── :schedulerstate └── x86_64 ├── :bininfo ├── :depends ├── :full │   ├── doddata │   └── doddata.cookie ├── :full.solv ├── :packstatus ├── :repodone ├── :repoinfo └── :schedulerstate As you can see the x86_64 import is ok and working. The aarch64 import is incomplete. The schedulerstate says: cat universe/aarch64/\:schedulerstate --- pst0 <unresolvable preinstalls: nothing provides perl-modules-5.32details brokencodee --- And here I didn't get further. I have checked all ":full.solv" solvables via dumpsolv to see if there is any reference to "perl-modules-5.32" in that solvables. There is none (5.34 is the version on 22.04) I also checked the prjconf for any occurence of "5.32", there is none. I have no idea where this reference to perl-modules-5.32 should come from. Do you have an idea ? what produces ":schedulerstate" ? Thanks for any pointer you can provide Regards, Marcus -- Public Key available via: https://keybase.io/marcus_schaefer/key.asc keybase search marcus_schaefer ------------------------------------------------------- Marcus Schäfer Am Unterösch 9 Tel: +49 7562 905437 D-88316 Isny / Rohrdorf Germany -------------------------------------------------------

Hi,

I don't think that's correct. universe/aarch64 is a DoD repo anyway, so why would the schedulerstate matter? The question is if the :full.solv file is complete.

Looking at the :full.solv of the universe repo it looks complete: -rw-r--r-- 1 obsrun obsrun 10439791 Mar 10 13:18 :full.solv dumpsolv of that file looks good to me

:schedulerstate is written by the scheduler. The "5.32" error comes from the projectconfig. So if "5.32" is no longer in the config, the scheduler has not looked at that repo for some time. What is the last modification date of the :schedulerstate file?

I also thought it might be because it has not yet computed the new state but looking at the ":schedulerstate" it has worked on it: -rw-r--r-- 1 obsrun obsrun 100 Mar 10 15:06 :schedulerstate We tested if the scheduler is actually working on the repo by deletion of all repos and adding them back. On the server the project we are testing this is in: /srv/obs/build/home:jalo271589:ubuntu:22.04/ ls -l drwxr-xr-x 4 obsrun obsrun 4096 Mar 10 13:18 standard drwxr-xr-x 4 obsrun obsrun 4096 Mar 10 13:17 universe drwxr-xr-x 4 obsrun obsrun 4096 Mar 10 13:17 universe-update drwxr-xr-x 4 obsrun obsrun 4096 Mar 10 13:17 update when I delete all repos, all four directories gets deleted. when I add the repos back, all four directories including its metadata gets created. So the scheduler worked on them and did its job. However, I also get that error about the unresolvable back and I don't understand why. As you told me it must be from the prjconf I also checked more than one time if I overlooked something. I then just made the prjconf empty: osc meta -e prjconf ==> delete all Still produces the same unresolvable for perl-modules-5.32 :-))

I think the aarch64 scheduler has not looked at that repo for some reason. Is the process running at all? What's in the scheduler logfile?

Please bear with me I'm new to that part of the obs code and functionality. I think the schedulers runs: 1629 ? S 11:30 /usr/bin/perl -w ./bs_sched aarch64 1630 ? S 123:52 /usr/bin/perl -w ./bs_sched x86_64 Where can I find the scheduler log ? Thanks Regards, Marcus -- Public Key available via: https://keybase.io/marcus_schaefer/key.asc keybase search marcus_schaefer ------------------------------------------------------- Marcus Schäfer Am Unterösch 9 Tel: +49 7562 905437 D-88316 Isny / Rohrdorf Germany -------------------------------------------------------

On Thu, Mar 10, 2022 at 05:37:11PM +0100, Marcus Schäfer wrote:

...

...

I think the aarch64 scheduler has not looked at that repo for some reason. Is the process running at all? What's in the scheduler logfile?

Here is the log excerpt from: scheduler_aarch64.log

--- 2022-03-10 15:06:03: [1629] looking at low prio home:jalo271589:ubuntu:22.04/standard (0/0/3/0/239) - home:jalo271589:ubuntu:22.04/standard scanning repo home:jalo271589:ubuntu:22.04/standard... sorting 0 packages checking packages building: 0, notready: 0, unfinished: 0 took 0 seconds to check the packages leaf prp, freeing data 2022-03-10 15:06:03: [1629] looking at low prio home:jalo271589:ubuntu:22.04/universe (0/0/2/0/239) - home:jalo271589:ubuntu:22.04/universe unresolvable preinstalls: nothing provides perl-modules-5.32

Ok, but that's home:jalo271589:ubuntu:22.04. I don't think this is your download on demand project, is it? osc buildconfig home:jalo271589:ubuntu:22.04 standard Cheers, Michael. -- Michael Schroeder SUSE Software Solutions Germany GmbH mls@suse.de GF: Felix Imendoerffer HRB 36809, AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}

Good morning. On Thu, 10 Mar 2022 18:07:04 +0100, Marcus Schäfer wrote:

Strange enough ":schedulerstate" for universe universe-update and update still says "<unresolvable preinstalls: nothing provides perl-modules-5.32" and it's not an old state, it has timestamp from the time of the action

The problem seems to have been solved (for now). https://build.opensuse.org/project/show/Ubuntu:22.04 confused me a bit as this still points to "impish" (not jammy) which is actually Ubuntu 21.10, right? However, the following looks quite good now: # grep -ol jammy /srv/obs/dods/ubuntu*aarch64 /srv/obs/dods/ubuntu:22.04::standard::aarch64 /srv/obs/dods/ubuntu:22.04::universe-update::aarch64 /srv/obs/dods/ubuntu:22.04::universe::aarch64 /srv/obs/dods/ubuntu:22.04::update::aarch64 # l /srv/obs/projects/ubuntu\:22.04.xml -rw-r--r-- 1 obsrun obsrun 25432 Mar 11 09:39 /srv/obs/projects/ubuntu:22.04.xml # l */**/:schedulerstate | grep aarch -rw-r--r-- 1 obsrun obsrun 123 Mar 11 10:48 standard/aarch64/:schedulerstate -rw-r--r-- 1 obsrun obsrun 123 Mar 11 10:48 universe-update/aarch64/:schedulerstate -rw-r--r-- 1 obsrun obsrun 123 Mar 11 10:48 universe/aarch64/:schedulerstate -rw-r--r-- 1 obsrun obsrun 123 Mar 11 10:48 update/aarch64/:schedulerstate # l */**/:full.solv | grep aarch -rw-r--r-- 1 obsrun obsrun 1137426 Mar 11 10:48 standard/aarch64/:full.solv -rw-r--r-- 1 obsrun obsrun 1018533 Mar 11 10:48 universe-update/aarch64/:full.solv -rw-r--r-- 1 obsrun obsrun 10472297 Mar 11 10:08 universe/aarch64/:full.solv -rw-r--r-- 1 obsrun obsrun 208 Mar 11 09:35 update/aarch64/:full.solv # grep unresolvable */**/:schedulerstate # Find current prj and prjconf in the attachment. Likely, this was an hickup on the Ubuntu repos. Thanks Jan

On Thu, Mar 10, 2022 at 06:07:04PM +0100, Marcus Schäfer wrote:

...

Ok, but that's home:jalo271589:ubuntu:22.04. I don't think this is your download on demand project, is it?

It is. Jan(jalo271589) and I are working on this issue and we don't wanted to destroy the root "ubuntu:22.04" project. So we created the same under Jan's home project for testing and fiddling with stuff

...

osc buildconfig home:jalo271589:ubuntu:22.04 standard

You can't imagine how often I have called this command today :-)

currently this is empty because I deleted all prjconf settings to be absolutely sure there is nothing that pulls in some perl-modules package.

-- %define _project home:jalo271589:ubuntu:22.04

### from home:jalo271589:ubuntu:22.04 %define _repository standard --

that's it

I don't see how this can work, you need the configuration for the base distribution, e.g. preinstalls and the like. You should get a "no build type" error because of this. Could you also share the output of: osc meta prj home:jalo271589:ubuntu:22.04 Thanks, Michael. -- Michael Schroeder SUSE Software Solutions Germany GmbH mls@suse.de GF: Felix Imendoerffer HRB 36809, AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}

On Freitag, 11. März 2022, 13:39:48 CET Jan Löser wrote:

On Fri, 11 Mar 2022 12:25:34 +0000, Michael Schroeder wrote:

...

Could you also share the output of:

osc meta prj home:jalo271589:ubuntu:22.04

I have already in a mail from this morning which still awaits approval from moderator:

The message is larger than the 40 KB maximum size

... because of attached prj and prjconf.

Anyway, looks like the problem is gone for now at least on our instance. Likely this was fixed in the dod repositories because we didn't changed obvious things.

One more thing: https://build.opensuse.org/project/show/Ubuntu:22.04 confused me a bit because it's still using "impish" release (21.10) and not "jammy" (22.04). You might want to do some s/impish/jammy/g.

The setup is not finished (it is therefore not announced yet and neither in the simple list of the webui). However, it is able to resolve the dependencies now. -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev