[opensuse-buildservice] first draft of a dod implementation
Hi, here's a first (incomplete!) version of a patch which adds dod support to the backend. Before I implement the missing parts in bs_repserver I just want to know if this is the right way to do it (especially the parts in bs_sched and BSSolv.xs (the implementation in the repserver is just a quick hack to test the scheduler)). Note: currently it doesn't use the changes which were committed in r9280 because the scheduler rescans the repo directly if the repserver fetched new packages (although this might be problem in case the build finishes before the scheduler finishes the reposcan/processed the new event - I need to check if this can happen...). Feedback is welcome:) Marcus
On Monday 30 November 2009 22:38:50 Marcus Hüwe wrote:
here's a first (incomplete!) version of a patch which adds dod support to the backend.
Forgive me my ignorance, but what is dod?
--
Cornelius Schumacher
On 2009-12-01 12:17:46 +0100, Cornelius Schumacher wrote:
On Monday 30 November 2009 22:38:50 Marcus Hüwe wrote:
here's a first (incomplete!) version of a patch which adds dod support to the backend.
Forgive me my ignorance, but what is dod?
download on demand. -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Mon, Nov 30, 2009 at 10:38:50PM +0100, Marcus Hüwe wrote:
here's a first (incomplete!) version of a patch which adds dod support to the backend. Before I implement the missing parts in bs_repserver I just want to know if this is the right way to do it (especially the parts in bs_sched and BSSolv.xs (the implementation in the repserver is just a quick hack to test the scheduler)). Note: currently it doesn't use the changes which were committed in r9280 because the scheduler rescans the repo directly if the repserver fetched new packages (although this might be problem in case the build finishes before the scheduler finishes the reposcan/processed the new event - I need to check if this can happen...).
I think it's ok to rely on the rescan event.
Feedback is welcome:)
Ok, here are a couple of comments. I just glanced over the patch, so some/most may be wrong. 1) I don't think we need both 'rpath' and 'path' as they are not both set in the same solvable. 2) I think we don't need %dodprojs, we can check the 'download' entry in projpacks. 3) I think the updatefrombins() call will throw out all dod entries breaking addrepo_dod 4) the dod solvables should have a hdrid of "dododo..." 5) hasdodurl() should be just "dodurl()", as it returns the url ;-) Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2009-12-01 16:11:34 +0100, Michael Schroeder wrote:
On Mon, Nov 30, 2009 at 10:38:50PM +0100, Marcus Hüwe wrote:
here's a first (incomplete!) version of a patch which adds dod support to the backend. Before I implement the missing parts in bs_repserver I just want to know if this is the right way to do it (especially the parts in bs_sched and BSSolv.xs (the implementation in the repserver is just a quick hack to test the scheduler)). Note: currently it doesn't use the changes which were committed in r9280 because the scheduler rescans the repo directly if the repserver fetched new packages (although this might be problem in case the build finishes before the scheduler finishes the reposcan/processed the new event - I need to check if this can happen...).
I think it's ok to rely on the rescan event.
Ok.
Feedback is welcome:)
Ok, here are a couple of comments. I just glanced over the patch, so some/most may be wrong.
1) I don't think we need both 'rpath' and 'path' as they are not both set in the same solvable.
Hmm yes it should be possible to just work with path.
2) I think we don't need %dodprojs, we can check the 'download' entry in projpacks.
Ok.
3) I think the updatefrombins() call will throw out all dod entries breaking addrepo_dod
No I patched the updatefrombins() call so that it'll keep all packages which have no bsid (the id which is calculated from the stat() call). This way we keep the dod entries + add the additional packages from the :full tree (so in the worst case the number of solvables is doubled).
4) the dod solvables should have a hdrid of "dododo..."
Hmm why do we need this? e.g. for the getbinaryversions() call in bs_repserver? (at least in the scheduler this isn't needed because when it starts to calculate the meta all hdrid's are already available)
5) hasdodurl() should be just "dodurl()", as it returns the url ;-)
:) Thanks a lot for your comments. Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
3) I think the updatefrombins() call will throw out all dod entries breaking addrepo_dod
No I patched the updatefrombins() call so that it'll keep all packages which have no bsid (the id which is calculated from the stat() call). This way we keep the dod entries + add the additional packages from the :full tree (so in the worst case the number of solvables is doubled).
Ok, I missed that.
4) the dod solvables should have a hdrid of "dododo..."
Hmm why do we need this? e.g. for the getbinaryversions() call in bs_repserver? (at least in the scheduler this isn't needed because when it starts to calculate the meta all hdrid's are already available)
It makes thinks cleaner, you don't need to special case dod packages, e.g. the "$pkgid = "dodododododododododododododododo" unless $pkgid;" line in the scheduler. (Maybe we should set the buildservice_id to "dod" as well. Hmm.) I also think that currently dod repos are kind of slow, as addrepo_dod() doesn't look at the old solv file like addrepo_scan. We could move the dod handling into addrepo_scan, i.e.: print " scanning repo $prp...\n"; my $dir = "$reporoot/$prp/$myarch/:full"; my $cache; if (-s "$dir.solv") { eval {$cache = $pool->repofromfile($prp, "$dir.solv");}; warn($@) if $@; if ($cache && $cache->isexternal()) { $repodatas{$prp}->{'lastscan'} = time(); return $cache; } } else { # new code starts here: my ($projid) = split('/', $prp, 2); if ($projpacks->{$projid}->{'download'}) { my $doddata = $projpacks->{$projid}->{'download'}->[0]; eval {$cache = Meta::parse("$dir/$doddata->{'metafile'}", "$doddata->{'mtype'}", { 'arch' => [ $myarch ] })}; if ($@) { warn("cannot read metadata: $@"); return undef; } for (values %$cache) { delete $_->{'id'}; # hmm, set to dod? $_->{'hdrmd5'} = 'dodododododododododododododododo'; } $cache->{'/url'} = $doddata->{'baseurl'}; $cache = $pool->repofromdata($prp, $cache); } } This would convert the dod data only at the first time and afterwards just use the standard code to manage the non-dod solvables. The drawback is that you have to delete the solv file if you change the metadata. What do you think? Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2009-12-02 12:09:52 +0100, Michael Schroeder wrote:
4) the dod solvables should have a hdrid of "dododo..."
Hmm why do we need this? e.g. for the getbinaryversions() call in bs_repserver? (at least in the scheduler this isn't needed because when it starts to calculate the meta all hdrid's are already available)
It makes thinks cleaner, you don't need to special case dod packages, e.g. the "$pkgid = "dodododododododododododododododo" unless $pkgid;" line in the scheduler.
Ah ok that makes sense.
(Maybe we should set the buildservice_id to "dod" as well. Hmm.)
I'm fine with this too but then we also need to change the check in create_considered() so that it prefers solvables with a buildservice_id != "dod". But that's no problem it's just a strcmp()...
I also think that currently dod repos are kind of slow, as addrepo_dod() doesn't look at the old solv file like addrepo_scan. We could move the dod handling into addrepo_scan, i.e.:
This would convert the dod data only at the first time and afterwards just use the standard code to manage the non-dod solvables.
That sounds good and much cleaner:)
The drawback is that you have to delete the solv file if you change the metadata.
I think the "speedup" outweighs the drawback because addrepo_(dod|scan) is often called if you start with an empty dod repo. So if you have a couple of dod repos it might be "important". Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2009-12-02 15:02:47 +0100, Marcus Hüwe wrote:
On 2009-12-02 12:09:52 +0100, Michael Schroeder wrote:
4) the dod solvables should have a hdrid of "dododo..."
Hmm why do we need this? e.g. for the getbinaryversions() call in bs_repserver? (at least in the scheduler this isn't needed because when it starts to calculate the meta all hdrid's are already available)
It makes thinks cleaner, you don't need to special case dod packages, e.g. the "$pkgid = "dodododododododododododododododo" unless $pkgid;" line in the scheduler.
Ah ok that makes sense.
While incorporating the changes into a new patch I just noticed that we cannot use "dododododo..." as a hdrmd5 because "hexstr2bytes()" in repodata.c doesn't like the "o" - I just replaced it with "d0d0d0d0...". I'll post the new patch in the next few days. Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2009-12-02 21:49:32 +0100, Marcus Hüwe wrote: <SNIP>
I'll post the new patch in the next few days.
Here's a new version of the patch (still misses some bits in bs_repserver): * incorporated all changes (I hope I didn't forget anything) * rewrote the "fetchbinary" mechanism in bs_repserver: * only serve one request for a given project+repo+arch at a time and serve the other requests for the same project+repo+arch afterwards (this way we can guarantee that the repserver doesn't download the same binaries multiple times) * added support for redirects (status 302) to BSWatcher.pm (if this idea or the implementation is nonsense please tell me and we can switch back to the old method:) ). Feedback, objections etc. are welcome. Marcus
participants (4)
-
Cornelius Schumacher -
Marcus Hüwe -
Marcus Rueckert -
Michael Schroeder