[opensuse-buildservice] withbinaries copyproject without rebuild

newer
[opensuse-buildservice] New OBS...

older
[opensuse-buildservice] Is there a...

Kanstantsin Shautsou

11 Dec 2013 11 Dec '13

14:08

Hi, i need do copyproject without rebuild for binaries (and save rebuild counter for future rebuilds), could you describe what files should i care and where (bs_sched, bs_repserver, bs_srcserver)? * As first step i need fix withbinaries option for repserver so it copy all needed files for scheduler like :meta/$pack, history? Right? What files could be recreated by scheduler? Should i copy :relsync, .bininfo, :jobhistory, :full.solv, :bininfo? * As second step, project copy in src_server must be with saved rebuild counter… I don’t use commit revisions in rpm revisions, so commit history and revisions are not critical for me… (i don’t use withhistory) Could you comment my current changes https://github.com/KostyaSha/open-build-service/commits/fedora-2.4-copy ? Should i use “meta” variable to indicate that this copy is called from copyproject and not from other OBS internals? Should $pack.mrev be truncated during copy like .rev?-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Show replies by date

Kanstantsin Shautsou

12 Dec 12 Dec

10:44

New subject: [opensuse-buildservice] Re: withbinaries copyproject without rebuild

I did more debug and found : - .mrev has duplicated entries during creation - https://github.com/openSUSE/open-build-service/issues/529 - no need to increase revision when makeolder isn’t set - why you do renumbering for “withhistory”? - lsrev() and addmeta() are changing hashes for SERVICE entries => it enough to just copy existed .rev with MD5SUMS files - why do you need random generator for SERVICE entries? It can avoid new revision when nothing changed (related https://github.com/openSUSE/open-build-service/issues/278 ) - copying .rev without changes allow to be in sync with “history” entries, i think it make sense to pass “withhistory” option to rep_server so it copy “history” and meta files… -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Michael Schroeder

16:20

New subject: [opensuse-buildservice] Re: withbinaries copyproject without rebuild

On Thu, Dec 12, 2013 at 01:44:34PM +0300, Kanstantsin Shautsou wrote:

...

I did more debug and found : - .mrev has duplicated entries during creation - https://github.com/openSUSE/open-build-service/issues/529

Yeah, that's a bug in the API or WEBUI.

...

- no need to increase revision when makeolder isn?t set

Why not? That really depends on your use case. As copying always resets the "rebuild counter", we always bump the revision so that the freshly build packages are "newer" than the old ones.

...

- why you do renumbering for ?withhistory??

Because we can't have multiple revisions with the same id. If the package did not exist before, it will get the same ids.

...

- lsrev() and addmeta() are changing hashes for SERVICE entries => it enough to just copy existed .rev with MD5SUMS files

Not in all cases. Maybe if "noservice" is given. Otherwise, all services are re-run (or course only for the latest revision).

...

- why do you need random generator for SERVICE entries? It can avoid new revision when nothing changed (related https://github.com/openSUSE/open-build-service/issues/278 )

Because the result does not only depend on the package content, but also on the services configured in the project. And, maybe more important, if the service is of the "fetch this url" type, the result may be different files, so we need to have a different identifier for the result. That's not true for package links, here the result only depends on the link content, so we don't need to have to create unique ids.

...

- copying .rev without changes allow to be in sync with ?history? entries, i think it make sense to pass ?withhistory? option to rep_server so it copy ?history? and meta files?

checkin history and meta files only live on the source server, so passing that to the repo server makes no sense. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Kanstantsin Shautsou

17:02

New subject: [opensuse-buildservice] Re: withbinaries copyproject without rebuild

On Dec 12, 2013, at 19:20 , Michael Schroeder wrote:

...

On Thu, Dec 12, 2013 at 01:44:34PM +0300, Kanstantsin Shautsou wrote:

...
I did more debug and found : - .mrev has duplicated entries during creation - https://github.com/openSUSE/open-build-service/issues/529

Yeah, that's a bug in the API or WEBUI.

...
- no need to increase revision when makeolder isn?t set

Why not? That really depends on your use case. As copying always resets the "rebuild counter", we always bump the revision so that the freshly build packages are "newer" than the old ones. Rebuild counter will be right if i copy .rev without changes in srcserver and history+meta in repserver.

...

...
- why you do renumbering for ?withhistory??

Because we can't have multiple revisions with the same id. If the package did not exist before, it will get the same ids.

But this is a problem only in case when we want increase vrev (and that’s why do additional commit) that has no sense when we have makeolder that adds the second commit.

...

...
- lsrev() and addmeta() are changing hashes for SERVICE entries => it enough to just copy existed .rev with MD5SUMS files

Not in all cases. Maybe if "noservice" is given. Otherwise, all services are re-run (or course only for the latest revision).

Even with noservices.

...

...
- why do you need random generator for SERVICE entries? It can avoid new revision when nothing changed (related https://github.com/openSUSE/open-build-service/issues/278 )

Because the result does not only depend on the package content, but also on the services configured in the project. And, maybe more important, if the service is of the "fetch this url" type, the result may be different files, so we need to have a different identifier for the result.

No matter how service configured, you have resulted files from what you can do md5 and then create summary md5, so when service fetch the same files you will have the same hash. And this can avoid new MD5SUMS with the same content + avoid new revisions.

...

That's not true for package links, here the result only depends on the link content, so we don't need to have to create unique ids.

...
- copying .rev without changes allow to be in sync with ?history? entries, i think it make sense to pass ?withhistory? option to rep_server so it copy ?history? and meta files?

checkin history and meta files only live on the source server, so passing that to the repo server makes no sense.

:meta/$pack and $pack/history lives in /build/project/repo/arch/ directory so if i use withhistory+withbinaries+ no makeolder - i can have one in one copy of project. I think we can have all variants of copy (including without rebuild) using existed parameters...

...

Cheers, Michael.

-- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}

-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Michael Schroeder

17:14

New subject: [opensuse-buildservice] Re: withbinaries copyproject without rebuild

On Thu, Dec 12, 2013 at 08:02:39PM +0300, Kanstantsin Shautsou wrote:

...

On Dec 12, 2013, at 19:20 , Michael Schroeder wrote:

...
On Thu, Dec 12, 2013 at 01:44:34PM +0300, Kanstantsin Shautsou wrote:

...
I did more debug and found : - .mrev has duplicated entries during creation - https://github.com/openSUSE/open-build-service/issues/529

Yeah, that's a bug in the API or WEBUI.

...
- no need to increase revision when makeolder isn?t set

Why not? That really depends on your use case. As copying always resets the "rebuild counter", we always bump the revision so that the freshly build packages are "newer" than the old ones. Rebuild counter will be right if i copy .rev without changes in srcserver and history+meta in repserver.

Yes, *if* you copy everything in the reposerver you can get away with not bumping the revision. The "$packid/history" file containing the "rebuild counter" is currently not copied, though.

...

...
...
- why you do renumbering for ?withhistory??

Because we can't have multiple revisions with the same id. If the package did not exist before, it will get the same ids. But this is a problem only in case when we want increase vrev (and that?s why do additional commit) that has no sense when we have makeolder that adds the second commit.

What's vrev got to do with the revisions? My point is that when the package already exists in the target project, with have multiple "rev=1" entries.

...

...
...
- lsrev() and addmeta() are changing hashes for SERVICE entries => it enough to just copy existed .rev with MD5SUMS files

Not in all cases. Maybe if "noservice" is given. Otherwise, all services are re-run (or course only for the latest revision). Even with noservices.

That's fixed with the latest commits.

...

...
...
- why do you need random generator for SERVICE entries? It can avoid new revision when nothing changed (related https://github.com/openSUSE/open-build-service/issues/278 )

Because the result does not only depend on the package content, but also on the services configured in the project. And, maybe more important, if the service is of the "fetch this url" type, the result may be different files, so we need to have a different identifier for the result. No matter how service configured, you have resulted files from what you can do md5 and then create summary md5, so when service fetch the same files you will have the same hash. And this can avoid new MD5SUMS with the same content + avoid new revisions.

I think you're missing my point. If "noservice" is not used in the copy, it is expected that the servic is re-run and may create a different result.

...

...
That's not true for package links, here the result only depends on the link content, so we don't need to have to create unique ids.

...
- copying .rev without changes allow to be in sync with ?history? entries, i think it make sense to pass ?withhistory? option to rep_server so it copy ?history? and meta files?

checkin history and meta files only live on the source server, so passing that to the repo server makes no sense. :meta/$pack and $pack/history lives in /build/project/repo/arch/ directory so if i use withhistory+withbinaries+ no makeolder - i can have one in one copy of project.

$pack/history is the build history, not the source history. The "withhistory" parameter is used to copy the source history.

...

I think we can have all variants of copy (including without rebuild) using existed parameters...

Yes, the "copy binaries" part is currently a bit lacking. M. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Kanstantsin Shautsou

17:37

New subject: [opensuse-buildservice] Re: withbinaries copyproject without rebuild

On Dec 12, 2013, at 20:14 , Michael Schroeder wrote:

...

On Thu, Dec 12, 2013 at 08:02:39PM +0300, Kanstantsin Shautsou wrote:

...
On Dec 12, 2013, at 19:20 , Michael Schroeder wrote:

...
On Thu, Dec 12, 2013 at 01:44:34PM +0300, Kanstantsin Shautsou wrote:

...
I did more debug and found : - .mrev has duplicated entries during creation - https://github.com/openSUSE/open-build-service/issues/529

Yeah, that's a bug in the API or WEBUI.

...
- no need to increase revision when makeolder isn?t set

Why not? That really depends on your use case. As copying always resets the "rebuild counter", we always bump the revision so that the freshly build packages are "newer" than the old ones. Rebuild counter will be right if i copy .rev without changes in srcserver and history+meta in repserver.

Yes, *if* you copy everything in the reposerver you can get away with not bumping the revision. The "$packid/history" file containing the "rebuild counter" is currently not copied, though. In my modifications i copy and seems scheduler is fine.

...
...
...
- why you do renumbering for ?withhistory??

Because we can't have multiple revisions with the same id. If the package did not exist before, it will get the same ids. But this is a problem only in case when we want increase vrev (and that?s why do additional commit) that has no sense when we have makeolder that adds the second commit.

What's vrev got to do with the revisions? My point is that when the package already exists in the target project, with have multiple "rev=1" entries.

...
...
...
- lsrev() and addmeta() are changing hashes for SERVICE entries => it enough to just copy existed .rev with MD5SUMS files

Not in all cases. Maybe if "noservice" is given. Otherwise, all services are re-run (or course only for the latest revision). Even with noservices.

That's fixed with the latest commits.

...
...
...
- why do you need random generator for SERVICE entries? It can avoid new revision when nothing changed (related https://github.com/openSUSE/open-build-service/issues/278 )

Because the result does not only depend on the package content, but also on the services configured in the project. And, maybe more important, if the service is of the "fetch this url" type, the result may be different files, so we need to have a different identifier for the result. No matter how service configured, you have resulted files from what you can do md5 and then create summary md5, so when service fetch the same files you will have the same hash. And this can avoid new MD5SUMS with the same content + avoid new revisions.

I think you're missing my point. If "noservice" is not used in the copy, it is expected that the servic is re-run and may create a different result. I don’t see that services are running during copy… Or maybe they just didn’t produce new revision...

...
...
That's not true for package links, here the result only depends on the link content, so we don't need to have to create unique ids.

...
- copying .rev without changes allow to be in sync with ?history? entries, i think it make sense to pass ?withhistory? option to rep_server so it copy ?history? and meta files?

checkin history and meta files only live on the source server, so passing that to the repo server makes no sense. :meta/$pack and $pack/history lives in /build/project/repo/arch/ directory so if i use withhistory+withbinaries+ no makeolder - i can have one in one copy of project.

$pack/history is the build history, not the source history. The "withhistory" parameter is used to copy the source history. Right, but for copyproject we have entry point in backend(==src_server), that call repserver. My idea: path withhistory to repserver (so it copy additionally meta+history) when we have withbinaries+withhistory arguments. Then i can have one-in-one copy for withbinaries+withhistory+no make older, in other cases you can bump any revision and new hashes you want :)

...

...
I think we can have all variants of copy (including without rebuild) using existed parameters...

Yes, the "copy binaries" part is currently a bit lacking.

M.

-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Kanstantsin Shautsou

13 Dec 13 Dec

11:36

New subject: [opensuse-buildservice] Re: withbinaries copyproject without rebuild

Another issues... Copybuild is executed for every package and scheduler schedule build on first copied package, because other packages are not copied yet. So, i stop worker, run copy, wait while everything copied, stop scheduler, wipe build jobs, run scheduler back and it become happy. I see that everywhere is used link(), rep_server puts binaries to job dir with link and then scheduler picks them with link(). Finally there are 6 hard links for one binary (from every :repo, :full, $packid/). Am i right that If i corrupt binary in src project, then all binaries will be broken in dst? On Dec 12, 2013, at 20:37 , Kanstantsin Shautsou wrote:

...

On Dec 12, 2013, at 20:14 , Michael Schroeder wrote:

...
On Thu, Dec 12, 2013 at 08:02:39PM +0300, Kanstantsin Shautsou wrote:

...
On Dec 12, 2013, at 19:20 , Michael Schroeder wrote:

...
On Thu, Dec 12, 2013 at 01:44:34PM +0300, Kanstantsin Shautsou wrote:

...
I did more debug and found : - .mrev has duplicated entries during creation - https://github.com/openSUSE/open-build-service/issues/529

Yeah, that's a bug in the API or WEBUI.

...
- no need to increase revision when makeolder isn?t set

Why not? That really depends on your use case. As copying always resets the "rebuild counter", we always bump the revision so that the freshly build packages are "newer" than the old ones. Rebuild counter will be right if i copy .rev without changes in srcserver and history+meta in repserver.

Yes, *if* you copy everything in the reposerver you can get away with not bumping the revision. The "$packid/history" file containing the "rebuild counter" is currently not copied, though. In my modifications i copy and seems scheduler is fine.

...
...
...
- why you do renumbering for ?withhistory??

Because we can't have multiple revisions with the same id. If the package did not exist before, it will get the same ids. But this is a problem only in case when we want increase vrev (and that?s why do additional commit) that has no sense when we have makeolder that adds the second commit.

What's vrev got to do with the revisions? My point is that when the package already exists in the target project, with have multiple "rev=1" entries.

...
...
...
- lsrev() and addmeta() are changing hashes for SERVICE entries => it enough to just copy existed .rev with MD5SUMS files

Not in all cases. Maybe if "noservice" is given. Otherwise, all services are re-run (or course only for the latest revision). Even with noservices.

That's fixed with the latest commits.

...
...
...
- why do you need random generator for SERVICE entries? It can avoid new revision when nothing changed (related https://github.com/openSUSE/open-build-service/issues/278 )

Because the result does not only depend on the package content, but also on the services configured in the project. And, maybe more important, if the service is of the "fetch this url" type, the result may be different files, so we need to have a different identifier for the result. No matter how service configured, you have resulted files from what you can do md5 and then create summary md5, so when service fetch the same files you will have the same hash. And this can avoid new MD5SUMS with the same content + avoid new revisions.

I think you're missing my point. If "noservice" is not used in the copy, it is expected that the servic is re-run and may create a different result. I don’t see that services are running during copy… Or maybe they just didn’t produce new revision...

...
...
That's not true for package links, here the result only depends on the link content, so we don't need to have to create unique ids.

...
- copying .rev without changes allow to be in sync with ?history? entries, i think it make sense to pass ?withhistory? option to rep_server so it copy ?history? and meta files?

checkin history and meta files only live on the source server, so passing that to the repo server makes no sense. :meta/$pack and $pack/history lives in /build/project/repo/arch/ directory so if i use withhistory+withbinaries+ no makeolder - i can have one in one copy of project.

$pack/history is the build history, not the source history. The "withhistory" parameter is used to copy the source history. Right, but for copyproject we have entry point in backend(==src_server), that call repserver. My idea: path withhistory to repserver (so it copy additionally meta+history) when we have withbinaries+withhistory arguments. Then i can have one-in-one copy for withbinaries+withhistory+no make older, in other cases you can bump any revision and new hashes you want :)

...
...
I think we can have all variants of copy (including without rebuild) using existed parameters...

Yes, the "copy binaries" part is currently a bit lacking.

M.

-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

3794

Age (days ago)

3796

Last active (days ago)

List overview

Download

6 comments

2 participants

participants (2)

Kanstantsin Shautsou
Michael Schroeder