[yast-devel] SP3 and factory have different binary sources

older
[yast-devel] Pending Pull Requests

Josef Reidinger

5 Jan 2017 5 Jan '17

09:04

Hi, during silent time between christmass and new year Ludwig contact me ( in CC ), that they have problem to see that we have same source for SP3 and factory. In general problem is that it is created by two different jenkins worker, so in the end tarballs have different timestamps and for scripts it is different tarballs unless they implement non-trivial comparison. Ludwig also have some ideas how to solve it. One is using link from OBS to IBS. Then remaining challenge is to create submit requests in IBS. One idea can be to have jenkins workers that only send sr. Another idea I have which can be even easier is to simply have job that compare version of SP3 and Devel:YaST:SP3 and if it differs, then create submit request. If we do it in reasonable times ( like once in a hour ), then I think it is good balance between performance and up to date source. We also have one more challenge and it is how to make external worker more reliable as currently we have only one external worker for yast, so we should add some redundancy here. It probably need some discussion if we can e.g. provide some of our VMs to operate on external network. Any more ideas? Or do you thing that we do not need to care about binary incompatibility? Josef -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Show replies by date

Ladislav Slezak

9 Jan 9 Jan

10:50

Dne 5.1.2017 v 10:04 Josef Reidinger napsal(a): [...]

...

Ludwig also have some ideas how to solve it. One is using link from OBS to IBS. Then remaining challenge is to create submit requests in IBS. One idea can be to have jenkins workers that only send sr. Another idea I have which can be even easier is to simply have job that compare version of SP3 and Devel:YaST:SP3 and if it differs, then create submit request.

Having links is probably a better idea because everybody can easily find out that the sources are the same and see the intention. Having some "magic" running in Jenkins (outside the OBS/IBS) might be a bit tricky for the people outside the YaST team. [...]

...

Any more ideas? Or do you thing that we do not need to care about binary incompatibility?

Um, what are actually the disadvantages of the current state? Does it make accepting the SRs more difficult? I personally do not care much about the exact match of the source tarballs, on package update we always build and send a completely new tarball anyway. (We do not use the old tarball + a patch.) -- Ladislav Slezák YaST Developer SUSE LINUX, s.r.o. Corso IIa Křižíkova 148/34 18600 Praha 8 -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Ludwig Nussel

18 Jan 18 Jan

16:20

New subject: [yast-devel] Re: SP3 and factory have different binary sources

[if there were replies for this on the list only I didn't get them] Any update here? Josef Reidinger wrote:

...

during silent time between christmass and new year Ludwig contact me ( in CC ), that they have problem to see that we have same source for SP3 and factory. In general problem is that it is created by two different jenkins worker, so in the end tarballs have different timestamps and for scripts it is different tarballs unless they implement non-trivial comparison.

...

Ludwig also have some ideas how to solve it. One is using link from OBS to IBS. Then remaining challenge is to create submit requests in IBS. One idea can be to have jenkins workers that only send sr. Another idea I have which can be even easier is to simply have job that compare version of SP3 and Devel:YaST:SP3 and if it differs, then create submit request. If we do it in reasonable times ( like once in a hour ), then I think it is good balance between performance and up to date source.

Last week I talked with Adrian and mls. One open feature request for them is to somehow allow automatic triggering of submit requests within obs if a build succeeded. So in the longer run obs itself may help speeding this up.

...

We also have one more challenge and it is how to make external worker more reliable as currently we have only one external worker for yast, so we should add some redundancy here. It probably need some discussion if we can e.g. provide some of our VMs to operate on external network.

Any more ideas? Or do you thing that we do not need to care about binary incompatibility?

We do need to care, otherwise sources are simply not identical. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Josef Reidinger

19 Jan 19 Jan

08:38

New subject: [yast-devel] Re: SP3 and factory have different binary sources

On Wed, 18 Jan 2017 17:20:26 +0100 Ludwig Nussel wrote:

...

[if there were replies for this on the list only I didn't get them]

Any update here?

Hi, no update. I suggest to talk with lukas ocilka to move forward. As he is product owner and he prioritize work and we are currently quite overwhelmed with development of CAASP, so you need to convince him we should invest time to change way how we submitting stuff. Thanks Josef

...

Josef Reidinger wrote:

...
during silent time between christmass and new year Ludwig contact me ( in CC ), that they have problem to see that we have same source for SP3 and factory. In general problem is that it is created by two different jenkins worker, so in the end tarballs have different timestamps and for scripts it is different tarballs unless they implement non-trivial comparison.

...
Ludwig also have some ideas how to solve it. One is using link from OBS to IBS. Then remaining challenge is to create submit requests in IBS. One idea can be to have jenkins workers that only send sr. Another idea I have which can be even easier is to simply have job that compare version of SP3 and Devel:YaST:SP3 and if it differs, then create submit request. If we do it in reasonable times ( like once in a hour ), then I think it is good balance between performance and up to date source.

Last week I talked with Adrian and mls. One open feature request for them is to somehow allow automatic triggering of submit requests within obs if a build succeeded. So in the longer run obs itself may help speeding this up.

...
We also have one more challenge and it is how to make external worker more reliable as currently we have only one external worker for yast, so we should add some redundancy here. It probably need some discussion if we can e.g. provide some of our VMs to operate on external network.

Any more ideas? Or do you thing that we do not need to care about binary incompatibility?

We do need to care, otherwise sources are simply not identical.

cu Ludwig

-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Josef Reidinger

13:16

New subject: [yast-devel] Re: SP3 and factory have different binary sources

On Wed, 18 Jan 2017 17:20:26 +0100 Ludwig Nussel wrote:

...

[if there were replies for this on the list only I didn't get them]

Any update here?

One more idea, that can really helps in less expensive way ( regarding changes to our build system ). See current topic on @research mailing list ( internal sorry ). Maybe we can build our tarball with tar -cv --mtime=@0 --owner=0 --group=0 \ --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0,mtime:=0 \ -f tar2.tar h so it will be same even if created on different machines. Will this work for you? Josef

...

Josef Reidinger wrote:

...
during silent time between christmass and new year Ludwig contact me ( in CC ), that they have problem to see that we have same source for SP3 and factory. In general problem is that it is created by two different jenkins worker, so in the end tarballs have different timestamps and for scripts it is different tarballs unless they implement non-trivial comparison.

...
Ludwig also have some ideas how to solve it. One is using link from OBS to IBS. Then remaining challenge is to create submit requests in IBS. One idea can be to have jenkins workers that only send sr. Another idea I have which can be even easier is to simply have job that compare version of SP3 and Devel:YaST:SP3 and if it differs, then create submit request. If we do it in reasonable times ( like once in a hour ), then I think it is good balance between performance and up to date source.

Last week I talked with Adrian and mls. One open feature request for them is to somehow allow automatic triggering of submit requests within obs if a build succeeded. So in the longer run obs itself may help speeding this up.

...
We also have one more challenge and it is how to make external worker more reliable as currently we have only one external worker for yast, so we should add some redundancy here. It probably need some discussion if we can e.g. provide some of our VMs to operate on external network.

Any more ideas? Or do you thing that we do not need to care about binary incompatibility?

We do need to care, otherwise sources are simply not identical.

cu Ludwig

-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Ludwig Nussel

13:20

New subject: [yast-devel] Re: SP3 and factory have different binary sources

Josef Reidinger wrote:

...

On Wed, 18 Jan 2017 17:20:26 +0100 Ludwig Nussel wrote:

...
[if there were replies for this on the list only I didn't get them]

Any update here?

One more idea, that can really helps in less expensive way ( regarding changes to our build system ).

See current topic on @research mailing list ( internal sorry ). Maybe we can build our tarball with

tar -cv --mtime=@0 --owner=0 --group=0 \ --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0,mtime:=0 \ -f tar2.tar h

so it will be same even if created on different machines. Will this work for you?

No idea :-) You'd have to try to see if the result is identical. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Ladislav Slezak

20 Jan 20 Jan

12:51

New subject: [yast-devel] Re: SP3 and factory have different binary sources

Dne 19.1.2017 v 14:20 Ludwig Nussel napsal(a): [...]

...

No idea :-) You'd have to try to see if the result is identical.

I have tried that tar command manually and it works, the recreated tarball has the same SHA1 sum. But it is be a bit tricky to incorporate this into the rake task. We use the standard Rake::PackageTask class for this, it allows to optionally specify the "tar" command, but it does not allow to pass additional parameters :-( I tried a trick with tar_command = "tar --mtime=@0 --owner=0 ..." but that does not work as the options must be placed after the "jcfv" command otherwise it fails with an argument error. So my PoC solution [1] simply creates the target directory and builds the tarball with Rake::PackageTask and then repackages the directory again using the extra tar options. The only drawback is that the files in the tarball have zero timestamps (01-01-1970), which IMO is not nice as sometimes they are useful :-( What do you think about it? [1] https://github.com/openSUSE/packaging_rake_tasks/pull/32 -- Ladislav Slezák YaST Developer SUSE LINUX, s.r.o. Corso IIa Křižíkova 148/34 18600 Praha 8 -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Arvin Schnell

13:07

New subject: [yast-devel] Re: SP3 and factory have different binary sources

On Fri, Jan 20, 2017 at 01:51:18PM +0100, Ladislav Slezak wrote:

...

Dne 19.1.2017 v 14:20 Ludwig Nussel napsal(a): [...]

...
No idea :-) You'd have to try to see if the result is identical.

I have tried that tar command manually and it works, the recreated tarball has the same SHA1 sum.

But it is be a bit tricky to incorporate this into the rake task.

We use the standard Rake::PackageTask class for this, it allows to optionally specify the "tar" command, but it does not allow to pass additional parameters :-(

I tried a trick with tar_command = "tar --mtime=@0 --owner=0 ..." but that does not work as the options must be placed after the "jcfv" command otherwise it fails with an argument error.

So my PoC solution [1] simply creates the target directory and builds the tarball with Rake::PackageTask and then repackages the directory again using the extra tar options.

The only drawback is that the files in the tarball have zero timestamps (01-01-1970), which IMO is not nice as sometimes they are useful :-(

Set the file timestamps to the last file modification in git. That should give useful timestamps and identical tars. ciao Arvin PS: In general setting the timestamps is considered "stupid" since it breaks 'make'. -- Arvin Schnell, Senior Software Engineer, Research & Development SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

2663

Age (days ago)

2678

Last active (days ago)

List overview

Download

7 comments

4 participants

participants (4)

Arvin Schnell
Josef Reidinger
Ladislav Slezak
Ludwig Nussel

[yast-devel] SP3 and factory have different binary sources

tags

participants (4)