[yast-devel] SP3 and factory have different binary sources
Hi, during silent time between christmass and new year Ludwig contact me ( in CC ), that they have problem to see that we have same source for SP3 and factory. In general problem is that it is created by two different jenkins worker, so in the end tarballs have different timestamps and for scripts it is different tarballs unless they implement non-trivial comparison. Ludwig also have some ideas how to solve it. One is using link from OBS to IBS. Then remaining challenge is to create submit requests in IBS. One idea can be to have jenkins workers that only send sr. Another idea I have which can be even easier is to simply have job that compare version of SP3 and Devel:YaST:SP3 and if it differs, then create submit request. If we do it in reasonable times ( like once in a hour ), then I think it is good balance between performance and up to date source. We also have one more challenge and it is how to make external worker more reliable as currently we have only one external worker for yast, so we should add some redundancy here. It probably need some discussion if we can e.g. provide some of our VMs to operate on external network. Any more ideas? Or do you thing that we do not need to care about binary incompatibility? Josef -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 5.1.2017 v 10:04 Josef Reidinger napsal(a): [...]
Ludwig also have some ideas how to solve it. One is using link from OBS to IBS. Then remaining challenge is to create submit requests in IBS. One idea can be to have jenkins workers that only send sr. Another idea I have which can be even easier is to simply have job that compare version of SP3 and Devel:YaST:SP3 and if it differs, then create submit request.
Having links is probably a better idea because everybody can easily find out that the sources are the same and see the intention. Having some "magic" running in Jenkins (outside the OBS/IBS) might be a bit tricky for the people outside the YaST team. [...]
Any more ideas? Or do you thing that we do not need to care about binary incompatibility?
Um, what are actually the disadvantages of the current state? Does it make accepting the SRs more difficult? I personally do not care much about the exact match of the source tarballs, on package update we always build and send a completely new tarball anyway. (We do not use the old tarball + a patch.) -- Ladislav Slezák YaST Developer SUSE LINUX, s.r.o. Corso IIa Křižíkova 148/34 18600 Praha 8 -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
[if there were replies for this on the list only I didn't get them] Any update here? Josef Reidinger wrote:
during silent time between christmass and new year Ludwig contact me ( in CC ), that they have problem to see that we have same source for SP3 and factory. In general problem is that it is created by two different jenkins worker, so in the end tarballs have different timestamps and for scripts it is different tarballs unless they implement non-trivial comparison.
Ludwig also have some ideas how to solve it. One is using link from OBS to IBS. Then remaining challenge is to create submit requests in IBS. One idea can be to have jenkins workers that only send sr. Another idea I have which can be even easier is to simply have job that compare version of SP3 and Devel:YaST:SP3 and if it differs, then create submit request. If we do it in reasonable times ( like once in a hour ), then I think it is good balance between performance and up to date source.
Last week I talked with Adrian and mls. One open feature request for them is to somehow allow automatic triggering of submit requests within obs if a build succeeded. So in the longer run obs itself may help speeding this up.
We also have one more challenge and it is how to make external worker more reliable as currently we have only one external worker for yast, so we should add some redundancy here. It probably need some discussion if we can e.g. provide some of our VMs to operate on external network.
Any more ideas? Or do you thing that we do not need to care about binary incompatibility?
We do need to care, otherwise sources are simply not identical. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Wed, 18 Jan 2017 17:20:26 +0100
Ludwig Nussel
[if there were replies for this on the list only I didn't get them]
Any update here?
Hi, no update. I suggest to talk with lukas ocilka to move forward. As he is product owner and he prioritize work and we are currently quite overwhelmed with development of CAASP, so you need to convince him we should invest time to change way how we submitting stuff. Thanks Josef
Josef Reidinger wrote:
during silent time between christmass and new year Ludwig contact me ( in CC ), that they have problem to see that we have same source for SP3 and factory. In general problem is that it is created by two different jenkins worker, so in the end tarballs have different timestamps and for scripts it is different tarballs unless they implement non-trivial comparison.
Ludwig also have some ideas how to solve it. One is using link from OBS to IBS. Then remaining challenge is to create submit requests in IBS. One idea can be to have jenkins workers that only send sr. Another idea I have which can be even easier is to simply have job that compare version of SP3 and Devel:YaST:SP3 and if it differs, then create submit request. If we do it in reasonable times ( like once in a hour ), then I think it is good balance between performance and up to date source.
Last week I talked with Adrian and mls. One open feature request for them is to somehow allow automatic triggering of submit requests within obs if a build succeeded. So in the longer run obs itself may help speeding this up.
We also have one more challenge and it is how to make external worker more reliable as currently we have only one external worker for yast, so we should add some redundancy here. It probably need some discussion if we can e.g. provide some of our VMs to operate on external network.
Any more ideas? Or do you thing that we do not need to care about binary incompatibility?
We do need to care, otherwise sources are simply not identical.
cu Ludwig
-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Wed, 18 Jan 2017 17:20:26 +0100
Ludwig Nussel
[if there were replies for this on the list only I didn't get them]
Any update here?
One more idea, that can really helps in less expensive way ( regarding changes to our build system ). See current topic on @research mailing list ( internal sorry ). Maybe we can build our tarball with tar -cv --mtime=@0 --owner=0 --group=0 \ --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0,mtime:=0 \ -f tar2.tar h so it will be same even if created on different machines. Will this work for you? Josef
Josef Reidinger wrote:
during silent time between christmass and new year Ludwig contact me ( in CC ), that they have problem to see that we have same source for SP3 and factory. In general problem is that it is created by two different jenkins worker, so in the end tarballs have different timestamps and for scripts it is different tarballs unless they implement non-trivial comparison.
Ludwig also have some ideas how to solve it. One is using link from OBS to IBS. Then remaining challenge is to create submit requests in IBS. One idea can be to have jenkins workers that only send sr. Another idea I have which can be even easier is to simply have job that compare version of SP3 and Devel:YaST:SP3 and if it differs, then create submit request. If we do it in reasonable times ( like once in a hour ), then I think it is good balance between performance and up to date source.
Last week I talked with Adrian and mls. One open feature request for them is to somehow allow automatic triggering of submit requests within obs if a build succeeded. So in the longer run obs itself may help speeding this up.
We also have one more challenge and it is how to make external worker more reliable as currently we have only one external worker for yast, so we should add some redundancy here. It probably need some discussion if we can e.g. provide some of our VMs to operate on external network.
Any more ideas? Or do you thing that we do not need to care about binary incompatibility?
We do need to care, otherwise sources are simply not identical.
cu Ludwig
-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Josef Reidinger wrote:
On Wed, 18 Jan 2017 17:20:26 +0100 Ludwig Nussel
wrote: [if there were replies for this on the list only I didn't get them]
Any update here?
One more idea, that can really helps in less expensive way ( regarding changes to our build system ).
See current topic on @research mailing list ( internal sorry ). Maybe we can build our tarball with
tar -cv --mtime=@0 --owner=0 --group=0 \ --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0,mtime:=0 \ -f tar2.tar h
so it will be same even if created on different machines. Will this work for you?
No idea :-) You'd have to try to see if the result is identical. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 19.1.2017 v 14:20 Ludwig Nussel napsal(a): [...]
No idea :-) You'd have to try to see if the result is identical.
I have tried that tar command manually and it works, the recreated tarball has the same SHA1 sum. But it is be a bit tricky to incorporate this into the rake task. We use the standard Rake::PackageTask class for this, it allows to optionally specify the "tar" command, but it does not allow to pass additional parameters :-( I tried a trick with tar_command = "tar --mtime=@0 --owner=0 ..." but that does not work as the options must be placed after the "jcfv" command otherwise it fails with an argument error. So my PoC solution [1] simply creates the target directory and builds the tarball with Rake::PackageTask and then repackages the directory again using the extra tar options. The only drawback is that the files in the tarball have zero timestamps (01-01-1970), which IMO is not nice as sometimes they are useful :-( What do you think about it? [1] https://github.com/openSUSE/packaging_rake_tasks/pull/32 -- Ladislav Slezák YaST Developer SUSE LINUX, s.r.o. Corso IIa Křižíkova 148/34 18600 Praha 8 -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Fri, Jan 20, 2017 at 01:51:18PM +0100, Ladislav Slezak wrote:
Dne 19.1.2017 v 14:20 Ludwig Nussel napsal(a): [...]
No idea :-) You'd have to try to see if the result is identical.
I have tried that tar command manually and it works, the recreated tarball has the same SHA1 sum.
But it is be a bit tricky to incorporate this into the rake task.
We use the standard Rake::PackageTask class for this, it allows to optionally specify the "tar" command, but it does not allow to pass additional parameters :-(
I tried a trick with tar_command = "tar --mtime=@0 --owner=0 ..." but that does not work as the options must be placed after the "jcfv" command otherwise it fails with an argument error.
So my PoC solution [1] simply creates the target directory and builds the tarball with Rake::PackageTask and then repackages the directory again using the extra tar options.
The only drawback is that the files in the tarball have zero timestamps (01-01-1970), which IMO is not nice as sometimes they are useful :-(
Set the file timestamps to the last file modification in
git. That should give useful timestamps and identical tars.
ciao Arvin
PS: In general setting the timestamps is considered "stupid"
since it breaks 'make'.
--
Arvin Schnell,
participants (4)
-
Arvin Schnell
-
Josef Reidinger
-
Ladislav Slezak
-
Ludwig Nussel