[opensuse-buildservice] Uploading huge sources

older
[opensuse-buildservice] Moving a...

Claudio Freire

8 May 2012 8 May '12

12:41

I have a few packages that are part of a game's data, and have huge source tarballs, around 600M combined. I just got the factory SR declined with the usual "no services allowed, use localrun". Thing is, in this case, there's no way I can upload the tarballs from my computer. I don't have the bandwidth or connection stability to do that. What options do I have? How can I get OBS to do the "localrun" for me? -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Show replies by date

Johannes Weberhofer

8 May 8 May

12:45

You could define a service which downloads the files from their original source. Also a re-compression of the files as bz2 or xz can be done by those services. There are two ways for downloading: Eighter the URL can be defined in the spec and a service to download all files can be defined; Or you define download services for the sources seperately. Best regards, Johannes Am 08.05.12 14:41, schrieb Claudio Freire:

...

I have a few packages that are part of a game's data, and have huge source tarballs, around 600M combined.

I just got the factory SR declined with the usual "no services allowed, use localrun". Thing is, in this case, there's no way I can upload the tarballs from my computer. I don't have the bandwidth or connection stability to do that.

What options do I have? How can I get OBS to do the "localrun" for me?

-- Johannes Weberhofer Weberhofer GmbH, Austria, Vienna -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Brian K. White

20:10

On 5/8/2012 8:45 AM, Johannes Weberhofer wrote:

...

Also a re-compression of the files as bz2 or xz can be done by those services.

Just a side note. I wish people weren't suggesting recompression/repackaging of upstream sources in source rpms. I understand that as long as we are talking about tars and cpios where the archive is distinct from the compression, and complete without the compression, unlike say .zip where the archive and the compression are not seperate or seperable, then it seems harmless to strip off the .gz and apply a .xz as long as you never actually touched the .tar, didn't un-tar and re-tar the actual files. You could view the .gz vs .xz as merely part of the transport of the tar, like ftp or http, or on-the-fly compression in http or ssh or modems, etc. But it still voids the upstream published checksums and makes the source untrustworthy for the end user. Wrapping would be ok from an integrity standpoint, containing a pristine original .zip within a .xz, but probably saves less space than repackaging and definitely costs more cpu. And in some cases like .zip the compression isn't a separate layer that can be removed without changing the contents anyways. It's better to have one absolute rule that you can always count on. "The included source is pristine". Not, "What you're looking at might be different from upstream but it's not changed in any way that matters we promise". It's just poor data integrity policy to allow it at all and allow for even the possibility of introducing some change in the upstream source, while still calling it a copy of the upstream source. Recompressing should be no more OK than changing the archive file name, changing the top level directory name, changing filename cases within the archive (from .zips created on case-insensitive windows boxes), stripping dos line endings, editing #includes and other references to reflect those changed directory and filenames, etc... That's all build-time stuff that should be done (and thus, documented) in spec/patches/scripts at build-time. Convince upstream to adopt .wonderzipperninethousand if it's worth it, but don't change what upstream publishes, at all, and still call it the upstream source. -- bkw -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Claudio Freire

20:23

On Tue, May 8, 2012 at 5:10 PM, Brian K. White wrote:

...

That's all build-time stuff that should be done (and thus, documented) in spec/patches/scripts at build-time.

Convince upstream to adopt .wonderzipperninethousand if it's worth it, but don't change what upstream publishes, at all, and still call it the upstream source.

I understand your points and agree in general. Still, there *are* policies specifying the preferred compression method of source packages. What then? -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Brian K. White

22:33

On 5/8/2012 4:23 PM, Claudio Freire wrote:

...

On Tue, May 8, 2012 at 5:10 PM, Brian K. White wrote:

...
That's all build-time stuff that should be done (and thus, documented) in spec/patches/scripts at build-time.

Convince upstream to adopt .wonderzipperninethousand if it's worth it, but don't change what upstream publishes, at all, and still call it the upstream source.

I understand your points and agree in general.

Still, there *are* policies specifying the preferred compression method of source packages.

What then?

Preferred just means preferred. If the upstream offers tar.gz, tar.bz2 and tar.xz, use the tar.xz If the upstream only offers .zip, I personally will not repackage that. I include the original .zip, and anything I need to do to the contents during/after unpacking, I do in the spec. I am putting forward that in those less clear situations between, where the upstream does not offer tar.xz, and something like .zip isn't involved that makes it easy to point out why you should preserve the original in that case, that it's a bad policy to tell people to repackage upstream sources. I don't deny such policy exists right now, I'm saying I think it's a bad one. -- bkw -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Claudio Freire

9 May 9 May

00:19

On Tue, May 8, 2012 at 7:33 PM, Brian K. White wrote:

...

I am putting forward that in those less clear situations between, where the upstream does not offer tar.xz, and something like .zip isn't involved that makes it easy to point out why you should preserve the original in that case, that it's a bad policy to tell people to repackage upstream sources.

I don't deny such policy exists right now, I'm saying I think it's a bad one.

Trying to argue against .zip, I'm beginning to agree and think it probably is. Size reductions are always marginal and don't warrant the repacking IMHO. The final rpm/deb will be compressed with whatever the build system prefers anyway, so this is only about source packages. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Paul Elliott

06:43

On Tuesday, May 08, 2012 05:33:29 PM Brian K. White wrote:

...

Preferred just means preferred.

If the upstream offers tar.gz, tar.bz2 and tar.xz, use the tar.xz

I just had a package fail to package on several distros, include some old suse distros, because it used xz. When I switched it to bz2, it packaged on all distros! I did not recompress anything, the upstream offered all three! I think this policy is dumb. -- Paul Elliott 1(512)837-1096 pelliott@BlackPatchPanel.com PMB 181, 11900 Metric Blvd Suite J http://www.free.blackpatchpanel.com/pme/ Austin TX 78758-3117

Marcus Meissner

09:21

On Wed, May 09, 2012 at 01:43:21AM -0500, Paul Elliott wrote:

...

On Tuesday, May 08, 2012 05:33:29 PM Brian K. White wrote:

...
Preferred just means preferred.

If the upstream offers tar.gz, tar.bz2 and tar.xz, use the tar.xz

I just had a package fail to package on several distros, include some old suse distros, because it used xz.

? How so? On some you might need to BuildRequire: xz to get it unpack.

...

When I switched it to bz2, it packaged on all distros!

I did not recompress anything, the upstream offered all three!

I think this policy is dumb.

Use whatever you think is best. openSUSE Factory meanwhile has the policy of "do not recompress yourself", so just include the smallest supported source repositoriy available from upstream. Ciao, Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Paul Elliott

16:32

On Wednesday, May 09, 2012 04:21:29 AM Marcus Meissner wrote:

...

On Wed, May 09, 2012 at 01:43:21AM -0500, Paul Elliott wrote:

...
On Tuesday, May 08, 2012 05:33:29 PM Brian K. White wrote:

...
Preferred just means preferred.

If the upstream offers tar.gz, tar.bz2 and tar.xz, use the tar.xz

I just had a package fail to package on several distros, include some old suse distros, because it used xz.

? How so? On some you might need to BuildRequire: xz to get it unpack.

Some distros do not have anything to satisfy this requirement or it has been renamed to something difficult to find.

...

...
When I switched it to bz2, it packaged on all distros!

I did not recompress anything, the upstream offered all three!

-- Paul Elliott 1(512)837-1096 pelliott@BlackPatchPanel.com PMB 181, 11900 Metric Blvd Suite J http://www.free.blackpatchpanel.com/pme/ Austin TX 78758-3117

Brian K. White

19:04

On 5/9/2012 2:43 AM, Paul Elliott wrote:

...

On Tuesday, May 08, 2012 05:33:29 PM Brian K. White wrote:

...
Preferred just means preferred.

If the upstream offers tar.gz, tar.bz2 and tar.xz, use the tar.xz

I just had a package fail to package on several distros, include some old suse distros, because it used xz.

When I switched it to bz2, it packaged on all distros!

I did not recompress anything, the upstream offered all three!

I think this policy is dumb.

I wasn't going as far as to call it dumb, just a mistake. It's reasonable on the surface. I just don't like the way that it chips away at the notion that keeping the upstream preserved as sacrosanct. IE it's better to have a rule that applies the same everywhere always, absolutely, that you just don't touch the original at all, rather than try to have fuzzy rules that it's ok to change some aspects but not others in some situations but not others. No, instill and preserve a habit that you simply always preserve the original, period. Then no one ever get's it wrong. It's part of robust procedure design. It also falls under "Make it easy for them to get it right". Otherwise in 3 years there will be entirely different people working for suse, who will not have been part of this conversation and who may not have been raised by people who learned the hard way that you must maintain good data hygiene habits or else you are left with details getting lost over time and details not being trustworthy later, and there will be some wizbang new archiver like star or dar or xar or afio, only even newer and possessing even more desirable new features than those, and these new kids will see that hey we felt free to implement the auto recompressor, why not also an auto rearchiver? The re-archived sources will be more efficiently searchable, arranged in blocks that somehow make them more efficient for raid/rsync/zfs/btrfs/snapshotting/flash-drives/etc and have random access without unpacking and have parseable xml headers and be able to handle exotic new file types and metadata that don't even exist right now etc etc ... we won't change the files, just unpack and repack and when they are unpacked again they will be exactly the same as if you had unpacked the original simple tar we promise..., you know except that the directories will have proper acl xattr's etc etc... in other words, not the same. -- bkw -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Cristian Morales Vega

8 May 8 May

12:47

On 8 May 2012 13:41, Claudio Freire wrote:

...

I have a few packages that are part of a game's data, and have huge source tarballs, around 600M combined.

I just got the factory SR declined with the usual "no services allowed, use localrun". Thing is, in this case, there's no way I can upload the tarballs from my computer. I don't have the bandwidth or connection stability to do that.

What options do I have? How can I get OBS to do the "localrun" for me?

When the change from _service files was announced this case was supposed to be handled correctly. But since the "download_files" service is in mode "disabled" in the openSUSE:Factory project I don't see how this can be solved. In fact, shouldn't it by in a "tryremote" mode (that doesn't exist)? -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Claudio Freire

12:55

On Tue, May 8, 2012 at 9:47 AM, Cristian Morales Vega wrote:

...

In fact, shouldn't it by in a "tryremote" mode (that doesn't exist)?

I wish... I've seen trylocal in the docs, no mention of tryremote. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stephan Kulow

14:39

On 08.05.2012 14:55, Claudio Freire wrote:

...

On Tue, May 8, 2012 at 9:47 AM, Cristian Morales Vega wrote:

...
In fact, shouldn't it by in a "tryremote" mode (that doesn't exist)?

I wish... I've seen trylocal in the docs, no mention of tryremote. There is no such way. All I can suggest is to run the service on my workstation and commit it for you.

I would love if there was a way to trigger services and then commit the result back, but it's not in the concept of services - we would need to add support for that into the webui. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Claudio Freire

15:59

On Tue, May 8, 2012 at 11:39 AM, Stephan Kulow wrote:

...

On 08.05.2012 14:55, Claudio Freire wrote:

...
On Tue, May 8, 2012 at 9:47 AM, Cristian Morales Vega wrote:

...
In fact, shouldn't it by in a "tryremote" mode (that doesn't exist)?

I wish... I've seen trylocal in the docs, no mention of tryremote. There is no such way. All I can suggest is to run the service on my workstation and commit it for you.

That would be very appreciated :-) vegastrike-data [0] is the heaviest, but -extra[1], -music[2] and -speech[3] would also be appreciated. Do you require me to give you maintainership?

...

I would love if there was a way to trigger services and then commit the result back, but it's not in the concept of services - we would need to add support for that into the webui.

trylocal should work, if done from the webui. Such a feature would be great. Otherwise I may have to pester you every time the tarball is updated. We wouldn't want that ;-) Another patch-to-be-developed added to the list I guess. [0] https://build.opensuse.org/package/show?package=vegastrike-data&project=games [1] https://build.opensuse.org/package/show?package=vegastrike-extra&project=games [2] https://build.opensuse.org/package/show?package=vegastrike-music&project=games [3] https://build.opensuse.org/package/show?package=vegastrike-speech&project=games -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Stephan Kulow

9 May 9 May

08:48

On 08.05.2012 17:59, Claudio Freire wrote:

...

On Tue, May 8, 2012 at 11:39 AM, Stephan Kulow wrote:

...
On 08.05.2012 14:55, Claudio Freire wrote:

...
On Tue, May 8, 2012 at 9:47 AM, Cristian Morales Vega wrote:

...
In fact, shouldn't it by in a "tryremote" mode (that doesn't exist)?

I wish... I've seen trylocal in the docs, no mention of tryremote. There is no such way. All I can suggest is to run the service on my workstation and commit it for you.

That would be very appreciated :-)

I hope I catched all. My hard disc ran full twice - this games doesn't seem to be for mortals :) Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Claudio Freire

12:08

On Wed, May 9, 2012 at 5:48 AM, Stephan Kulow wrote:

...

...
...
...
I wish... I've seen trylocal in the docs, no mention of tryremote. There is no such way. All I can suggest is to run the service on my workstation and commit it for you.

That would be very appreciated :-)

I hope I catched all. My hard disc ran full twice - this games doesn't seem to be for mortals :)

Thanks :-) I've been trying to make "osc service run" in sourceforge's shell servers, which ought to have the bandwidth, for next time. Not a walk in the park. I'll post instructions when I succeed :) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

Claudio Freire

8 May 8 May

16:49

On Tue, May 8, 2012 at 11:39 AM, Stephan Kulow wrote:

...

I would love if there was a way to trigger services and then commit the result back, but it's not in the concept of services - we would need to add support for that into the webui.

https://features.opensuse.org/313455 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org

4370

Age (days ago)

4371

Last active (days ago)

List overview

Download

16 comments

7 participants

participants (7)

Brian K. White
Claudio Freire
Cristian Morales Vega
Johannes Weberhofer
Marcus Meissner
Paul Elliott
Stephan Kulow