[opensuse-buildservice] Backend storage questions
Hi, I am running an OBS private instance. Thanks for the great platform. I am curious about how OBS deals with sources storage on the backend. We will be linking git commits in a busy repository to automated package builds in OBS. Each git commit will therefore result in a new tarball being generated, and a new revision in OBS's revision control for that package. This will mean a lot of commits and a lot of tarballs. How will OBS deal with that, will we have disk space consumption issues in future? Looking on the backend, I see roughly how file storage works. At /srv/obs/sources/<pkg> I see that each file from each revision in the history is stored independently and wholly (no binary diffs at play, for example). Over time, it looks like this will use up enough disk space to cause us some problems. Will OBS ever purge files from ancient revisions? If I purge such files manually, will anything break beyond the ability to back through history to that particular revision? How do large instances e.g. openSUSE deal with this? Now looking at storage of resultant built binaries, it appears that OBS only keeps the most recently built version. Is this correct, and can this be tweaked? Thanks, Daniel -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am Mittwoch, 25. September 2013, 07:41:50 schrieb Daniel Drake:
Hi,
I am running an OBS private instance. Thanks for the great platform.
I am curious about how OBS deals with sources storage on the backend. We will be linking git commits in a busy repository to automated package builds in OBS. Each git commit will therefore result in a new tarball being generated, and a new revision in OBS's revision control for that package. This will mean a lot of commits and a lot of tarballs. How will OBS deal with that, will we have disk space consumption issues in future?
Looking on the backend, I see roughly how file storage works. At /srv/obs/sources/<pkg> I see that each file from each revision in the history is stored independently and wholly (no binary diffs at play, for example). Over time, it looks like this will use up enough disk space to cause us some problems.
Will OBS ever purge files from ancient revisions?
No, because we have the rule that we always have the source for all ever build binaries.
If I purge such files manually, will anything break beyond the ability to back through history to that particular revision?
there is some experimental code (written by B1-Systems) in git master obs_admin tool to remove source files. Check for --remove-old-sources option.
How do large instances e.g. openSUSE deal with this?
We buy hard disks :)
Now looking at storage of resultant built binaries, it appears that OBS only keeps the most recently built version. Is this correct, and can this be tweaked?
Yes, that is correct. We use release mechnism for binaries we do release for official maintained products. This creates an entire source and binary copy into another project/package container. Check openSUSE:12.3:Update project for an example. It is thinkable to add support to always keep X number of binaries, but no one works on that atm. bye adrian -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi, On Sep 25, 2013, at 15:41 , Daniel Drake <drake@endlessm.com> wrote:
Will OBS ever purge files from ancient revisions? If I purge such files manually, will anything break beyond the ability to back through history to that particular revision? How do large instances e.g. openSUSE deal with this?
Now looking at storage of resultant built binaries, it appears that OBS only keeps the most recently built version. Is this correct, and can this be tweaked?
We basically do the same: Use OBS for continuous integration tests and thus build a huge set of package multiple times every say (based on git commits). We had 1.5TB of pure source history in just 2 month and we did not want to by new storage every year ;). So we sponsored the development of a cleanup task in obs_admin (see Adrians last post). It has been developed by B1-Systems. We run this task as a cronjob once a day (in the night) and it does its job. -- mit freundlichen Gruessen/with best regards, Carsten Hoeger Open-Xchange GmbH -------------------------------------------------------------------------------- Open-Xchange AG, Rollnerstr. 14, 90408 Nürnberg, Amtsgericht Nürnberg HRB 24738 Vorstand: Rafael Laguna de la Vera, Carsten Dirks Aufsichtsratsvorsitzender: Richard Seibt European Office: Open-Xchange GmbH, Martinstr. 41, D-57462 Olpe, Germany Amtsgericht Siegen, HRB 8718, Geschäftsführer: Frank Hoberg, Martin Kauss US Office: Open-Xchange Inc., P.O. Box 143, Ardsley on Hudson, New York 10503 -------------------------------------------------------------------------------- -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (3)
-
Adrian Schröter -
Carsten Höger -
Daniel Drake