[opensuse-factory] Short term staging projects [WAS: One of the options for staging projects]
On Tue, Dec 3, 2013 at 10:30 AM, Michal Hrusecky <mhrusecky@suse.cz> wrote:
Greg Freemyer - 9:34 3.12.13 wrote:
From here
--- create short term staging project - A BtrFS read/write snapshot of factory is created. This should only take a couple seconds
--- Update the affected packages based on the SR / SR batch - I assume a few more seconds
--- Run the build logic and build the updated packages and anything that depends on them - this is the time consumer, put hopefully this class of build can be given top priority.
to here we don't need special workflow as it would be handled by OBS
--- If no build errors, forward original SR / SR batch to factory reviewers and destroy BtrFS snapshot.
--- if build errors, hoover up the build logs from all the builds and attach them to the SR, reject the SR, and destroy the BtrFS snapshot.
If the above is totally automated, it seems the BtrFS snapshot may only live for a an hour or less typically.
With that process it seems the overhead of creating a staging projects is minimal and even single package SRs of leaf packages could be run through a quick staging compile / dependency compile. ===
Not a good idea to hack a new OBS from scratches living next to the official one. And I don't see what should be the difference between the short term and long term ones. Short ones are rejected right away without striving for solution in staging?
I'm not proposing a new OBS. I'm proposing the short term staging projects be created via BtrFS snapshots. I don't know how OBS manages the data, but I assuming it is on a different filesystem instance than the OBS application software. I guess I left the design goal in my head. I'm not expert with BtrFS, but I've been a storage engineer in my past and at least with other snapshot creating technologies I've worked with you had to design the implementation around the storage technology. As to the possibility of most SRs create a staging projects, that has 2 fundamental problems from my possibly naive perspective: - a human seems to be required in the process described so far. If most SRs are to be staged in some way, then for the vast majority of the SRs the whole thing from creation to destruction needs to be automated. The long term staging projects on the other hand are designed for human interaction. - Having dozens/hundreds of BtrFS snapshots in use simultaneously for a significant duration is probably a horrendous idea. It is much better to use linked packages in all likelihood. Thus it makes sense (to me) to treat the 2 use cases as separate with different goals and technical solutions: ======= short term staging projects ======== Goals: - Identify SRs that have BuildRequired dependencies in the devel project, but not in factory - Identify SRs that break other packages in factory that are not in the devel project. - For successful SRs, neither the submitter nor the factory reviewer should see any additional workload as compared to the current process - Must be very fast / efficient to create, execute, and destroy the staging project and totally automated. Technical assumption: - creating a staging project with 6,000 linked packages is slow / inefficient from a storage resource perspective - creating a staging project via a BtrFS snapshot is fast / efficient from a storage resource perspective Proposed solution: create a new SR automated workflow that uses BtrFS snapshots to - create a staging project, - runs the test builds - collect results - destroy the staging project.
-- How many "leaf" SRs per day are there anyway?
No idea.
-- How many simultaneous BtrFS snapshots can exist and still get reasonable performance?
No idea and doesn't matter.
It does for the proposal I made.
--- If one assumes 10 simultaneous BtrFS snapshots is reasonable to maintain, then a queue could be established to push leaf package staging projects through. If the average leaf staging project lives for 1 hour, then that's 240 SR / SR batches per day that could be treated this way..
Scheduling, building takes some time, there are other stuff, overall I think that we don't need staging for everything (although it might be nice).
Agreed that it is a nicety. When it was first proposed, my first thought was no way. Just the effort of creating all those staging projects would be rediculous. Then I thought about how efficient it would be to use BtrFS snapshots for that step and I changed my mind.
-- Michal HRUSECKY SUSE LINUX, s.r.o. Greg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Greg Freemyer - 13:51 3.12.13 wrote:
...
Agreed that it is a nicety. When it was first proposed, my first thought was no way. Just the effort of creating all those staging projects would be rediculous. Then I thought about how efficient it would be to use BtrFS snapshots for that step and I changed my mind.
BtrFS cannot help to create more staging projects more easily. Staging project is OBS project containing package and rebuilding whatever depends on the package. OBS project is some little data but mostly metadata in some database. Staging project is NOT a directory. To compile everything needed, you need to schedule what comes first and what second. That is one part of what OBS does. Then you need to send a build job to the cluster of build machines and let them compile stuff. They also do some clever stuff not to recompile everything unless it is needed. I really can't see how is BtrFS related to that. -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Dec 3, 2013 at 3:31 PM, Michal Hrusecky <mhrusecky@suse.cz> wrote:
Greg Freemyer - 13:51 3.12.13 wrote:
...
Agreed that it is a nicety. When it was first proposed, my first thought was no way. Just the effort of creating all those staging projects would be rediculous. Then I thought about how efficient it would be to use BtrFS snapshots for that step and I changed my mind.
BtrFS cannot help to create more staging projects more easily. Staging project is OBS project containing package and rebuilding whatever depends on the package. OBS project is some little data but mostly metadata in some database. Staging project is NOT a directory.
Okay, that surprises me. I assumed every package had a directory somewhere that held the tarball, spec, patches, etc. If that isn't the case, how resource intensive is it to create a staging project? and to run thru the dependancy analysis to see what needs to build? The compiles themselves get pushed out to the build farm and I suspect OBS has the capacity for that part of the extra load. Greg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Greg Freemyer - 15:44 3.12.13 wrote:
On Tue, Dec 3, 2013 at 3:31 PM, Michal Hrusecky <mhrusecky@suse.cz> wrote:
Greg Freemyer - 13:51 3.12.13 wrote:
...
Agreed that it is a nicety. When it was first proposed, my first thought was no way. Just the effort of creating all those staging projects would be rediculous. Then I thought about how efficient it would be to use BtrFS snapshots for that step and I changed my mind.
BtrFS cannot help to create more staging projects more easily. Staging project is OBS project containing package and rebuilding whatever depends on the package. OBS project is some little data but mostly metadata in some database. Staging project is NOT a directory.
Okay, that surprises me. I assumed every package had a directory somewhere that held the tarball, spec, patches, etc.
It probably has as well, don't know about the internals that much, but staging project contains mostly metadata that needs to be interpreted, it can't live just without OBS.
If that isn't the case, how resource intensive is it to create a staging project? and to run thru the dependancy analysis to see what needs to build? The compiles themselves get pushed out to the build farm and I suspect OBS has the capacity for that part of the extra load.
Creating and calculating dependencies is not that big problem, main problem is waiting to get scheduled and waiting for an empty worker I would say. And to wait for all dependencies to get built. In general with infinite build power, it should be possible to create a staging project for everything. But in the real world, I would prefer using some rule of thumb. -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Dec 3, 2013 at 3:54 PM, Michal Hrusecky <mhrusecky@suse.cz> wrote:
Greg Freemyer - 15:44 3.12.13 wrote:
On Tue, Dec 3, 2013 at 3:31 PM, Michal Hrusecky <mhrusecky@suse.cz> wrote:
Greg Freemyer - 13:51 3.12.13 wrote:
...
Agreed that it is a nicety. When it was first proposed, my first thought was no way. Just the effort of creating all those staging projects would be rediculous. Then I thought about how efficient it would be to use BtrFS snapshots for that step and I changed my mind.
BtrFS cannot help to create more staging projects more easily. Staging project is OBS project containing package and rebuilding whatever depends on the package. OBS project is some little data but mostly metadata in some database. Staging project is NOT a directory.
Okay, that surprises me. I assumed every package had a directory somewhere that held the tarball, spec, patches, etc.
It probably has as well, don't know about the internals that much, but staging project contains mostly metadata that needs to be interpreted, it can't live just without OBS.
If that isn't the case, how resource intensive is it to create a staging project? and to run thru the dependancy analysis to see what needs to build? The compiles themselves get pushed out to the build farm and I suspect OBS has the capacity for that part of the extra load.
Creating and calculating dependencies is not that big problem, main problem is waiting to get scheduled and waiting for an empty worker I would say. And to wait for all dependencies to get built. In general with infinite build power, it should be possible to create a staging project for everything. But in the real world, I would prefer using some rule of thumb.
Since OBS had all the build servers added to it last spring (I think) I have rarely had to wait on a build job. I have often had to wait on the scheduler and publisher. My impression is we have the ability to build almost 400 x86 packages at a time, and I'm guessing over 50,000 x86 packages get built every 24-hours. (I'm looking at the monthly graph of packages to build and noticing how fast we can go from 50,000 to zero Look at Nov. 16 and 18 as examples. For both almost 50,000 outstanding build jobs goes to almost zero in what looks like less than a day. ,<https://build.opensuse.org/monitor> On the otherhand, there is limited parallelism in the dependency checker / scheduler from my external view. Look at the graph for repositories to recalculate. The scale only goes to 80 and it has spent the vast majority of last month off the scale. So 250 leaf SR submissions a day being run through an automated compile won't tax the build farm itself all the much more than it already is. But my assumption is still that 250 staging project creations (with 6000 packages each) per day would be a heavy load to add to an already maxed out resource. The factory project already has the dependancy status of all 6,000 projects calculated, so it still seems to me that a more or less instant snapshot creator like BtrFS would somehow be able to greatly reduce the load of creating shadow copies. And at that point, I think we can drop this discussion until someone who knows the internal details of OBS steps in one way or the other. Greg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (2)
-
Greg Freemyer -
Michal Hrusecky