On Tue, Dec 3, 2013 at 3:54 PM, Michal Hrusecky
Greg Freemyer - 15:44 3.12.13 wrote:
On Tue, Dec 3, 2013 at 3:31 PM, Michal Hrusecky
wrote: Greg Freemyer - 13:51 3.12.13 wrote:
...
Agreed that it is a nicety. When it was first proposed, my first thought was no way. Just the effort of creating all those staging projects would be rediculous. Then I thought about how efficient it would be to use BtrFS snapshots for that step and I changed my mind.
BtrFS cannot help to create more staging projects more easily. Staging project is OBS project containing package and rebuilding whatever depends on the package. OBS project is some little data but mostly metadata in some database. Staging project is NOT a directory.
Okay, that surprises me. I assumed every package had a directory somewhere that held the tarball, spec, patches, etc.
It probably has as well, don't know about the internals that much, but staging project contains mostly metadata that needs to be interpreted, it can't live just without OBS.
If that isn't the case, how resource intensive is it to create a staging project? and to run thru the dependancy analysis to see what needs to build? The compiles themselves get pushed out to the build farm and I suspect OBS has the capacity for that part of the extra load.
Creating and calculating dependencies is not that big problem, main problem is waiting to get scheduled and waiting for an empty worker I would say. And to wait for all dependencies to get built. In general with infinite build power, it should be possible to create a staging project for everything. But in the real world, I would prefer using some rule of thumb.
Since OBS had all the build servers added to it last spring (I think) I have rarely had to wait on a build job. I have often had to wait on the scheduler and publisher. My impression is we have the ability to build almost 400 x86 packages at a time, and I'm guessing over 50,000 x86 packages get built every 24-hours. (I'm looking at the monthly graph of packages to build and noticing how fast we can go from 50,000 to zero Look at Nov. 16 and 18 as examples. For both almost 50,000 outstanding build jobs goes to almost zero in what looks like less than a day. ,https://build.opensuse.org/monitor On the otherhand, there is limited parallelism in the dependency checker / scheduler from my external view. Look at the graph for repositories to recalculate. The scale only goes to 80 and it has spent the vast majority of last month off the scale. So 250 leaf SR submissions a day being run through an automated compile won't tax the build farm itself all the much more than it already is. But my assumption is still that 250 staging project creations (with 6000 packages each) per day would be a heavy load to add to an already maxed out resource. The factory project already has the dependancy status of all 6,000 projects calculated, so it still seems to me that a more or less instant snapshot creator like BtrFS would somehow be able to greatly reduce the load of creating shadow copies. And at that point, I think we can drop this discussion until someone who knows the internal details of OBS steps in one way or the other. Greg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org