On Tue, Dec 3, 2013 at 10:30 AM, Michal Hrusecky wrote:

Greg Freemyer - 9:34 3.12.13 wrote:

From here

...

--- create short term staging project - A BtrFS read/write snapshot of factory is created. This should only take a couple seconds

--- Update the affected packages based on the SR / SR batch - I assume a few more seconds

--- Run the build logic and build the updated packages and anything that depends on them - this is the time consumer, put hopefully this class of build can be given top priority.

to here we don't need special workflow as it would be handled by OBS

...

--- If no build errors, forward original SR / SR batch to factory reviewers and destroy BtrFS snapshot.

--- if build errors, hoover up the build logs from all the builds and attach them to the SR, reject the SR, and destroy the BtrFS snapshot.

If the above is totally automated, it seems the BtrFS snapshot may only live for a an hour or less typically.

With that process it seems the overhead of creating a staging projects is minimal and even single package SRs of leaf packages could be run through a quick staging compile / dependency compile. ===

Not a good idea to hack a new OBS from scratches living next to the official one. And I don't see what should be the difference between the short term and long term ones. Short ones are rejected right away without striving for solution in staging?

I'm not proposing a new OBS. I'm proposing the short term staging projects be created via BtrFS snapshots. I don't know how OBS manages the data, but I assuming it is on a different filesystem instance than the OBS application software. I guess I left the design goal in my head. I'm not expert with BtrFS, but I've been a storage engineer in my past and at least with other snapshot creating technologies I've worked with you had to design the implementation around the storage technology. As to the possibility of most SRs create a staging projects, that has 2 fundamental problems from my possibly naive perspective: - a human seems to be required in the process described so far. If most SRs are to be staged in some way, then for the vast majority of the SRs the whole thing from creation to destruction needs to be automated. The long term staging projects on the other hand are designed for human interaction. - Having dozens/hundreds of BtrFS snapshots in use simultaneously for a significant duration is probably a horrendous idea. It is much better to use linked packages in all likelihood. Thus it makes sense (to me) to treat the 2 use cases as separate with different goals and technical solutions: ======= short term staging projects ======== Goals: - Identify SRs that have BuildRequired dependencies in the devel project, but not in factory - Identify SRs that break other packages in factory that are not in the devel project. - For successful SRs, neither the submitter nor the factory reviewer should see any additional workload as compared to the current process - Must be very fast / efficient to create, execute, and destroy the staging project and totally automated. Technical assumption: - creating a staging project with 6,000 linked packages is slow / inefficient from a storage resource perspective - creating a staging project via a BtrFS snapshot is fast / efficient from a storage resource perspective Proposed solution: create a new SR automated workflow that uses BtrFS snapshots to - create a staging project, - runs the test builds - collect results - destroy the staging project.

...

-- How many "leaf" SRs per day are there anyway?

No idea.

...

-- How many simultaneous BtrFS snapshots can exist and still get reasonable performance?

No idea and doesn't matter.

It does for the proposal I made.

...

--- If one assumes 10 simultaneous BtrFS snapshots is reasonable to maintain, then a queue could be established to push leaf package staging projects through. If the average leaf staging project lives for 1 hour, then that's 240 SR / SR batches per day that could be treated this way..

Scheduling, building takes some time, there are other stuff, overall I think that we don't need staging for everything (although it might be nice).

Agreed that it is a nicety. When it was first proposed, my first thought was no way. Just the effort of creating all those staging projects would be rediculous. Then I thought about how efficient it would be to use BtrFS snapshots for that step and I changed my mind.

-- Michal HRUSECKY SUSE LINUX, s.r.o. Greg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org