Mailinglist Archive: opensuse-factory (678 mails)

< Previous Next >
[opensuse-factory] Tumbleweed snapshot repositories
  • From: Jimmy Berry <jberry@xxxxxxxx>
  • Date: Tue, 25 Oct 2016 15:05:58 -0500
  • Message-id: <1959348.yad0fyZL7N@boomba.local>
A proposal for review and discussion:

In the same way that TW is released in snapshots the repo itself would benefit
from being presented by snapshot. Currently, if one wants to install a new
package or perhaps *-debuginfo packages problems can occur especially on TW
(and I have run into these both personally and with folks I support).

A new package brings in a bunch of new dependencies that are compiled against
newer versions of already installed dependencies and one ends up with an
unusable package or worse.

Alternatively, dependency checks may only be resolvable with a full system
update anyway. The end result is that a TW install can be unmodifiable in a
short period of time without updating to a newer snapshot first. This is
compounded when rolling back via snapper due to an issue that the user is
waiting to be resolved.

By providing separate repos per TW snapshot these issues can be
alleviated since new package installs will have been built using the
same deps and build environment the user is currently running. The
separate repos could be easily handled with either a modification to
zypper or a wrapper, but having it built into zypper would seem
preferable.

When the user wants to perform the equivalent of `zypper ref &&
zypper dup` the following would occur:

- zypper replaces any snapshotted repos with the newest version (or a specific
one)
- zypper ref
- zypper dup

The first step is the only part that could use some helper functions and could
be implemented in a variety of manors. Publishing a text file containing the
latest/current snapshot in each repo should make it easy for zypper to know
what is the latest, completed snapshot.

The repos could also be prefixed at a variety of levels. Just to
provide an example:

http://download.opensuse.org/tumbleweed/repo/oss/

could become:

http://download.opensuse.org/tumbleweed/20161013/repo/oss/

depending on the level at which it is implemented on the repo side.
Since everything else remains the same the amount of work required
should be rather minimal.

On the storage side, either entire copies could be used for simplicity or a
symlink structure setup where unchanged packages are symlinked to previous
snapshot and only the changed packages are present as a non-symlink. Given
there is concern for mirror requirements the symlink approach will most likely
be preferred.

Snapshot repos could be kept for some predefined length of time and
then discarded.

This would automatically work with snapper since the repo URLs would
be stored in zypp configuration and thus captured by btrfs snapshots.
Additionally, non-btrfs filesystems could still benefit from being
able to rollback by switching to older TW snapshot repos or remain on
their current snapshot when they are aware of an issue in newer TW
snapshots that affects them.

Devel project on OBS or third-party repositories using OBS should be able to
utilize the same mechanism for providing snapshotted repos if desired. I would
imagine a new parameter in the .repo file or similar to explicitly indicate
that the repo is snapshotted rather than attempting to detect based on URL.

Additionally, providing snapshots could resolve the issue of not being able to
tell when a snapshot is ready to be downloaded. Currently, there is a delay
between the time a snapshot is released and has been replicated to the
mirrors. As such one can update in the middle of this process with varying
results. Obviously, Factory has a snapshot point already defined, but doing so
for third-party or devel project may be a bit more tricky. It would be rather
nice to have zypper be able to quickly tell if all repos are updated for the
latest snapshot rather than having the manually try and assess the situation.

Probably the simplest and most consistent approach would be to snapshot devel
repos building against openSUSE:Factory/snapshot at the same point the
snapshot was created (or perhaps released). If packages fail to build against
the new snapshot they can be updated and will be included with the next TW
snapshot. Building against openSUSE:Factory will provide early warning.

Tying devel project to Factory/snapshot is probably the only reasonable
approach since otherwise each repo will have its own snapshot point/process
and it will become rather complex. If a package breaks on a new snapshot it
probably makes sense to not publish the old version, in the new snapshot repo,
that way users should see a vendor change. Blocking the whole project from
publishing could be a configuration option as well, but probably more than
required.

Since devel/third-party projects should build against openSUSE:Factory/
snapshot they have to start after Factory is done building and can potentially
not be completed by the time Factory/snapshot has published. Having all this
in place would make it clear when everything is ready for update.

- new TW snapshot repo available
- devel projects repo available for new snapshot

Snapshot repos have the side effect of providing repo "locking" which should
make updates with extra repos more reproducible and safe. Obviously, the non-
snapshotted repos that currently exist can remain along side and folks could
be free to choose.

All in all virtually everything needed to set this up is already in place and
would just require:

- a consensus on where to place snapshot in repo path
(probably influenced heavily by the technical aspects of OBS)
- light zypper addition to handle automatically switching repo URLs
- decide on snapshot point in devel/third-party repos

I look forward to your thoughts.

--
Jimmy
< Previous Next >