Mailinglist Archive: opensuse-features (368 mails)

< Previous Next >
[openFATE 305634] Debian-like dist-upgrade live system full version upgrade
  • From: fate_noreply@xxxxxxx
  • Date: Tue, 30 Jun 2009 15:35:10 +0200 (CEST)
  • Message-id: <feature-305634-121@xxxxxxxxxxxxxx>
Feature changed by: Stanislav Visnovsky (visnov)
Feature #305634, revision 121
Title: Debian-like dist-upgrade live system full version upgrade

openSUSE-11.2: Candidate
Priority
Requester: Mandatory
Projectmanager: Mandatory

Info Provider: Robert Davies (robopensuse)
Requested by: Federico Lucifredi (flucifredi)
Partner organization: openSUSE.org

Description:
With the 11.2 cycle, we want to offer users the ability to perform a
live system upgrade in the manner of Debian's dist-upgrade.
For the purpose of this cycle, we want to support dist-upgrade from the
previous version (11.1) only, as this is a sufficiently complicated
problem as is.
From the user's view, the difference is between being able to update
the system incrementally within the given version or service pack
running, to being ble to migrate with a system command ("zypper dup" or
similar) to a higher version altogether.
In the Debian experience, the set of base distributions is not
necessarily limited, but it has been Ubuntu's practice to define what
starting points other than "release n-1" are allowed (for instance, all
LTS versions are purported to be able to "apt dist-upgrade" to the top
of the line, although I have heard of problems trying to jump two years
- 6.06->8.10 - in a fell swoop in this manner :-).
In the openSUSE scope, we should aim to be able to "dup" between
incremental versions, starting from 11.1 to 11.2, and later 11.x to
12.0.

Relations:
+ - Fedora Upgrade Instructions (feature/id:
+ http://fedoraproject.org/wiki/YumUpgradeFaq)
+ - Debian Upgrade How-to (feature/id:
http://www.debian.org/releases/stable/i386/release-notes/ch-upgrading.en.html)
- Wiki page for implementation (url:
http://en.opensuse.org/Upgrade/11.2)

Business case (Partner benefit):
openSUSE.org: With the introduction of the Zypper stack to SLE, we
finally reached the state of a featureful (which YOU was not) and fast,
reliable (which ZLM was not) update stack in the platform.
For enterprise use, some tweaks are still desirable (changelogs,
rollback, ...) which we are looking at, as well as improvements on the
Enterprise management front, which we are working on with our SRM
colleagues.
The only really significant competitive feature we are missing at this
point is the Debian/Ubuntu dist-upgrade functionality, which has a
powerful psychological impact at the Enterprise level and a much more
tangible impact at the small user / single user level: many with no IT
department do use Ubuntu these days on the basis that "chasing" Fedora
and openSUSE along the six-month upgrade cycle is too much for them,
and feel they can save time on Ubuntu with the combination of dist-
upgrade and the longer LTS cycle.
The rationale for pursuing this is to revoke the special status of
coolness this functionality gives Ubuntu, and to terminate the negative
influence that may have on our SLE sales (from the expert's personal
opinion, the preference then easily spills into purchasing).

Discussion:
#1: Federico Lucifredi (flucifredi) (2009-01-07 20:42:15)
This is the #1 feature in the systems management scope for 11.2 - I
have no doubt it will be fun :-)

#4: Klaus Kämpf (kwk) (2009-01-09 11:32:53)
Passing to mls for technical evaluation (solver + autobuild)

#5: Michael Schröder (mlschroe) (2009-01-09 12:48:10)
Any hint on what features are currently missing?

#6: Klaus Kämpf (kwk) (2009-01-09 13:30:57) (reply to #5)
From the top of my head:
How to handle
* Library ABI changes (e.g. major gcc/g++ upgrades) ?
* Core package changes (e.g. devs.rpm to udev) ?
* Kernel changes (if application/deamons need a specific kernel abi,
dbus comes into mind) ?
* Failure handling (network breakdown, package update errors, ...) ?
* Booting of the new kernel ?


#7: Michael Schröder (mlschroe) (2009-01-09 14:45:09) (reply to #6)
And debian does this in some way?

#8: Robert Davies (robopensuse) (2009-01-17 12:43:42) (reply to #7)
The distupgrade feature differs from changing source list and doing
upgrade, so some utility process could be started. They may require
some documented procedure to be followed, for "tough" changes. But ABI
change, how long since ELF/glibc was introduced? C++ ABI ought not be
relevant for upgrade process, they are "just" applications and libaries
to be updated. Failure handling - you are told to back up your system
first, it is "best effort" not guaranteed. The less you have installed
the more likely it is to succeed. Debian offer choice of kernels, this
may "punt" the problem to a user controlled install and select from
GRUB menu. Debian have apt-cache possibility, to create central pool of
packages, to decrease liklihood of network troubles, as well as huge
number CD and DVD iso's.

#9: Thorsten Kukuk (kukuk) (2009-01-17 15:24:35) (reply to #8)
The glibc internal ABI changes frequently. Means, running applications
will continue to use the old glibc with the old ABI, but installed are
already the new plugins => running applications can crash.

#12: Robert Davies (robopensuse) (2009-01-31 15:36:30) (reply to #9)
But your upgrade tools are buggy if they are built to rely on
"plugins". If you try to dist upgrade a live system within GUI, with
all software running, and no breakage, you aim impossibly high. The
point is, Debian have had very positive user comments for years about
this feature, despite there being many caveats and no guarantee of
success.

#13: Robert Davies (robopensuse) (2009-01-31 15:42:55) (reply to #12)
That wasn't clear. The upgrade tools are built so they won't suffer
from changes. It is reasonable to ask user to not be running uncessary
applications during a dist upgrade.

#16: Duncan Mac-Vicar (dmacvicar) (2009-02-02 10:19:55) (reply to #12)
Thorsten is talking about general applications. Not plugins.

#10: Cristian Rodríguez (elvigia) (2009-01-20 03:36:38) (reply to #8)
ABI is important during the upgrade process, I have seen applications
crashing if you upgrade running a desktop enviroment, however we need
to worry more about other stuff first IMHO.

#11: Duncan Mac-Vicar (dmacvicar) (2009-01-29 16:51:48)
I think the "Debian" part of the title is missleading if what we want
is just to support dup officially.
I would like to see a list from some "Debian expert" on how to Debian
handles the issues Klaus described above. Appart of the network failure
(included in another feature: commit) I ignore (and did not experience
during my old Debian times) if Debian does anything special on dist-
upgrade which give them the honour to name this feature "Debian like".
Otherwise this feature should be closed as "done", (or just waiting for
the commit refactoring).
Federico, as you named the feature "Debian-like", It would be helpful
to know, appart of the download-first feature, what are you missing
from Debian so we can reach that point.

#14: Robert Davies (robopensuse) (2009-01-31 15:54:26) (reply to #11)
Technically you may be right, but that does not solve the perceived
problem for end users who want a very well tested, and documented
upgrade path. 11.1 shipped with release notes without solutions for
problems folk hit (eg) PAM stuff).

#15: Duncan Mac-Vicar (dmacvicar) (2009-02-02 10:22:12) (reply to #14)
Exactly, but if we don't know what specific problems are we hitting
during distupgrade the feature has no real work and specific problems
can be tracked as bugs.
FATE is not a place for "make this better" without concrete
requirements or "make an unfalible distupgrade".
If requester, prjmgr, or pm expects anything to be done in the area, we
need at least:
* A list of problems we hit during dist upgrade (I am aware of one
specific, which is download-install sequence if network goes down)
* The scope. As you mention, asking to close every application (or
service) is not much to ask. Therefore the scope has to be set, because
if the scope is "distupgrade and glbc upgrade while oracle process
hospital monitoring equipment transactions" then this is a very
expensive feature.


#17: Duncan Mac-Vicar (dmacvicar) (2009-02-02 10:21:02) (reply to #14)
btw, can you describe in technical detail the pam upgrade problem so we
can document it?

#18: Thorsten Kukuk (kukuk) (2009-02-02 10:26:58) (reply to #17)
Yes, the PAM maintainers would like to know more details, too.

#19: Robert Davies (robopensuse) (2009-02-02 13:45:04) (reply to #18)
There were a number of different forum poster's hitting it, just after
11.1 released (I think due to zypper dup not YaST upgrade and therefore
unreported to BugZilla). It is not something I had, because I did fresh
installs. Interestingly though I have used Debian less, I have done
several dist-upgrade's, simply because it is expected to work well. So
it is a mistake to focus on specific tech issues alone. This problem is
also about expectations and user perceptions. Announcing "support" for
dist upgrade and appealing for test, so work rounds can be found, for
release notes at time matters to.

#21: Cristian Rodríguez (elvigia) (2009-02-02 15:29:32) (reply to #19)
as you failed to elaborate on the problem, I will.. Some people end
with broken /etc/pam.d/login they have the following line
session required pam_resmgr.so but that module is gone, and rpm
upgrade does not remove that line (probably because that file has been
modified)

#24: JP Rosevear (jproseve) (2009-02-02 10:15:19) (reply to #21)
Isn't that simply a packaging problem?

#25: Klaus Kämpf (kwk) (2009-02-02 19:42:45) (reply to #24)
In this case, it looks like a packaging problem, indeed.
However, we had more complicated upgrade tasks in the past, like
dropping devs.rpm and running udev instead.
See Removing devs.rpm
(http://en.opensuse.org/Software_Management/Upgrade/Devs_Rpm) for a
discussion of the problem.

#30: Peter Poeml (poeml) (2009-03-10 15:31:10) (reply to #25)
Interesting discussion about that particular issue, but simple to
solve. What works is keeping devs.rpm, and/or simply rpm -e --justdb
it. (It's enough to do so later.)
The more complicated issues in the past where a case where the kernel
changed its major version, and some networking applications could not
start with the (still running) old kernel. Didn't prevent an online
upgrade over the network though, and applications could keep running.
When skipping several openSUSE versions, and differences have
accumulated, in can happen that an upgrade in one step is not possible.
It's not possible to upgrade from 9.0 to 10.1 without first updating to
9.1, due to the kernel. These issues are quickly found by
experimentation though.
Other than that, the most hurting issue during remote upgrades is the
difficulty to ensure persistent device naming for NICs, because that is
reimplemented / changed with each openSUSE version, the upgrade
procedure is not well documented, and funny surprises are the rule, and
it often simply doesn't work.
Other small hurdles are the increasing modularization of kernel, which
can lead to lack of support for chipsets and storage in the initrd,
package split-offs (mingetty) that lead to lack of essential components
after the upgrade. It pays to make sure that all packages are installed
which are in the "minimal" patter of that openSUSE version. And the
required but disappeared PAM module locked us out once also...
A word to all the non-believers who think that upgrades don't work: I
have upgraded dozens of systems remotely during the last decade. All my
productive systems are repeatedly upgraded that way since day one.
Unfortunately, the tools we have didn't help with this. Apart from
using rpm, I started doing this with yum a few years ago (if it works,
metadata is available, falling back to rpm if needed).
Last year I documented some steps:
http://poeml.de/~poeml/10.1-11.0-update/update-11.0.howto. With today's
conveniency of virtualization at hand, I actually did an upgrade by
experiment with a VM first, to minimize the risk, and documented all
the steps. Since I had to upgrade several 10.1 machines, this seemed
like a good idea to reduce the risk and save brain energy. The notes
from it are a good outcome, too.

#26: Klaus Kämpf (kwk) (2009-02-02 19:43:24) (reply to #21)
Whats the bug number for this problem ?

#20: Robert Davies (robopensuse) (2009-02-02 14:48:34)
On Scope, here are the Release Notes for Lenny -
http://www.uk.debian.org/releases/testing/amd64/release-notes/ 4.1.4 -
Prepare a safe environment for the upgrade - text mode or via ssh
login, Article on how to go from Etch to Lenny -
http://www.debianadmin.com/howto-upgrade-from-debian-etch-40-to-lenny-50.html


#27: Federico Lucifredi (flucifredi) (2009-02-04 15:32:01) (reply to
#20)
this is very helpful, we should look at the bumps that Debian hits while
doing this.
On a separate note, just tried to "yum upgrade" frlom RHEL 5.2 to RHEL
5.3, and it went without a hitch. There may be something to be gathered
from Fedora as well here.

#22: Manuel Trujillo (toomany) (2009-02-02 15:44:44)
I think this is a really helpful characteristic, because a lot of
people like me, use openSUSE as server, and is a true hell to upgrade
servers in any datacenter, at kilometers off distance. I hope this
feature will be a reallity in the next 11.2.

#23: Duncan Mac-Vicar (dmacvicar) (2009-02-02 15:46:27) (reply to #22)
Please, if you just want to express your favorites, use the voting
feature. Comments is for discussion around solving the problem itself.

#28: Javier Llorente (javierllorente) (2009-03-10 14:08:21)
Just a few comments... A few weeks ago, I upgraded one server from 11.0
to 11.1 via ssh using zypper dup. zypper and its related libraries have
to be upgraded first in order to avoid problems, especially if the
upgrade process is interrupted. Another issue besides the order of the
upgrade, would be restarting daemons and dealing with new .conf. Maybe
we could check if it has been edited, if not, then automerge it. Some
stuff could malfunction due to old config files. We could also have a
manual merge option. Another thing... wouldn't it be better to download
all the packages at once and install them afterwards instead of
downloading/installing package by package?
On a side note, I want to say that my upgrade went smoothly (rcpbind
wasn't installed but I consider that a small issue). zypper has been
improved a lot. It would be great if zypper dup was officially
supported as another upgrade method.

#29: Peter Poeml (poeml) (2009-03-10 14:35:19) (reply to #28)
Regarding config merge on update, I have played with two things. First,
the yum plugin called yum-merge-conf, which allows for interactive
merging of config files during the upgrade. Works and gets the job
done, and could be done in zypper as well, however from my own
experience I have to say that it protracts the update more than I
sometimes like, and I typically want to do this job in a separate step
(after the packages are updated). not protract the update for merging
all files.
So I rather use another approach, which is a script called rpmresolve
(https://build.opensuse.org/package/view_file?file=rpmresolve&package=servertools&project=home%
3Apoeml), which works in a similar way, based on presence of
rpmsave/rpmnew files. It opens vim in diff mode, lets you merge and
asks you which version you want to keep. Works well for me since years,
it just might require refinement of some things of the implementation
in a more generic way (e.g. calling $EDITOR with different option than -
d).



--
openSUSE Feature:
https://features.opensuse.org/305634

< Previous Next >
This Thread