Sam Clemens wrote:

In your original post, you, perhaps inadvertently, implied that the machine couldn't be taken offline to do a fresh install.

Well --... it "can't" because it would cause me too much grief. I *would* like to get to having a redundant, failover system, but it hasn't been a priority.

Wholesale replacement of basic system libraries will end in utter chaos, as the install system nees the new libraries, and the older software needs the old libraries.

---- This is not a black & white issue. It's not "work this way or not work at all". Third party application vendors are not going tolerate their apps breaking everytime the OS upgrades. Suse has included a "compat" package for older glib-version programs for some time now. If you follow the package dependencies, the old programs might require lib-v2, while new glib might be at lib-v3 -- but "compat" will still contain lib-v2 -- so if you install the new glib and compat at the same time, nothing breaks.

If your in-place software forks off ANY child processes, and their required libraries are missing, the whole server is going to be worthless for the rest of your installation procedure anyways.

This is not windows....a *fork* duplicates the parent space. If the libraries were in the parent, they will be duplicated into the child's address space. If you run a new program, it will pick up whatever *new* library is needed -- two different versions of the library can be in memory at the same time -- processes using the same version of a library can share read-only segments, but if another group uses a different version, that different version is in a different location in memory. The *old* library *stays* around on disk until the last processes using that library exits. Only after all processes using an old library have exited will the inode for that file be released and the file space be reclaimed. This is not windows -- where the libraries in memory are mapped to device,sector. In unix it is relative to the libraries inode -- which doesn't go away when deleted if it is still being used in memory.

Updating a kernel compiled against the same libraries is significantly different than upgrading all of the libraries.

1. The new kernel is only copied into place...it doesn't become active until AFTER the reboot.

The new libraries don't become active until you launch new programs. The programs already in memory continue to use any old libraries they have open -- only after you "reboot the program" (stop & restart it) will it pickup the new libraries -- JUST like the kernel. It's not random.

2. Replaced libraries (old ones removed) causes significant chaos -- the reason is that the system is "live"...and the old executables are expecting the old libraries, not the replacement libraries.

---- Well -- it depends on what you mean and how you go about it. Did you pay attention to the rpm requisites? If you do, the old programs will complain if you take away their old library, but if you upgrade glib with a new version and add a "compat" that deals with previous versions (which SuSE does), then you don't have the problem.

I once made the mistake of trying to upgrade glibc5 to glibc6. The result was not pleasant.

---- If you ignored the pre-reqs, I could see why. OTOH, I have also replaced libraries -- even going from one ARCHITECTURE to another (i386 to x86_64)...even that was doable (though a bit messy) which is why I'm wondering about scripts to aid in such things.

And I've been a *nix person since the early 1980's.

--- Maybe earlier than me...not sure....um...late 80's maybe? I admit, much of my experience has been working at unix vendors, so maybe I have had an insider's view more than most. But most of the people around me thought this stuff was pretty basic.

glibc is not considered to be "user level software".

From my perspective, it is -- programs run in kernel space or user space. Glibc is used and linked to programs running in user-space, not in kernel space. Glibc is an application library that can run on linux or on BSD or on other *nix'es. It's designed to be non-system specific. That pretty much puts it in user space on any system I'm familiar with.

It's intrinsic to nearly everything except the kernel itself, and that ONLY because the kernel statically links whatever it needs.

So dynamically loading modules don't exist in the kernel? Or in Windows? or...whereever? The kernel doesn't statically linkk everything -- it uses a specific list of functions that are known to exist and stay in kernel space. It's not happenstance nor just a change of a flag to static -- they are different versions than one exist in the user-level glibc.

In every place I've ever worked "why not upgrade?" is not a sufficient reason to upgrade a server.

I gave several reasons beyond that,

If it's doing what it needs to do, then there's no reason to upgrade.

But I always want them to do more -- I'm a software engineer, computer scientist. I'm not IT-support. So there is some importance in not being too out of date. Suse9.3, for example had a kernel that didn't provide very good SATA support. Neither does it provide SAS support. As I already wrote -- I upgrade to support newer software and hardware.

But, also, at every place I've ever been at, when an upgrade of that sort comes around, it's not done on a production machine while it's in use. The upgrade is FIRST done on a test machine, and tested for days to weeks. Once the test configuration is approved, THEN it's put onto the production machine.

Actually ... have been running 10.3 for ... several months on multiple machines. My 9.3 server is the last system to be upgraded because of stability concerns. It wasn't until I used 10.3 for a few months and upgraded one of the platforms from i586 suse->x86_64 (also live). My server is the *Last* machine to be upgraded....not the first. Still don't want to take it off line for an upgrade that can and usually does take several days, minimum, up to several weeks (depending on what else is going on). By not taking it down, I can leave it half way upgraded -- but running and serviceable, and not stress out about not having my main server offline (which would through my other systems off line as well, it serving DNS, NFS, SMB-domain master, http-proxy(squid), SOCKS, file-system backup, email server...among a few other functions... So rather than changing everything on the system at once, which might result in the system not being able to due its normal duties for some period -- I do the upgrade piecemeal and make sure each important new piece works before I go on to break, er, upgrade something else! :-)

If you were using industry standard testing procedures, you would have worked out this upgrade process already on the non-production machine, and therefore, wouldn't be asking this question on this list.

Asking which question? If anyone else does piecemeal upgrades? OR if anyone else has or uses any tools that assist them? Or if anyone else would find such programs useful?

So, in summary, your company needs to become acquianted with the concept of "test platform" vs "production platform"

If only the budget would allow such find distinctions... though, in truth it's better than it was. At one point the server was also a prime compile and development machine as well as being used for testing. I'm sure you are familiar with the oft' repeated statement that most people don't make even close to full use of their hardware -- that it just sits...well... that's not usually me.

Untested installations, and especially upgrade procedures are NEVER done on a production machine. That's *WHY* you should have test machines.

I've been running 10.3 for some time on test machines. I just haven't upgraded my most stable server exactly because of the phrase "IF IT AIN'T BROKE...etc"... But there comes a time when its time to move forward...*carefully* -- not by reformatting the system partition with a new install (which is what would be required to go from 9.3->10.3), but one "step" at a time, so as not to break things. One "step" doesn't necessarily mean 1 single rpm package. I seem to remember one rpm command having around 30-35 required packages on the same command line. At the time, I had to track those deps down manually. (It's not like the rpm has a "what-provides" query feature that works to find out what rpm's need to be added to an rpm line to make an install work. RPM's "what provides" query only works "retrospectively" on what's already installed :-(.

Are you finding software bugs in your 9.3 software?

Not so much lately, but I did report ones found when 9.3 was newer. I continue to do so for 10.x. But I am finding software deficiencies in 9.3 -- not so much support for SATA or SAS, firewire-800 also required a newer kernel.

...

Not a bad sentiment, but this is the computer industry.

Same holds true.

One Fortune 50 company I worked at had an HP-UX server which was so old HP sent us a letter telling us that this particular machine would no longer be covered under the site maintenance contract, because the hardware was more than 10 years old.

And how many releases back will SuSE support upgrading from? Only 1. How long does MS claim they'll support older software? About 2 years if you are lucky. Does HP even still market HPUX? I thought it was near dying -- that they'd been switching too linux for the most part. Most of the big-iron companies with 10 year support contracts are having financial difficulties. It becomes increasingly costly to support old or out-of-date hardware. Even my server -- when I topped out it's memory, the chips for it were already twice as high as newer chips because it was from the previous generation...supporting older HW gets increasingly costly. If it died, I'd like just replace it with one of the newer workstations (used as test machine) that's about 12x faster.... But it's a source of "pride" being able to keep old hardware running and *useful* --- not having to waste money on new systems when not needed.

Upgrading IT solely for the purpose of upgrading is one of the surest ways to cause chaos in the workplace -- AND make the I.T. people look really, REALLY bad.

I, and the machine are not part of the IT department -- I and the machine are in research.

Stop viewing the OS installation as a fashion statement. Your girlfriends at the bar aren't going to be whispering to each other, "Can you BELIEVE it...she's using LAST SPRING's software."

It really is more about what color desktop you are sporting...whether or not it color coordinates with your outfits, and is in style...etc....:)

...

choose to move ahead -- on my schedule when its convenient for me. That's the wrong schedule.

NO...it's the right schedule -- because I do things when I am capable of doing them given time, resources, energy money...etc.

You need to do it when it's convenient for the business. That's why you're employed.

It's convenient when I have the time...:-) It wouldn't be convenient, ever, if my upgrade knocked my other systems off the net for a few hours (unless I'm very swamped dealing with the problem, in which case, I won't notice it...:-)).

IF it's necessary to upgrade this machine now, then you need to hop on a plane, and do this upgrade properly.

If only I got paid enough to do that...*sigh*...

At this point, I'm leaving the remaining 50% of this unaddressed, because I have other things I need to do.

That's fine -- as this is mostly an "intellectual" discussion for me at this point. The original question was trying to discern if anyone else worked this way, had utils they used, or knew of, or if they would find such utils helpful. or if "anyone else" was "interested in this subject". Or if there was a better place to ask and talk about this -- For whatever reason, the people on this list aren't very open to experimental hacking -- it seems like more people are focused on, or coming from IT issues rather than exploring what one can do and how one might improve on things. If people don't think about what works, what doesn't work, and what's a pain, things won't change or improve. It is real unfortunately, but most people (myself included at times) have lost any sense of wonder or fun or sense of exploring and testing what works and what doesn't. It's not about "right" and "wrong" or "black" and "white", but finding "better" ways to do things (where "better" is obviously a multi-value variable)... -l -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Sam Clemens wrote:

Linda Walsh wrote:

...

Sam Clemens wrote:

...

In your original post, you, perhaps inadvertently, implied that the machine couldn't be taken offline to do a fresh install.

---

Well --... it "can't" because it would cause me too much grief. I *would* like to get to having a redundant, failover system, but it hasn't been a priority.

...

Wholesale replacement of basic system libraries will end in utter chaos, as the install system nees the new libraries, and the older software needs the old libraries.

---- This is not a black & white issue. It's not "work this way or not work at all".

Linda, do yourself a favor, and STOP BEATING YOUR HEAD AGAINST THIS BRICK WALL because it's *not* going to crumble.

What part of "all ready done it; it works" do you not understand? I don't know how many times I've said it -- and others in this thread have done it going between different dist-vendors. It's been done, I've done it. I'll do it again. Get beyond "can't be done". It's already happened. The suse 9.3 server I've been talking about is already mostly converted... This is from the current running system -- its not a simulation -- try it on your system and see what comes up:

rpm -qa --queryformat="%{distribution}\n"|sort|uniq -c 11 (none) 1 SuSE Linux 9.1 (i586) 88 SuSE Linux 9.3 (i586) 1 SuSE Linux 9.3 (i686) 9 openSUSE 10.2 (i586) 670 openSUSE 10.3 (i586) 2 openSUSE 10.3 (i686)

(uptime 19 days at this point)... How can you claim this can't be done or that it doesn't work? This isn't a hypothetical and it's not the first time.

You cannot replace selected parts of an engine while the entire engine is running. This goes for tightly bound computer systems (in this case, ANYTHING which uses glibc) as much as physical objects.

Sorry for bothering you. I really did say my message was addressed to others who did things the way I did. I didn't ask for a debate on whether or not it could be done. Whether or not it can be done is "irrelevant". You live in your world where it can't be done, and I live in my fantasy world where things work. Can't you just allow me and people like me the fantasy that our systems are actually functional or that the words I am typing are being transmitted to you through my fantasy system. It's ok that it doesn't work, because in my fantasy, it works for me. FWIW, on linux, read-only libraries that have the 'right' alignment can be paged into memory directly w/o using buffers or a paging file. This is the preferred storage for commonly used libraries. A file may have multiple names, but ultimately, file information on disk is stored in groupings. Ignoring extended attributes, the information for one file is stored under "datapoint" or "node" on disk. For short, they are often called i-nodes or inodes. Inodes are the "handle" the operating system uses to reference information on a disk. Not only can file-containers (of their information) be referenced by multiple names on the file system (one can create a duplicate name using the "ln" command (without the "-s" flag) to create another "link" from a 2nd name to the same 'inode' referenced by a first name. This type of link, BTW, is called a hard-link -- (vs. links created with ln's "-s" flag" which are termed symbolic or soft links). After you've created a 2nd or third name to an inode, you can delete the first name and the information still remains on disk accessible through the 2nd name. As long as there is one reference to to the file's information 'node', the OS keeps the space reserved. Now if you delete all the file system's name to an inode, the file system can release the space. However, there is another way an inode on disk can remain reserved: by the OS itself. As long as the OS holds a link to a file on disk, the information (or data) in the file remains untouched. When files needing libraries are loaded into memory, the memory system keeps track of what is loaded into a demand-paged by associating each in-memory buffer with a file-system inode and offset. This means that under normal circumstances, if you delete a file on disk, and then create a new file with the same name, the memory subsystem still reserves the physical space of the old file on disk and is still able to page off new information out of that file into memory for programs that need it. Additionally, when a program "forks" a child, the child gets a copy of the parent's address space -- this includes block in memory that are mapped to a demand paged library on disk. So those programs will also also be able to demand-page in any library call they wish, from the original "deleted" file. Only when the process holding the memory mapped to the inode's information on disk exits or executes a new program does the inode get deleted -- but this means that long running daemon programs can hold open a deleted file for as long as they or a forked copy of them remains running. This essentially means that any program that had the deleted file open before it was "replaced", will still be running with the old library, which, effectively, only exists in memory (physically still resides on disk until the last file handle is released, but there's no easy way to get at it -- not impossible, if someone was determined, but not going to happen accidentally. So when you replace a library on disk, the operation, by established norm, uses the O_CREAT flag ask for a new-inode number to store the new file into. An update program could be written -- to controvert established update procedures and open an existing inode to write into -- but *normally*, under linux, this would result in a fatal EACCESS error because you are trying to open a file for write that is mapped or opened as a read-only file. The system won't let an updating program "accidentally" open a file that is 'read-only' locked (due to it being used as paging backing store) in write or update mode. So an update program has to create any update or replacement in a new inode -- preventing garbage being written into a file actively being used as page backing (note: this wasn't always true on all unix variants, but has been on linux for as long as I remember). Any new file doesn't overwrite or upset running programs. Any program dependent on that file (by the same name) will run with the new information if it is started after the file is updated. If the library file has the same name (implying same version), the interface *should* be binary compatible. If it's not, that would *likely* be a bug. If you upgrade the version of a library, it will have a new number. Any existing installed rpm will complain about your removal of the old version-number when you try to replace the library. Thus, if you follow the rpm's "advice" (which, unfortunately is more often than not, over-conservative in its requirements), you will have to install a "side-by-side" compatibility library that will have the same version number as the previous library. It's design intent is to provide binary compatibility for older applications that might want to use the old version of the library. Thus, in one rpm command, glibc can be updated, without updating any old programs as long as the new compatibility library package is installed as well. I'm sorry if I told you stuff you already knew, but I wanted to make sure you understood my 'fantasy' and why in my world, I can replace glibc with a new version and the new version's legacy compatibility file, and still have all the applications, new and old, play nicely together.

We've explained it to you several different ways. You're just not seeing it.

---- None of you have "explained" it nor laid out in what way the above, normally working update process has been broken by SuSE. If you have knowledge of deliberate acts or decisions by SuSE to defeat the normal linux memory and file mechanisms, it's possible to pervert the normal process and purposefully create failure. But that would be 'evil'...of course we aren't talking about google here, and SuSE has been 'touched' by Microsoft, so...I suppose anything is possible, but FWIW, I usually run my own, mostly vanilla kernel that I build myself, so I at least have some clue about how the OS should be functioning at the lowest level (obviously the specific code changes frequently, but certain features should remain compatible between versions or it wouldn't be behaving like Linux).

If you are really convinces that we're all wrong, and insist on doing this crazy thing...well, have fun on the job hunt.

If you are really convinced that linux doesn't work as I have described, then indeed, I'm woefully outdated in my technical knowledge and would appreciate if you could tell me why SuSE would deliberately work to override normal unix/linux mechanisms that protect against garbage being written into a file's paging store. Moreover, if you would explain how they do it and how it works on a standard "vanilla" linux kernel I'm I'm not the only person who would be very interested to know. Please tell, otherwise, please don't vehemently proclaim that you know something won't work -- especially when multiple people already claim to have done it. If this list is intended to be only for/by "IT" people, I guess I'm confused and lost. I would have thought people who adamantly insist on IT protocol being the only one-true-right way would be on the SUSE-enterprise list, not the "Open" SuSE' list. I'm sorry for being somewhat abrasive, as, from my perspective, you are telling me that if I try to sail around the world, I'll fall off the edge when I believe I have already sailed around the world. In your defense, you may be seeing me as someone who's claiming to be able to flap their arms and fly...but I assure you, I have to flap my wings just like everyone else! :-) -l -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Tue, 06 May 2008 19:47:49 -0700, Linda Walsh wrote:

if you upgrade glib with a new version and add a "compat" that deals ^^ it's glibc. glib is a totally different east. with previous versions (which SuSE does), then you don't have the problem.

You have problems, at least until ldconfig has run. And with glibc you *do* have a problem because for a long time SuSE named binary library according to its soname. But because glibc since version 6 uses symbol versions the library version seldom changes. A compat package can't help you in this case. Believe me: doing major version updates in a live system is bound to get you into trouble and you'll most probably spend lots of time cleaning up afterwards. Philipp -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Carlos E. R. wrote:

Because you can not do it one package at a time.

First, let me be clear -- I said package or package group. If you install an rpm and *satisfy* its dependencies, it *should* run if it's dependencies are properly specified.

Don't you understand? The moment you replace glibc - and you need to do it - - all the old programs will not run. System down.

Negatory.... Any programs already running in memory will keep using the glibc they already loaded and mapped into their address space. Second -- if it is the same glibc version, it should be backwards compatible. If the major version changes...well, yeah, that can be extra hairy. But its on *WINDOWS* that you can't replace the library out from underneath a program. This is why I'm frustrated -- people are so subconsciously programmed by the MS way, that they believe there is no other.

1: All the new packages are compiled against a new set of system libraries, therefore you need the new set of system libraries. 2: All the old packages are compiled against the old set of system libraries, therefore you need the old set of system libraries.

---- The above is very simplistic and not entirely true. Unless the libraries change major versions, they are likely compatible (more often than not). More recently, SuSE was smarter than you guys seem to think -- they've often shipped a "compat" library, so old glib-linked programs will run with the new glibc. Otherwise 3rd party vendors would have a fit. You can't just "invalidate" all previous applications without causing alot of headache and losing many customers -- why do you think most win98 applications still run on XP w/o change?

3: The new packages will not run with the old set of system libraries, and the old packages will not run with the new set of system

The old package will run on the new-libraries when they are properly configed and installed -- otherwise all the application vendors would throw up their hands and refuse to support SUSE. Does anyone bother to think? You can't just invalidate all previous 3rd party apps -- it kills the market.

4: You can't have both sets of libraries.

This is just plain untrue. If they are different versions of the libraries, then yes you can install both versions. Unix has version numbers on ".so" files since before linux was around.

You have two options:

1: Bit the bullet and fully upgrade the system, while running another system: be it the dvd, or an auxiliary partition.

Or...finish what I'm doing. Seems to be working just fine despite all the naysayers. did ... I only have ~90 packages left out of ~800 total, so I've fewer than 10% of the system still running at 9.3. It doesn't crash. It doesn't die...just keeps on ticking. ...the system is running fine. I've done it this way since 7.x ->8.x ->9.x ->10.x i386(10.2)->x86_64(10.3) (the most challenging of the bunch). But this stuff is not rocket science.

2: Upgrade only some programs, using packages you compiled yourself against the set of libraries in 9.3. You can not use packages from 10.3. If some work, that's the exception.

It's bug when it doesn't work. That's what rpm pre-reqs are for. Do you really think software vendors are going to bother with your platform if you force them to put a new version everytime you upgrade the OS? They (and users) don't want that type of headache. My current package distribution (on the machine I'm doing the manual upgrade on) looks like: 11 (none) 1 SuSE Linux 9.1 (i586) 88 SuSE Linux 9.3 (i586) 1 SuSE Linux 9.3 (i686) 9 openSUSE 10.2 (i586) 670 openSUSE 10.3 (i586) 2 openSUSE 10.3 (i686) (uptime 19 days at this point)... Whether you believe it or not, it works just fine. You make sure the needed libraries are present and off ya go. Contrary to your assertion of mixing packages working by 'chance', the 10.3 nfs server packages refused to work with 10.3 on test systems, so I reverted to the working 10.2. Seems to work fine. The ones that are mostly left are two "problem packages" that I'm not looking forward to touching (bind & samba) and a bunch of interdependent yast programs. The last problem I'm trying to solve in the yast installation is an error of the form: (old, 9.3)rpmnam < Version conflicts with (new, 10.3)rpmnam In that situation, you parse the old rpm-name and version, and look for an updated package in the 10.3 database. If not there, you check the conflicts database to see if the command was obsoleted, if so, add the obsoleting packages to the rpm command name. Then retry the rpm command and see if there are any new errors. I *could*, if sufficiently motivated, do the entire thing without successive runs of RPM, by only using the RPM databases on the system and the sum of them contained in each package in the new dist. But this isn't a *random* process... It's a process guided by the rpm interdependency database. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Wed, May 7, 2008 at 11:13 AM, Sam Clemens <clemens.sam1@gmail.com> wrote:

Sylvester Lykkehus wrote:

...

Out of curiosity from reading this thread.

I have a couple of machines running, which were originally a 10.0 install.

I have done 'live' upgrades of those systems, 10.0 -> 10.2 -> 10.3, and plan on going -> 11.0.

I haven't run into any of the problems described in this thread, or perhaps I missed a point ?

All of those versions use glibc6, so there's no mid-upgrade replacement of glibc5.

...

I've used the smart package manager for all the upgrades, and my procedure

have been as follows:

...

Remove all repos Add repo's for new version Do a smart upgrade --update Remove all packages on the system, which is not in the new repo's (get rid

of those packages that was available for e.g. 10.2, but not 10.3)

...

Both machines function fine, and are not a mess. So, have I been lucky, or

would this be a safe procedure ?

...

You didn't have to replace glibc, so it's rather trivial.

But I did a 9.3 to 10.1 in-place upgrade (which I thought included the glibc upgrade) using Yast Upgrade (essentially the same process Sylvester mentioned using on-line repositories) and it worked fine. There was one reboot, and it worked fine. -- ----------JSA--------- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org