Guru's RPM: YaST vs. Web
Hi, Hopefully Pascal will be able to answer this... In the past, I'd used the Guru's RPM repository by downloading individual RPMs and installing them with the "rpm" command. E.g.: % wget http://ftp4.gwdg.de/pub/linux/misc/suser-guru/rpm/packages//Office/tellico/t... % rpm -U tellico-1.0.2-1.guru.suse100.kde34.i686.rpm The above URL was obtained from http://linux01.gwdg.de/~pbleser/rpm-navigation.php?cat=/Office/tellico, which appeared in the Guru's RSS feed. When I learned that there was a YaST repository for the Guru's RPMs, I figured that was the way to go. However, for some reason the YaST repository does not seem to have the same versions as the section of the gwdg.de Web site referenced by the URL above. Instead, after adding http://ftp.gwdg.de/pub/linux/misc/suser-guru/rpm/10.0 as a YaST installation source I'm offered Tellico 0.13.8-3. Is this just a glitch, or will the YaST repository remain behind the other section of the Guru's repository? Randall Schulz
Hi, On Sun, 9 Oct 2005, Randall R Schulz wrote:
Hopefully Pascal will be able to answer this...
In the past, I'd used the Guru's RPM repository by downloading individual RPMs and installing them with the "rpm" command. E.g.:
% wget http://ftp4.gwdg.de/pub/linux/misc/suser-guru/rpm/packages//Office/tellico/t...
% rpm -U tellico-1.0.2-1.guru.suse100.kde34.i686.rpm
The above URL was obtained from http://linux01.gwdg.de/~pbleser/rpm-navigation.php?cat=/Office/tellico, which appeared in the Guru's RSS feed.
When I learned that there was a YaST repository for the Guru's RPMs, I figured that was the way to go. However, for some reason the YaST repository does not seem to have the same versions as the section of the gwdg.de Web site referenced by the URL above. Instead, after adding http://ftp.gwdg.de/pub/linux/misc/suser-guru/rpm/10.0 as a YaST installation source I'm offered Tellico 0.13.8-3.
Is this just a glitch, or will the YaST repository remain behind the other section of the Guru's repository?
ftp.gwdg.de has by far not enough I/O bandwidth to serve all tasks just-in-time. This brave knight is delivering > 4 TB a day for the third day in sequence today, with more than 3500 session every moment. Already since the server bottleneck with SUSE-9.3, the main external rsyncs and the APT repository generations happen on ftp4, and ftp.gwdg.de afterwards is fetching the results with rsync. This may lead to temporary inconsistencies, but until now, those were "very" temporarily: the refresh intervalls (both: repository generation and repository rsync) were both 4 hours, and usually the new ftp4 repository was in-place at ftp 80 minutes later with about 10 minutes "inconsistency possibility". But this time the load is so extreme on ftp.gwdg.de that even a simple rsync of an APT repository (only about 10 files and 10 000 symlinks) needs up to a whole day, and, even more worse, ftp4 this time is in the same situation that ftp3 had seen during 9.3: too much disk I/O, so a repository generation at ftp4 which needed 30 minutes some weeks ago will last 10 to 40 hours today. A next fucking circumstance is that rsync-2.6.6 (introduced at ftp4 with the update to 10.0 on tuesday) seems to be flakey with timeout errors, I guess there is a real rsync bug which triggers under high load only. There is no way out, only a chance to get thru. So, change ftp to ftp4 in your YaST installation source configuration if you run into inconsistencies, but be aware that you extend my pain each time you use it. ;-)) Cheers -e -- Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org)
Eberhard, On Sunday 09 October 2005 11:22, Eberhard Moenkeberg wrote:
...
Is this just a glitch, or will the YaST repository remain behind the other section of the Guru's repository?
... [Explanation of the woes of running a popular server] ...
Thanks for the information.
So, change ftp to ftp4 in your YaST installation source configuration if you run into inconsistencies, but be aware that you extend my pain each time you use it. ;-))
Death by a thousand cuts, eh? Sorry. I will exercise patience. Is there a mirror for the YaST-compatible Guru's RPM Repository? Surely some of the popular mirrors would make enough space available, wouldn't they?
Cheers -e
Randal Schulz
Hi, On Sun, 9 Oct 2005, Randall R Schulz wrote:
On Sunday 09 October 2005 11:22, Eberhard Moenkeberg wrote:
...
Is this just a glitch, or will the YaST repository remain behind the other section of the Guru's repository?
... [Explanation of the woes of running a popular server] ...
Thanks for the information.
So, change ftp to ftp4 in your YaST installation source configuration if you run into inconsistencies, but be aware that you extend my pain each time you use it. ;-))
Death by a thousand cuts, eh?
Sorry. I will exercise patience.
Is there a mirror for the YaST-compatible Guru's RPM Repository? Surely some of the popular mirrors would make enough space available, wouldn't they?
Come on, just change ftp to ftp4 and go on. 1000 OpenSUSE users more or less a day really is peanuts. ;-)) Even more: if you all would start your YOU at the same time, it can be very efficient because the second YOU'er already hits a buffer cache match (and so on). So let's try: please as many as possible of the readers here (but nowhere else ;-)) change their YOU and/or YaST installation source to ftp4, and let the 0 minute of each full hour be the moment to trigger installation or update. I will report here if I see an effect. Cheers -e -- Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org)
Eberhard, On Sunday 09 October 2005 11:51, Eberhard Moenkeberg wrote:
...
Is there a mirror for the YaST-compatible Guru's RPM Repository? Surely some of the popular mirrors would make enough space available, wouldn't they?
Come on, just change ftp to ftp4 and go on. 1000 OpenSUSE users more or less a day really is peanuts. ;-))
Yowsa! That made a _huge_ difference in the set of packages I see in YaST Install & Remove. I was preparing a posting about how I couldn't find a bunch of the dependencies for running iPodder (a Python-based podcast aggregator), but it's all moot, now. Thanks for pointing this out and sorry to make you say it twice. I really do appreciate all the work of all you folks in making this new release available.
Even more: if you all would start your YOU at the same time, it can be very efficient because the second YOU'er already hits a buffer cache match (and so on).
So let's try: please as many as possible of the readers here (but nowhere else ;-)) change their YOU and/or YaST installation source to ftp4, and let the 0 minute of each full hour be the moment to trigger installation or update.
Ummm... I'm not sure if you're kidding or not. You don't _really_ want us all to synchronize our access to your server, do you?
I will report here if I see an effect.
Cheers -e
Randall Schulz
Hi, On Sun, 9 Oct 2005, Randall R Schulz wrote:
On Sunday 09 October 2005 11:51, Eberhard Moenkeberg wrote:
Even more: if you all would start your YOU at the same time, it can be very efficient because the second YOU'er already hits a buffer cache match (and so on).
So let's try: please as many as possible of the readers here (but nowhere else ;-)) change their YOU and/or YaST installation source to ftp4, and let the 0 minute of each full hour be the moment to trigger installation or update.
Ummm... I'm not sure if you're kidding or not. You don't _really_ want us all to synchronize our access to your server, do you?
I am not kidding. The synchronizing effort at http://ftp.gwdg.de for the i386 ISOs (implemented by Christoph Thiel at download.opensuse.org) is a great push. OK, all non-SUSE-using customers of ftp.gwdg.de are paying tax for this, but SUSE had all the last (10? 9? Maybe only 8) years more then 2/3 of the ftp.gwdg.de output, and now for a hopefully limited time it is more then 9/10. I consider this a service effort. If we would achieve this synchronous action (remember: "SUSE-10.0 YOU time at ftp4 is minute 0 of every hour"), I see a good chance for a major improvement in service without more stress to the server. If it happens, I simply have to watch that the outgoing sum of ftp + ftp4 does not exceed our bandwidth reserve. Not an easy task. ;-)) Cheers -e -- Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org)
On Sun, Oct 09, 2005 at 11:39:13PM +0200, Eberhard Moenkeberg wrote:
I am not kidding. The synchronizing effort at http://ftp.gwdg.de for
Nice website. Very up to date with the OS and all. Oh well, if it ain't broke, don't fix it. Just to be sure. is it http://ftp you want or ftp://ftp you want us to use?
the i386 ISOs (implemented by Christoph Thiel at download.opensuse.org) is a great push. OK, all non-SUSE-using customers of ftp.gwdg.de are paying tax for this, but SUSE had all the last (10? 9? Maybe only 8) years more then 2/3 of the ftp.gwdg.de output, and now for a hopefully limited time it is more then 9/10. I consider this a service effort.
If we would achieve this synchronous action (remember: "SUSE-10.0 YOU time at ftp4 is minute 0 of every hour"), I see a good chance for a major improvement in service without more stress to the server.
OK, so I do put the following in my crontab: 0 * * * * /usr/bin/online_update -u http://ftp.gwdg.de/pub/suse or am I misreading things? houghi -- Quote correct (NL) http://www.briachons.org/art/quote/ Zitiere richtig (DE) http://www.afaik.de/usenet/faq/zitieren Quote correctly (EN) http://www.netmeister.org/news/learn2quote.html
Hi, On Mon, 10 Oct 2005, houghi wrote:
On Sun, Oct 09, 2005 at 11:39:13PM +0200, Eberhard Moenkeberg wrote:
I am not kidding. The synchronizing effort at http://ftp.gwdg.de for
Nice website. Very up to date with the OS and all. Oh well, if it ain't broke, don't fix it.
Just to be sure. is it http://ftp you want or ftp://ftp you want us to use?
http://ftp4... ftp://ftp4... rsync://ftp4... starting around minute 0 of an hour.
the i386 ISOs (implemented by Christoph Thiel at download.opensuse.org) is a great push. OK, all non-SUSE-using customers of ftp.gwdg.de are paying tax for this, but SUSE had all the last (10? 9? Maybe only 8) years more then 2/3 of the ftp.gwdg.de output, and now for a hopefully limited time it is more then 9/10. I consider this a service effort.
If we would achieve this synchronous action (remember: "SUSE-10.0 YOU time at ftp4 is minute 0 of every hour"), I see a good chance for a major improvement in service without more stress to the server.
OK, so I do put the following in my crontab: 0 * * * * /usr/bin/online_update -u http://ftp.gwdg.de/pub/suse
or am I misreading things?
No no, please not at EVERY hour... The thought was: if you do, do it at minute 0. Once a day should be enough. Cheers -e -- Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org)
On Mon, Oct 10, 2005 at 04:23:38PM +0200, Eberhard Moenkeberg wrote:
No no, please not at EVERY hour...
The thought was: if you do, do it at minute 0. Once a day should be enough.
Ok, Thought you might wanted to have it done every hour for testing purposes. :-) houghi -- Quote correct (NL) http://www.briachons.org/art/quote/ Zitiere richtig (DE) http://www.afaik.de/usenet/faq/zitieren Quote correctly (EN) http://www.netmeister.org/news/learn2quote.html
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Randall R Schulz wrote:
Eberhard, On Sunday 09 October 2005 11:22, Eberhard Moenkeberg wrote:
...
Is this just a glitch, or will the YaST repository remain behind the other section of the Guru's repository? ... [Explanation of the woes of running a popular server] ... Thanks for the information.
And I don't have much to add to what Eberhard already wrote ;) Furthermore, to reduce the burden on the gwdg.de side, I generate the YaST2 and Red Carpet repository metadata on my server, so it is "just" uploaded and not computed on ftp4.gwdg.de That means, in a purely theoretical world, that there shouldn't be any inconsistency between the YaST2 repository metadata and the availability of the packages (using YaST2's software manager or y2pmsh). But if you meant an inconsistency between the newsfeed / list of latest packages on the website and the availability of packages in the YaST2 repositories, you are correct. The rsync issues Eberhard wrote about are affecting my site quite dramatically, as the last 3 or 4 days my newest packages haven't been uploaded. This afternoon an rsync job was able to complete successfully (by luck, it seems), so my queue is more or less empty now. But we don't know whether it'll keep on working or not (probably not). The list of latest packages is just a file where I add the entries "manually". Hence, if I announce a package there but the rsync doesn't fetch the package properly, it is inconsistent. Sorry for the inconvenience, but as Eberhard explained, it's hell for the ftp*.gwdg.de servers at the moment.
So, change ftp to ftp4 in your YaST installation source configuration if you run into inconsistencies, but be aware that you extend my pain each time you use it. ;-)) Death by a thousand cuts, eh? Sorry. I will exercise patience. Is there a mirror for the YaST-compatible Guru's RPM Repository? Surely some of the popular mirrors would make enough space available, wouldn't they?
Well, unfortunately not. If someone would like to stand up and mirror my packages, I'd be the first
one to be happy about it, but I didn't have any offers for that (yet?).
If you'd like to support me in what I'm doing, and especially the many users who download my
packages, *and* you have bandwidth and at least 3 GB of space to host it, let me know :)
Normally a few servers are also mirroring the suser-* directories of ftp.gwdg.de, so there should be
a few out there, but I don't have a list.
If you run into a mirror that does, please send me an e-mail (guru AT unixtech DOT be) to let me
know, I'll publish that information on my website.
Again, I'm really sorry for the inconvenience, but I'm afraid there's not much to do about it at the
moment. Seems like 10.0 draws a lot of attention and... downloads ;)
cheers
- --
-o) Pascal Bleser http://linux01.gwdg.de/~pbleser/
/\\
Hi, On Sun, 9 Oct 2005, Pascal Bleser wrote:
Randall R Schulz wrote:
Eberhard, On Sunday 09 October 2005 11:22, Eberhard Moenkeberg wrote:
...
Is this just a glitch, or will the YaST repository remain behind the other section of the Guru's repository? ... [Explanation of the woes of running a popular server] ... Thanks for the information.
And I don't have much to add to what Eberhard already wrote ;)
Furthermore, to reduce the burden on the gwdg.de side, I generate the YaST2 and Red Carpet repository metadata on my server, so it is "just" uploaded and not computed on ftp4.gwdg.de
That means, in a purely theoretical world, that there shouldn't be any inconsistency between the YaST2 repository metadata and the availability of the packages (using YaST2's software manager or y2pmsh).
But if you meant an inconsistency between the newsfeed / list of latest packages on the website and the availability of packages in the YaST2 repositories, you are correct. The rsync issues Eberhard wrote about are affecting my site quite dramatically, as the last 3 or 4 days my newest packages haven't been uploaded. This afternoon an rsync job was able to complete successfully (by luck, it seems), so my queue is more or less empty now. But we don't know whether it'll keep on working or not (probably not).
The list of latest packages is just a file where I add the entries "manually". Hence, if I announce a package there but the rsync doesn't fetch the package properly, it is inconsistent.
Sorry for the inconvenience, but as Eberhard explained, it's hell for the ftp*.gwdg.de servers at the moment.
So, change ftp to ftp4 in your YaST installation source configuration if you run into inconsistencies, but be aware that you extend my pain each time you use it. ;-))
Death by a thousand cuts, eh? Sorry. I will exercise patience. Is there a mirror for the YaST-compatible Guru's RPM Repository? Surely some of the popular mirrors would make enough space available, wouldn't they?
Well, unfortunately not. If someone would like to stand up and mirror my packages, I'd be the first one to be happy about it, but I didn't have any offers for that (yet?). If you'd like to support me in what I'm doing, and especially the many users who download my packages, *and* you have bandwidth and at least 3 GB of space to host it, let me know :)
Normally a few servers are also mirroring the suser-* directories of ftp.gwdg.de, so there should be a few out there, but I don't have a list.
If you run into a mirror that does, please send me an e-mail (guru AT unixtech DOT be) to let me know, I'll publish that information on my website.
Again, I'm really sorry for the inconvenience, but I'm afraid there's not much to do about it at the moment. Seems like 10.0 draws a lot of attention and... downloads ;)
Yes, this is the best "net" effort for SUSE ever. No doubt. So, my one "whining eye" is accompanied by the other - laughing as never before. This lousy bottleneck is making success, success like never before and never thought of. ;-)) And seeing the bottlenecks (at the servers, and at the distribution scheme, too), some good ideas for a better future are coming up. W. I. Lenin once said: "the practice is the proof of the truth" (in german: "die Praxis ist das Kriterium der Wahrheit"), and this will be my motor: First, I am struggling for a new server with 32 GB RAM, to have enough buffer cache even if the real disk I/O is suffering (as it will always be in these /. situations). No real chance from the GWDG budget, but maybe one of my external ftp.gwdg.de protegees will find a sponsor. I would love to use a quad Opteron like Newisys 4300-E (the original "reference design" for quad AMD Opterons and still on top, even with the new DualCore Opterons). But I guess Sanmina/Newisys does not care about ftp.gwdg.de and maybe even does not even have any picture about, so I need some ambassador for this idea more directly towards AMD. AMD has the A for "advanced", not for "America", and Dresden ("saxony valley" in Germany) is the Opteron production center, and the german government has declared to help AMD to convince Intel's bad practices against AMD in Germany (read: Europe). You see. I am dreaming especially about this hope... HP (the Opteron server market leader) already has sponsored kernel.org with two Proliant 585, so they have done their duty, I guess they guess. But second: I will try to get a better "real" disk I/O soon, by using two IDE-Raid arrays instaed of a single one, concatenated with LVM in striping mode. This can theoretically double the real disk I/O rate, but in practice noone would pay for a second IDE raid array currently, so I have to test with two "older generation" ones; but hopefully I will see a positive result which is good enough to dream upon. And third we can try the trick which had already success on ftp.gwdg.de with the i386 ISOs: to help that the most important files for SUSE-10.0-OSS will remain in buffer cache once they got there, by provocing cache hits again and again, tight enough in time. This is the idea behind "minute 0 of every hour for OpenSUSE" - please participate to make it work. ;-)) Cheers -e -- Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org)
On Sun, Oct 09, 2005 at 09:08:42PM +0200, Pascal Bleser wrote:
Normally a few servers are also mirroring the suser-* directories of ftp.gwdg.de, so there should be a few out there, but I don't have a list.
If you run into a mirror that does, please send me an e-mail (guru AT unixtech DOT be) to let me know, I'll publish that information on my website.
As you have the mirrors for both SUSE and openSUSE, it should not be to hard to write a script to test if and what suser-* directories are available. Somebody already has done it (http://www.acert.pt/~fribeiro/mirrorwatch/?distro=suse&tree=stable) but that one is hopelesly out of date. I have already written the person twice and have not recieved an answer Mmm. Will look into that, although it might not be as nice as the page above, it could become the new content of my webpage. I will only be looking at the official mirrors available from either the Novell or the openSUSE website. Don't hold your breath though. Not sure if I am going to do it. :-) houghi -- Quote correct (NL) http://www.briachons.org/art/quote/ Zitiere richtig (DE) http://www.afaik.de/usenet/faq/zitieren Quote correctly (EN) http://www.netmeister.org/news/learn2quote.html
Eberhard Moenkeberg schrieb:
Hi,
On Sun, 9 Oct 2005, Randall R Schulz wrote:
ftp.gwdg.de has by far not enough I/O bandwidth to serve all tasks just-in-time. This brave knight is delivering > 4 TB a day for the third day in sequence today, with more than 3500 session every moment.
Eberhard, i think we all apreciate your work and bandwidth you offer and simply want to say thank you, for what you are doing! I don't know, what we (i) would do without you!
Cheers -e
Thanks, -rauch
Hi, On Sun, 9 Oct 2005, Rauch Christian wrote:
Eberhard Moenkeberg schrieb:
On Sun, 9 Oct 2005, Randall R Schulz wrote:
ftp.gwdg.de has by far not enough I/O bandwidth to serve all tasks just-in-time. This brave knight is delivering > 4 TB a day for the third day in sequence today, with more than 3500 session every moment.
Eberhard, i think we all apreciate your work and bandwidth you offer and simply want to say thank you, for what you are doing!
I don't know, what we (i) would do without you!
I will taste this honey only if I see that you all get aware of your special "every hour minute 0" window at ftp4. ;-)) Cheers -e -- Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org)
On Sunday 09 October 2005 19:22, Eberhard Moenkeberg wrote:
This brave knight is delivering > 4 TB a day for the third day in sequence today, with more than 3500 session every moment.
These are very impressive figures. You are obviously having difficulties keeping up with demand, but the fact that the demand is there shows what a great service you are providing - we all very much appreciate it. -- Pob hwyl / Best wishes Kevin Donnelly www.kyfieithu.co.uk - Meddalwedd Rhydd yn Gymraeg www.cymrux.org.uk - Linux Cymraeg ar un CD
Hi, On Sun, 9 Oct 2005, Kevin Donnelly wrote:
On Sunday 09 October 2005 19:22, Eberhard Moenkeberg wrote:
This brave knight is delivering > 4 TB a day for the third day in sequence today, with more than 3500 session every moment.
These are very impressive figures. You are obviously having difficulties keeping up with demand, but the fact that the demand is there shows what a great service you are providing - we all very much appreciate it.
Yes. You understand, I need no drugs these days. ;-)) Cheers -e -- Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org)
participants (6)
-
Eberhard Moenkeberg -
houghi -
Kevin Donnelly -
Pascal Bleser -
Randall R Schulz -
Rauch Christian