[opensuse-factory] update-mirrors out of sync - please remove these hazardous sites from the list/redirectors
hi there, i just recently discovered that some older 10.2 box is completely out of sync as the updates/patches repository it got subscribed to during install time back then is only having patches from year 2007. this is total madness. somebody gotta keep this list clean and only distributing reliable and current mirrors. example is: <http://ftp.gwdg.de/pub/suse/update/10.2/repodata/> (current) <http://ftp.tu-ilmenau.de/Mirrors/ftp.suse.com/update/10.2/repodata/> (way outdated) i even re-ran the yast online-update configuration and the novell initscript or whatever it was called (/usr/bin/suse_register ?) re-subscribed me to this lagging mirror several times over again :( argh. :( p.s. i somewhere read about zypper updates also coming to older suse systems. is there a way to profit from newer zypper updates on these older suse/opensuse systems that maybe newer zypper builds feature a better and more intelligent handling of repositories and being in-/outof-sync with the main suse sources. there has to be some sane way of getting rid of unreliable downloadlocations. thanks. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Thursday 19 June 2008 06:57:12 pm ab wrote:
i just recently discovered that some older 10.2 box is completely out of sync as the updates/patches repository it got subscribed to during install time back then is only having patches from year 2007.
this is total madness. somebody gotta keep this list clean and only distributing reliable and current mirrors.
It is redirector problem if you use http://download.opensuse.org that we can't do much about, just complain as you did, or use some dependable mirror directly, as I do.
example is: <http://ftp.gwdg.de/pub/suse/update/10.2/repodata/> (current)
I use that one directly, although I'm on the other side of the ocean. Though, you may consider new, faster Goettingen server: ftp://ftp5.gwdg.de/pub/suse/update/10.2/repodata/ For me Goettingen is not the fastest mirror, but it is the most dependable.
<http://ftp.tu-ilmenau.de/Mirrors/ftp.suse.com/update/10.2/repodata/> (way outdated)
I'm tired of redirector and I don't want to deal with that for now. It needs improvements, like taking feedback from clients, but for that zypper and YaST Software Management have to be changed too, to give such feedback. Download errors, slow connection, missing packages, etc, than redirector will not slam user with the same broken mirror time and again. -- Regards, Rajko http://en.opensuse.org/Portal needs helpful hands. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Rajko M. wrote:
It is redirector problem if you use http://download.opensuse.org that we can't do much about, just complain as you did, or use some dependable mirror directly, as I do.
what do you mean? i wasnt using a redirectory back then. i doubt it was working the way it does today. back when i installed that 10.2 system and ran the online update configuration for the first time, it assigns you exactly one update repository. at least it was that way back then. and it was exactly that one outdated mirror server. i had some problems with the box and i was investigating and i found out that that update mirror was many months out of updates. then i re-ran the onlineupdate registration module inside yast2. and it again supplied me with that crap mirror several times, til i finally arrived at those gwdg servers. all i was saying that there is a need of keeping the lists updated what mirrors are reliable and getting rid of unreliable urls/sites from the distribution scripts and so on. again: i think i read about these problems of out-of-date-mirrors and zypper stuff somewhere. i hope this gets fixed and handled more intelligently and dependably in the near future and that older suse versions can still profit from these zypper enhancements. cheers. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Hi Rajko, On Thu, Jun 19, 2008 at 07:43:17PM -0500, Rajko M. wrote:
On Thursday 19 June 2008 06:57:12 pm ab wrote:
i just recently discovered that some older 10.2 box is completely out of sync as the updates/patches repository it got subscribed to during install time back then is only having patches from year 2007.
this is total madness. somebody gotta keep this list clean and only distributing reliable and current mirrors.
It is redirector problem if you use http://download.opensuse.org that we can't do much about, just complain as you did, or use some dependable mirror directly, as I do.
10.2 did not use download.opensuse.org infrastructure; it used hardcoded mirrors. (10.2 was the last release to do that.) No redirector involved.
I'm tired of redirector and I don't want to deal with that for now.
I would be *very* interested in hearing the cause which makes you tired of it. What are your problems with it? (If you still think you have problems with it, that is.)
It needs improvements, like taking feedback from clients, but for that zypper and YaST Software Management have to be changed too, to give such feedback. Download errors, slow connection, missing packages, etc, than redirector will not slam user with the same broken mirror time and again.
Exactly, there are things that cannot be solved on the server side. Some things can only be done by clients. If you are interested, you might want to read http://en.opensuse.org/Libzypp/Failover and let us know what you think about the proposal. Your feedback would be valuable. Thanks, Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development
On Thursday 19 June 2008 08:27:05 pm Peter Poeml wrote:
Hi Rajko,
On Thu, Jun 19, 2008 at 07:43:17PM -0500, Rajko M. wrote:
On Thursday 19 June 2008 06:57:12 pm ab wrote:
i just recently discovered that some older 10.2 box is completely out of sync as the updates/patches repository it got subscribed to during install time back then is only having patches from year 2007.
this is total madness. somebody gotta keep this list clean and only distributing reliable and current mirrors.
It is redirector problem if you use http://download.opensuse.org that we can't do much about, just complain as you did, or use some dependable mirror directly, as I do.
10.2 did not use download.opensuse.org infrastructure; it used hardcoded mirrors. (10.2 was the last release to do that.) No redirector involved.
It was so long ago that I have forgotten exact method :-)
I'm tired of redirector and I don't want to deal with that for now.
I would be *very* interested in hearing the cause which makes you tired of it. What are your problems with it?
(If you still think you have problems with it, that is.)
I'll see with 11.0 how it works, but basic problem was that some server did not catch up with Factory and often there was missing file breaking update. If I repeated zypper command I would get same mirror, and the update will fail on the same spot, but that is mentioned in article http://en.opensuse.org/Libzypp/Failover and it is designed behavior in 10.3 , but it can annoying at the times. ...
If you are interested, you might want to read http://en.opensuse.org/Libzypp/Failover and let us know what you think about the proposal. Your feedback would be valuable.
OK. The only thing that I can see right now is mentioned in: Example scenarios Server breakage: (redirector) New installation Client that can use failover mirrors can have list of mirrors already included out of the box, so if redirector is offline in the moment of installation, that list can be used. It can be obsolete, but if client has ability to switch to another mirror it will find good one in the list.
Thanks, Peter
-- Regards, Rajko http://en.opensuse.org/Portal needs helpful hands. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Hi Rajko, On Thu, Jun 19, 2008 at 09:31:49PM -0500, Rajko M. wrote:
I'm tired of redirector and I don't want to deal with that for now.
I would be *very* interested in hearing the cause which makes you tired of it. What are your problems with it?
(If you still think you have problems with it, that is.)
I'll see with 11.0 how it works, but basic problem was that some server did not catch up with Factory and often there was missing file breaking update.
Cave: you are mixing two things it seems -- "11.0" and "catch up with Factory" are two things. While catching up with Factory you can see a lot of problems, ranging from innovative breakage of the metadata, to inconsistencies of the metadata during its update/replacement. Both of which doesn't have much to do with the method of the distribution (download redirector). The main thing to achieve is to get a consistent metadata set of *Factory* onto the download server (download.opensuse.org). The rest is taken care of by the redirector. If the redirector wasn't able to do this, there'd be a lot of problems with the update trees and the buildservice repositories trees. (Both also very short-lived.)
If I repeated zypper command I would get same mirror, and the update will fail on the same spot, but that is mentioned in article http://en.opensuse.org/Libzypp/Failover and it is designed behavior in 10.3 , but it can annoying at the times.
Yes.
...
If you are interested, you might want to read http://en.opensuse.org/Libzypp/Failover and let us know what you think about the proposal. Your feedback would be valuable.
OK. The only thing that I can see right now is mentioned in: Example scenarios Server breakage: (redirector) New installation
Client that can use failover mirrors can have list of mirrors already included out of the box, so if redirector is offline in the moment of installation, that list can be used. It can be obsolete, but if client has ability to switch to another mirror it will find good one in the list.
Yes, that would be one way to deal with the "Internet gone fishing" or "Redirector gone fishing" situation during installation. Feel free to add it to the wiki page. Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development
Hi Rajko, I'm posting this followup to the zypp-devel list, since it better fits there than on opensuse-factory. On Fri, Jun 20, 2008 at 12:34:50PM +0200, Peter Poeml wrote:
Client that can use failover mirrors can have list of mirrors already included out of the box, so if redirector is offline in the moment of installation, that list can be used. It can be obsolete, but if client has ability to switch to another mirror it will find good one in the list.
Yes, that would be one way to deal with the "Internet gone fishing" or "Redirector gone fishing" situation during installation.
Feel free to add it to the wiki page.
Thanks for adding this bit to the wiki page. [1] I added the following note about security. ''Moreover, for security reasons it might be more reasonable to preconfigure clients only with one or two "trusted" mirrors, and not with the full list. It might be preferrable to delay updates on the client until the origin server is reachable again, rather than exposing the client to rogue mirrors.'' It is noteworthy that the origin server (download.opensuse.org) does not redirect for metadata and signatures to any mirror, so the clients always get those critical files from the origin site. If clients are using mirrors directly, the tradeoff is that they are less secure. Peter [1] http://en.opensuse.org/Libzypp/Failover -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development
Hi, On Thu, 19 Jun 2008, Rajko M. wrote:
On Thursday 19 June 2008 06:57:12 pm ab wrote:
i just recently discovered that some older 10.2 box is completely out of sync as the updates/patches repository it got subscribed to during install time back then is only having patches from year 2007.
this is total madness. somebody gotta keep this list clean and only distributing reliable and current mirrors.
It is redirector problem if you use http://download.opensuse.org that we can't do much about, just complain as you did, or use some dependable mirror directly, as I do.
example is: <http://ftp.gwdg.de/pub/suse/update/10.2/repodata/> (current)
I use that one directly, although I'm on the other side of the ocean. Though, you may consider new, faster Goettingen server: ftp://ftp5.gwdg.de/pub/suse/update/10.2/repodata/
From release of OpenSUSE 11.0 yesterday, I see that ftp connections to ftp5.gwdg.de have a severe waiting time before the "data connection" becomes running. No timeout, just wait - was about 1 hour 15 hours before, is still several minutes currently. This is due to the way new sockets are handled by the kernel. So using http: instead of ftp: is a good workaround because it does not need a second port connection. But, using ftp5.gwdg.de "by hand", be aware: since 9:30 h MEST yesterday the server delivers the maximum (1 GBit/sec), so you will probably get less throughput against ftp3.gwdg.de. But the DVD ISOs are only on ftp5.
For me Goettingen is not the fastest mirror, but it is the most dependable.
Surely. I like to see me as a professional, and I could not without this. But some day in beginning of july ftp5 will have to do a reboot, to switch to a bigger filesystem (8 TB XFS - is 5 TB ext3fs currently). At the same time ftp5 will switch to SLES10 SP2 from SP1, but this will not need extra time. Only some fresh mirrorings < 2 h will become invalid (the new filesystem gets rsynced continuously, but each internal run needs about 2 hours to fulfill).
<http://ftp.tu-ilmenau.de/Mirrors/ftp.suse.com/update/10.2/repodata/> (way outdated)
I'm tired of redirector and I don't want to deal with that for now. It needs improvements, like taking feedback from clients, but for that zypper and YaST Software Management have to be changed too, to give such feedback. Download errors, slow connection, missing packages, etc, than redirector will not slam user with the same broken mirror time and again.
Please don't talk bad about the OpenSUSE redirector without reflection. It is one of the best and most needed ideas since long, a solution for a lot of problems at the mirror sides, and - in my guess - already very reliable. And it does the actual redirection on a previous per-file verification. At home, I use the http://download.opensuse.org/ URLs, and I see no need to use "my" ftp5.gwdg.de instead. Viele Grüße Eberhard Mönkeberg (emoenke@gwdg.de, em@kki.org) -- Eberhard Mönkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoenke@gwdg.de Tel.: +49 (0)551 201-1551 ------------------------------------------------------------------------- Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG) Am Fassberg 11, 37077 Göttingen URL: http://www.gwdg.de E-Mail: gwdg@gwdg.de Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150 Geschäftsführer: Prof. Dr. Bernhard Neumair Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger Sitz der Gesellschaft: Göttingen Registergericht: Göttingen Handelsregister-Nr. B 598 -------------------------------------------------------------------------
On Thursday 19 June 2008 08:39:17 pm Eberhard Moenkeberg wrote:
I use that one directly, although I'm on the other side of the ocean. Though, you may consider new, faster Goettingen server: ftp://ftp5.gwdg.de/pub/suse/update/10.2/repodata/
From release of OpenSUSE 11.0 yesterday, I see that ftp connections to ftp5.gwdg.de have a severe waiting time before the "data connection"
This didn't catch me this time. I updated to final before release rush.
For me Goettingen is not the fastest mirror, but it is the most dependable.
Surely. I like to see me as a professional, and I could not without this.
And you are professional. For years when I had problem with local mirrors resorting to GWDG was solution.
But some day in beginning of july ftp5 will have to do a reboot, ...
I guess that you rebooted server many times in last 10 years, and somehow I was able to access repositories any time I wanted. ...
I'm tired of redirector and I don't want to deal with that for now. ... Please don't talk bad about the OpenSUSE redirector without reflection.
Reflection is "for now". I have download.opensuse.org again in openSUSE 11.0 installation. In 10.3 it is still gwdg, I was simply tired of switching every time some mirror had a glitch.
Viele Grüße Eberhard Mönkeberg (emoenke@gwdg.de, em@kki.org)
-- Viele Grüße Rajko --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Friday 20 June 2008 04:39:17 Eberhard Moenkeberg wrote:
this is total madness. somebody gotta keep this list clean and only distributing reliable and current mirrors.
Right now I cannot even add the main repository because it has non-matching checksums. There must be something wrong with it because even gwdg behaves the same. On x86_64. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Hi, I am responding to the subject: "what about bittorent for everything?" (rather old mail -- I didn't realize the question until now though, the original subject of the thread was different)) The answer is: No. Bittorrent is not suitable for everything, because 1) P2P is only suitable for large files; but many files are tiny. Its overhead and complexity makes it unattractive for small files. P2P's success is largely based on its ability to escape legal control. 3) P2P is not optimal for networks, when used at large scale. See http://en.wikipedia.org/wiki/Segmented_downloading for general information, or more specifically, read up on the "Application-Layer Traffic Optimization (ALTO)" problem at http://alto.tilab.com/. Executive summary: P2P technology lacks knowledge about network topology. Metalinks however can be used for everything. They incur very small overhead, and they embrace P2P, in addition to HTTP/FTP, and have additional advantages over those technologies, for example the ability to embed PGP signatures, which is unique. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Monday 2008-07-21 at 17:09 +0200, Peter Poeml wrote:
Hi,
I am responding to the subject: "what about bittorent for everything?"
(rather old mail -- I didn't realize the question until now though, the original subject of the thread was different))
The answer is: No.
Bittorrent is not suitable for everything, because
1) P2P is only suitable for large files; but many files are tiny. Its overhead and complexity makes it unattractive for small files. P2P's success is largely based on its ability to escape legal control.
I read a study of some European university that contradicted this. They experimentally used bittorrent in the intranet to deploy software updates to the entire intranet (thousands of computer updating (windows?) at the same time, and they discovered that it was way faster than having some dedicated update servers in the same intranet. They found a legal use for it that was better that the normally used solution. For the purpose they used a version of bittorrent client that could not look up in Internet, only intranet. I don't have the reference handy, but some one may recollect this study. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) iD8DBQFIhN2wtTMYHG2NR9URAhQ+AJ9KRnAjGBrWXmIDllL/yxoV+YXd6ACfaGb4 xHx2mOvP2lQaRTr4AeMq3DA= =tl/g -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Hello, on Montag, 21. Juli 2008, Carlos E. R. wrote:
The Monday 2008-07-21 at 17:09 +0200, Peter Poeml wrote:
1) P2P is only suitable for large files; but many files are tiny. Its overhead and complexity makes it unattractive for small files. P2P's success is largely based on its ability to escape legal control.
I read a study of some European university that contradicted this. They experimentally used bittorrent in the intranet to deploy software updates to the entire intranet (thousands of computer updating (windows?) at the same time, and they discovered that it was way faster than having some dedicated update servers in the same intranet. They found a legal use for it that was better that the normally used solution.
Yes, that's a usecase where p2p fits perfectly: - _all_ machines in the network need (and seed) the same file - all machines have 100 MBit upload connections in the intranet - of course some hundred seeders are faster than a singe server, which is limited by hard disk or (with enough RAM) network interface performance - assuming the BitTorrent client is clever enough, it can download the files from its direct neigborhood and not from the other "end" of the network Unfortunately the situation in the internet differs: - unfortunately ;-) must people still run the other[tm] operating system, so the percentage of people who need and seed the openSUSE updates would be much lower - internet connection of most people is worse than in an intranet. Especially, the upload bandwith is usually quite small. This doesn't mean that BitTorrent is useless (I'm seeding the 11.0 KDE CD - 33 GB uploaded so far ;-) - but the overhead doesn't make it very useful for small files IMHO. Regards, Christian Boltz -- [ Yes ] [ No ] ... used for harmless errors or simple questions: "It's high time you had your cup of coffee! Would you like your KDE to prepare one for you?" [Lukas Ocilka in opensuse-factory - YaST2 button styleguide] --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Mon, Jul 21, 2008 at 09:04:14PM +0200, Carlos E. R. wrote:
The Monday 2008-07-21 at 17:09 +0200, Peter Poeml wrote:
"what about bittorent for everything?" [...] The answer is: No.
Bittorrent is not suitable for everything, because
1) P2P is only suitable for large files; but many files are tiny. Its overhead and complexity makes it unattractive for small files. P2P's success is largely based on its ability to escape legal control.
I read a study of some European university that contradicted this. They experimentally used bittorrent in the intranet to deploy software updates to the entire intranet (thousands of computer updating (windows?) at the same time, and they discovered that it was way faster than having some dedicated update servers in the same intranet. They found a legal use for it that was better that the normally used solution.
Not a contradiction - in such in-vitro circumstances this is conceivable, without traffic shaping as done by ISPs. Btw normally in such a network you need (and have) a proxy cache hiearchy anyway. A P2P network can deal well with such a sudden burst of content, that is to be replicated in identical form onto many clients. But with openSUSE we are looking at a completely different worload, we are not talking about 100 files but about a million, with high fluctuation and hard to predict.
For the purpose they used a version of bittorrent client that could not look up in Internet, only intranet.
I don't have the reference handy, but some one may recollect this study.
- -- Cheers, Carlos E. R.
Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development
Hi Andreas, your posting is not about Factory in any way, as far as I can see, and should probably go to another place. I would suggest opensuse-security; there it is of high relevance. On Fri, Jun 20, 2008 at 01:57:12AM +0200, ab wrote:
i just recently discovered that some older 10.2 box is completely out of sync as the updates/patches repository it got subscribed to during install time back then is only having patches from year 2007.
this is total madness. somebody gotta keep this list clean and only distributing reliable and current mirrors.
example is: <http://ftp.gwdg.de/pub/suse/update/10.2/repodata/> (current)
<http://ftp.tu-ilmenau.de/Mirrors/ftp.suse.com/update/10.2/repodata/> (way outdated)
i even re-ran the yast online-update configuration and the novell initscript or whatever it was called (/usr/bin/suse_register ?) re-subscribed me to this lagging mirror several times over again :(
That is something that we can change. Could you please open a bug report about it? I can't fix it myself, but you can assign it to me (poeml at novell com) and I can route it to someone who can fix it.
argh. :(
Sorry about your frustration.
p.s. i somewhere read about zypper updates also coming to older suse systems. is there a way to profit from newer zypper updates on these older suse/opensuse systems that maybe newer zypper builds feature a better and more intelligent handling of repositories and being in-/outof-sync with the main suse sources.
there has to be some sane way of getting rid of unreliable downloadlocations. thanks.
Yes, there is a sane way. This is something which we fixed with 10.3. 10.3 will only send you to mirrors which actually get updates, and in fact the update server will not redirect at all for certain files. Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development
participants (7)
-
ab -
Carlos E. R. -
Christian Boltz -
Eberhard Moenkeberg -
Peter Poeml -
Rajko M. -
Silviu Marin-Caea