[opensuse] round robin dns for the repo mirrors
grr, heads up, ran into a nice little glitch in the whole dns load balancing scheme for the repo mirrors. At least some of the hostnames don't actually point to a server but to a dns loadbalancer that selects from a pool of physical servers. You get whatever you get, unless you want to start using IP's. The problem is, at least sometimes, the different physical hosts _don't have the same contents_ as each other. This royally screwed up an rsync mirror script I have which TRIES to be careful and test for the existence of the target dir before performing a potentially destructive rsync from it. In the past I had whole sets of my local repos deleted when they dissappeared off the remote servers. So, I added a check to my script, instead of just rsyncing, check if it's still there first, then rsync for real only after verifying that it's still there. That kind of all blows up if I test a directory on mirrors.kernel.org and it's THERE, and then I do a real rsync to the same exact host/path, only this time I got a different IP and the directory is NOT THERE. What the heck can you even DO in that case? Even removing --del from my rsync options doesn't solve it. My repo's getting cluttered up with old files I could live with, but if I never know if the index files (whose filenames don't change as updates are applied) are some old copy from an out of date host or a current copy at any given moment, depending on the luck of the draw the last time the mirror script ran. The exact example I noticed just now was I was working on the mirror script and using 10.2/oss as a test case for a repo that has dissappeared off most servers. I was trying to get it so that when 10.3 goes away, my script notices it and doesn't wipe out my local copy that day. When I run this test, it alternates every other time, pass, fail, pass, fail, ... tst () { rsync -qn rsync://${1}/ >/dev/null 2>&1 && echo yes || echo no ; } X="mirrors.kernel.org/mirrors/opensuse/distribution/10.2/repo/oss" tst $X yes tst $X no tst $X yes tst $X no If I ping -c 1 mirrors.kernel.org repeatedly, it flip flops between two different ips, which explains everything. That might explain Davids mystery with that X11/ati repo the other day too. I'm glad I noticed this before 10.3 disappears, because obviously even a good simple empirical test like this isn't useful and my stuff would have been deleted anyways. Which would have been highly..something, annoying or frustrating arent strong enough words. -- Brian K. White brian@aljex.com http://profile.to/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
If I ping -c 1 mirrors.kernel.org repeatedly, it flip flops between two different ips, which explains everything.
That is broken per my understanding of how dns was designed, but I may be out of date. Same thing happens on my laptop which I just installed with vanilla 11.1 last week. No network tweaks. I'm connected right now via DNS to a wireless router. Per /etc/resolv.conf I'm using the router as a dns server. I think it is pointing to a Comcast dns server. (Not sure about that.) I'm pretty sure that a local dns server should be caching the ip for mirrors.kernel.org for at least a few minutes. (And often for hours.) I've not done much with round robbin dns, but I'm pretty sure the same applies. So one of: a) mirrors.kernel.org has a misconfigured dns setup b) Comcast has a misconfigured dns caching setup c) my wireless router is misconfigured d) OpenSUSE out of the box using dhcp has a misconfigured dns setup. All seem pretty likely, and I could see Brian and I having the same issue, even if he uses a different ISP or router. Before we start blaming anyone, maybe someone can say how it is supposed to work. Greg -- Greg Freemyer Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer First 99 Days Litigation White Paper - http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Feb 11, 2009 at 10:02:50AM -0500, Greg Freemyer wrote:
If I ping -c 1 mirrors.kernel.org repeatedly, it flip flops between two different ips, which explains everything.
That is broken per my understanding of how dns was designed, but I may be out of date. Same thing happens on my laptop which I just installed with vanilla 11.1 last week. No network tweaks.
I'm connected right now via DNS to a wireless router. Per /etc/resolv.conf I'm using the router as a dns server. I think it is pointing to a Comcast dns server. (Not sure about that.)
I'm pretty sure that a local dns server should be caching the ip for mirrors.kernel.org for at least a few minutes. (And often for hours.)
I've not done much with round robbin dns, but I'm pretty sure the same applies.
So one of:
a) mirrors.kernel.org has a misconfigured dns setup b) Comcast has a misconfigured dns caching setup c) my wireless router is misconfigured d) OpenSUSE out of the box using dhcp has a misconfigured dns setup.
Neither of these is the case; nothing is wrong.
All seem pretty likely, and I could see Brian and I having the same issue, even if he uses a different ISP or router.
Before we start blaming anyone, maybe someone can say how it is supposed to work.
It's just something that one often doesn't expect, and which is not optimal for a large file server which long sync times and frequently changing content. (Maybe kernel.org should actually run MirrorBrain in front of the servers, the redirector that we have developed at openSUSE) Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development
Brian K. White wrote:
That kind of all blows up if I test a directory on mirrors.kernel.org and it's THERE, and then I do a real rsync to the same exact host/path, only this time I got a different IP and the directory is NOT THERE.
You could lookup, cache, test, use the IP?
What the heck can you even DO in that case? Even removing --del from my rsync options doesn't solve it.
You don't have a --delete as well by any chance ?? How about --max-delete=NUM ?? Or make the downloads using a multi-version script (I use dirvish). You can delete the old version afterwards if you wish. Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
----- Original Message ----- From: "Dave Howorth" <dhoworth@mrc-lmb.cam.ac.uk> To: <opensuse@opensuse.org> Sent: Wednesday, February 11, 2009 10:30 AM Subject: Re: [opensuse] round robin dns for the repo mirrors
Brian K. White wrote:
That kind of all blows up if I test a directory on mirrors.kernel.org and it's THERE, and then I do a real rsync to the same exact host/path, only this time I got a different IP and the directory is NOT THERE.
You could lookup, cache, test, use the IP?
*headsmack* perfectly good idea of course.
What the heck can you even DO in that case? Even removing --del from my rsync options doesn't solve it.
You don't have a --delete as well by any chance ??
Yes. Removing it would help, and in fact that's what I did, but it doesn't solve the problem, because instead of the dir being gone, it could just be out of date. For the rpms that's actually tolerable, sorta, because the different versions of the same packages have different filenames, but all the index files have filenames that don't change over time, so, doing an update and getting my repodata.xml overwritten with an old one still screws up my client boxes that try to do updates from my repo. However, with the changing ip problem solved above I can handle this possibility by picking some file that's always supposed to be there in a regular known static spot, like an index, and compare it's timestamp against my local copy.
How about --max-delete=NUM ??
Or make the downloads using a multi-version script (I use dirvish). You can delete the old version afterwards if you wish.
Thats probably the simplest answer. Let the update do whatever it will, and just be able to roll it back. Because honestly usually it's fine, so I'd only be having to actually roll back once in a great while, or never if the next update fixed it before I happened to need that repo. -- Brian K. White brian@aljex.com http://profile.to/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
how to config xdmcp? ----- Original Message ----- From: "Brian K. White" <brian@aljex.com> To: <opensuse@opensuse.org> Sent: Wednesday, February 11, 2009 10:26 PM Subject: [opensuse] round robin dns for the repo mirrors
grr, heads up, ran into a nice little glitch in the whole dns load balancing scheme for the repo mirrors.
At least some of the hostnames don't actually point to a server but to a dns loadbalancer that selects from a pool of physical servers. You get whatever you get, unless you want to start using IP's.
The problem is, at least sometimes, the different physical hosts _don't have the same contents_ as each other. This royally screwed up an rsync mirror script I have which TRIES to be careful and test for the existence of the target dir before performing a potentially destructive rsync from it. In the past I had whole sets of my local repos deleted when they dissappeared off the remote servers. So, I added a check to my script, instead of just rsyncing, check if it's still there first, then rsync for real only after verifying that it's still there.
That kind of all blows up if I test a directory on mirrors.kernel.org and it's THERE, and then I do a real rsync to the same exact host/path, only this time I got a different IP and the directory is NOT THERE. What the heck can you even DO in that case? Even removing --del from my rsync options doesn't solve it. My repo's getting cluttered up with old files I could live with, but if I never know if the index files (whose filenames don't change as updates are applied) are some old copy from an out of date host or a current copy at any given moment, depending on the luck of the draw the last time the mirror script ran.
The exact example I noticed just now was I was working on the mirror script and using 10.2/oss as a test case for a repo that has dissappeared off most servers. I was trying to get it so that when 10.3 goes away, my script notices it and doesn't wipe out my local copy that day.
When I run this test, it alternates every other time, pass, fail, pass, fail, ...
tst () { rsync -qn rsync://${1}/ >/dev/null 2>&1 && echo yes || echo no ; } X="mirrors.kernel.org/mirrors/opensuse/distribution/10.2/repo/oss"
tst $X yes tst $X no tst $X yes tst $X no
If I ping -c 1 mirrors.kernel.org repeatedly, it flip flops between two different ips, which explains everything.
That might explain Davids mystery with that X11/ati repo the other day too.
I'm glad I noticed this before 10.3 disappears, because obviously even a good simple empirical test like this isn't useful and my stuff would have been deleted anyways. Which would have been highly..something, annoying or frustrating arent strong enough words.
-- Brian K. White brian@aljex.com http://profile.to/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk!
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
__________ Information from ESET NOD32 Antivirus, version of virus signature database 3846 (20090211) __________
The message was checked by ESET NOD32 Antivirus.
N§²æìr¸yéZ)z{.±ï®Ë±Êâmê)z{.±ê+Z+i×b¶*'jW(f§vǦj)h¥éìºÇ¾ éi¢§²ë ¢¸
On 2/11/09, Jian Nianchuan <jnnccc@shao.ac.cn> wrote:
Huh? What do you need? Allen Registered Linux User 484485 (http://counter.li.org/) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hi Brian, On Wed, Feb 11, 2009 at 09:26:06AM -0500, Brian K. White wrote:
grr, heads up, ran into a nice little glitch in the whole dns load balancing scheme for the repo mirrors.
Repo mirrors? Do you mean http://download.opensuse.org/repositories/ mirrors -- i.e., these http://mirrors.opensuse.org/list/bs.html ?
At least some of the hostnames don't actually point to a server but to a dns loadbalancer that selects from a pool of physical servers. You get whatever you get, unless you want to start using IP's.
Well, that's DNS round robin. It is normally deployed when it indeed doesn't matter which IP you end up using, and when it doesn't matter if you use the one or the other across a series of requests. There is a number of use cases for that. Among them, download servers, provided that they have content that is identical enough (well enough synchronized) so clients don't run into problems. For short-lived content / content with high turn-over rate, as some buildservice repositories certainly are, this _can_ be a problem, especially since this might not be expected by the operators as such. Hence, we (openSUSE, i.e. download.opensuse.org) doesn't send you to such round-robin'd hostnames. Yes, download.opensuse.org is good for you [referring to HTTP access here], because problems like that can be fixed in a central place, so not everybody needs to loose hair over it. For the kernel.org mirrors, DNSrr is in use, and there is an additional complexity - GeoDNS, which makes DNS resolve to different IP adresses / DNS Aliases depending on the client's region of the world. Yes, also this issue is handled correctly for you by download.o.o. (Give up ;-) No, seriously, I can tell you what to do. Use 'host' and look at the returned data: from Europe, you'll see: # host mirrors.kernel.org mirrors.kernel.org is an alias for mirrors.geo.kernel.org. mirrors.geo.kernel.org is an alias for mirrors.eu.kernel.org. mirrors.eu.kernel.org has address 130.239.17.6 mirrors.eu.kernel.org has address 199.6.1.174 [...] from the US, you'll see: # host mirrors.kernel.org mirrors.kernel.org is an alias for mirrors.geo.kernel.org. mirrors.geo.kernel.org is an alias for mirrors.us.kernel.org. mirrors.us.kernel.org has address 149.20.20.135 mirrors.us.kernel.org has address 204.152.191.39 As you see, the hostname points to some other hostname, and in each case there are two of them. The two entries are treated equivalently by resolvers and they return them in random order, so any of them gets used. Which is the DNS round robin. You see the DNS Aliases, though, so you can of course use them directly. Note about HTTP: Provided that (!) the web server virtual host setup is covering this use case, you'll get the same result, but can make your client stick to one server. If the the virtual host setup is done in a way that it only responds to the "main" DNS name, then you are out of luck of course. It's not the case for the kernel.org mirrors, though. Well, for rsync, the vhost issue doesn't matter. Ah, I nearly forgot one crucial bit. Use a reverse lookup to see whether there's a hostname record for the rr'd IP addresses: % host 204.152.191.39 39.191.152.204.in-addr.arpa is an alias for 39.32-27.191.152.204.in-addr.arpa. 39.32-27.191.152.204.in-addr.arpa domain name pointer mirrors2.kernel.org. % host 149.20.20.135 135.20.20.149.in-addr.arpa is an alias for 135.128-27.20.20.149.in-addr.arpa. 135.128-27.20.20.149.in-addr.arpa domain name pointer mirrors1.kernel.org. Et voila, there we see that there's a hostname that can be used to access any of the machines directly: % host mirrors1.kernel.org. mirrors1.kernel.org has address 149.20.20.135 [...] % host mirrors2.kernel.org. mirrors2.kernel.org has address 204.152.191.39 [...] This is exactly what I use to scan those two hosts. Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development
----- Original Message ----- From: "Peter Poeml" <poeml@suse.de> To: <opensuse@opensuse.org> Sent: Wednesday, February 11, 2009 8:50 PM Subject: Re: [opensuse] round robin dns for the repo mirrors
Hi Brian,
[...] No, seriously, I can tell you what to do. Use 'host' and look at the returned data:
from the US, you'll see: # host mirrors.kernel.org mirrors.kernel.org is an alias for mirrors.geo.kernel.org. mirrors.geo.kernel.org is an alias for mirrors.us.kernel.org. mirrors.us.kernel.org has address 149.20.20.135 mirrors.us.kernel.org has address 204.152.191.39
As you see, the hostname points to some other hostname, and in each case there are two of them. The two entries are treated equivalently by resolvers and they return them in random order, so any of them gets used. Which is the DNS round robin.
You see the DNS Aliases, though, so you can of course use them directly.
[...]
Ah, I nearly forgot one crucial bit. Use a reverse lookup to see whether there's a hostname record for the rr'd IP addresses:
% host 204.152.191.39 39.191.152.204.in-addr.arpa is an alias for 39.32-27.191.152.204.in-addr.arpa. 39.32-27.191.152.204.in-addr.arpa domain name pointer mirrors2.kernel.org. % host 149.20.20.135 135.20.20.149.in-addr.arpa is an alias for 135.128-27.20.20.149.in-addr.arpa. 135.128-27.20.20.149.in-addr.arpa domain name pointer mirrors1.kernel.org.
Et voila, there we see that there's a hostname that can be used to access any of the machines directly:
% host mirrors1.kernel.org. mirrors1.kernel.org has address 149.20.20.135 [...] % host mirrors2.kernel.org. mirrors2.kernel.org has address 204.152.191.39 [...]
This is exactly what I use to scan those two hosts.
Nice run-down. Thanks much. -- Brian K. White brian@aljex.com http://profile.to/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (6)
-
Allen Zhu -
Brian K. White -
Dave Howorth -
Greg Freemyer -
Jian Nianchuan -
Peter Poeml