Hi! ftp.klid.dk is a mirror in Denmark, and we have had a problem with the data there, as we were using a rsync source that was not responding. For a week or so we have used another rsync source, and we should be OK now, but we are not showing up on the opensuse mirror list as having 11.1 isos, repo and update. What is wrong? best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi Keld, On Tue, Feb 24, 2009 at 10:05:20PM +0100, Keld Jørn Simonsen wrote:
Hi!
ftp.klid.dk is a mirror in Denmark, and we have had a problem with the data there, as we were using a rsync source that was not responding. For a week or so we have used another rsync source, and we should be OK now, but we are not showing up on the opensuse mirror list as having 11.1 isos, repo and update. What is wrong?
best regards keld
Hm, let's look. I see you on http://mirrors.opensuse.org/ and in the config everything looks alright. I see you on http://download.opensuse.org/update/11.1/rpm/i586/MozillaFirefox-3.0.4_3.0.5... and http://download.opensuse.org/distribution/11.0/iso/cd/openSUSE-11.0-Addon-La... but not for files in the /distribution/11.1 directory. I see what's wrong in the scan log: ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. The FTP server closed the connection; probably it does CPU usage accounting and it decided that this client is using too many resources. Thus the scanner could not advance to 11.1 in the tree. Can you switch that off for 195.135.220.0/22, or -better- create an rsync module for me? Access to 195.135.220.0/22 would be sufficient. Thanks! Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Tue, Feb 24, 2009 at 10:22:34PM +0100, Peter Poeml wrote:
Hi Keld,
On Tue, Feb 24, 2009 at 10:05:20PM +0100, Keld Jørn Simonsen wrote:
Hi!
ftp.klid.dk is a mirror in Denmark, and we have had a problem with the data there, as we were using a rsync source that was not responding. For a week or so we have used another rsync source, and we should be OK now, but we are not showing up on the opensuse mirror list as having 11.1 isos, repo and update. What is wrong?
best regards keld
Hm, let's look. I see you on http://mirrors.opensuse.org/ and in the config everything looks alright.
I see you on http://download.opensuse.org/update/11.1/rpm/i586/MozillaFirefox-3.0.4_3.0.5... and http://download.opensuse.org/distribution/11.0/iso/cd/openSUSE-11.0-Addon-La...
but not for files in the /distribution/11.1 directory.
I see what's wrong in the scan log:
ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing.
The FTP server closed the connection; probably it does CPU usage accounting and it decided that this client is using too many resources.
Thus the scanner could not advance to 11.1 in the tree.
Can you switch that off for 195.135.220.0/22, or -better- create an rsync module for me? Access to 195.135.220.0/22 would be sufficient.
I do not know why the 550 code is generated, and I dont know how to then change that for the opensuse probing - could you tell me where to look? It seems strange that the probing is ok for 11.0 but not for 11.1 Best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi Keld, On Wed, Feb 25, 2009 at 10:58:32AM +0100, Keld Jørn Simonsen wrote:
On Tue, Feb 24, 2009 at 10:22:34PM +0100, Peter Poeml wrote:
Hi Keld,
On Tue, Feb 24, 2009 at 10:05:20PM +0100, Keld Jørn Simonsen wrote:
Hi!
ftp.klid.dk is a mirror in Denmark, and we have had a problem with the data there, as we were using a rsync source that was not responding. For a week or so we have used another rsync source, and we should be OK now, but we are not showing up on the opensuse mirror list as having 11.1 isos, repo and update. What is wrong?
best regards keld
Hm, let's look. I see you on http://mirrors.opensuse.org/ and in the config everything looks alright.
I see you on http://download.opensuse.org/update/11.1/rpm/i586/MozillaFirefox-3.0.4_3.0.5... and http://download.opensuse.org/distribution/11.0/iso/cd/openSUSE-11.0-Addon-La...
but not for files in the /distribution/11.1 directory.
I see what's wrong in the scan log:
ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing.
The FTP server closed the connection; probably it does CPU usage accounting and it decided that this client is using too many resources.
Thus the scanner could not advance to 11.1 in the tree.
Can you switch that off for 195.135.220.0/22, or -better- create an rsync module for me? Access to 195.135.220.0/22 would be sufficient.
I do not know why the 550 code is generated, and I dont know how to then change that for the opensuse probing - could you tell me where to look? It seems strange that the probing is ok for 11.0 but not for 11.1
The reason for my suspicion is that I see something *similar* with some other mirrors, too, and the server basically closes the connection after a certain time, before the scan completes. (So the files it has seen end up as known in the database, and the remaining files not. Which is 11.1 in your case.) But my suspicion is mostly based on assumptions, because in most cases I have never been able to establish contact with the respective mirror operator, or never heard back from them. In one case it could actually be confirmed by the mirror admin identifying messages in his log that prove it. That was a different FTP server though (not vsftpd). But it was always some 5xx code, so I thought into the same direction regarding your mirror, too. Are there errors related to this in your logs? Checking vsftpd's source code, there are two possible meanings for code 550 in ftpcodes.h: #define FTP_FILEFAIL 550 #define FTP_NOPERM 550 Which doesn't sound like resource limiting; and I don't find options related to cpu usage accounging in vsftpd. 550 seems to indicate local problems or a configuration issue that prevents access to certain files or directories. When manually looking around in ftp://ftp.klid.dk/opensuse/distribution/11.1/ I have no problems though. Maybe too many open files (default ulimit of 1024?) is sometimes reached and could produce this error? I just did another scan, a verbose one, and this one actually completed without a hitch. (So the files missing before are automatically listed in the mirror database now.) Thanks, Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Wed, Feb 25, 2009 at 11:39:58AM +0100, Peter Poeml wrote:
Hi Keld,
On Wed, Feb 25, 2009 at 10:58:32AM +0100, Keld Jørn Simonsen wrote:
On Tue, Feb 24, 2009 at 10:22:34PM +0100, Peter Poeml wrote:
Hi Keld,
On Tue, Feb 24, 2009 at 10:05:20PM +0100, Keld Jørn Simonsen wrote:
Hi!
ftp.klid.dk is a mirror in Denmark, and we have had a problem with the data there, as we were using a rsync source that was not responding. For a week or so we have used another rsync source, and we should be OK now, but we are not showing up on the opensuse mirror list as having 11.1 isos, repo and update. What is wrong?
best regards keld
Hm, let's look. I see you on http://mirrors.opensuse.org/ and in the config everything looks alright.
I see you on http://download.opensuse.org/update/11.1/rpm/i586/MozillaFirefox-3.0.4_3.0.5... and http://download.opensuse.org/distribution/11.0/iso/cd/openSUSE-11.0-Addon-La...
but not for files in the /distribution/11.1 directory.
I see what's wrong in the scan log:
ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing. ftp.klid.dk: ftp status code 550, closing.
The FTP server closed the connection; probably it does CPU usage accounting and it decided that this client is using too many resources.
Thus the scanner could not advance to 11.1 in the tree.
Can you switch that off for 195.135.220.0/22, or -better- create an rsync module for me? Access to 195.135.220.0/22 would be sufficient.
I do not know why the 550 code is generated, and I dont know how to then change that for the opensuse probing - could you tell me where to look? It seems strange that the probing is ok for 11.0 but not for 11.1
The reason for my suspicion is that I see something *similar* with some other mirrors, too, and the server basically closes the connection after a certain time, before the scan completes. (So the files it has seen end up as known in the database, and the remaining files not. Which is 11.1 in your case.)
But my suspicion is mostly based on assumptions, because in most cases I have never been able to establish contact with the respective mirror operator, or never heard back from them. In one case it could actually be confirmed by the mirror admin identifying messages in his log that prove it. That was a different FTP server though (not vsftpd). But it was always some 5xx code, so I thought into the same direction regarding your mirror, too.
Are there errors related to this in your logs?
Checking vsftpd's source code, there are two possible meanings for code 550 in ftpcodes.h: #define FTP_FILEFAIL 550 #define FTP_NOPERM 550 Which doesn't sound like resource limiting; and I don't find options related to cpu usage accounging in vsftpd. 550 seems to indicate local problems or a configuration issue that prevents access to certain files or directories. When manually looking around in ftp://ftp.klid.dk/opensuse/distribution/11.1/ I have no problems though.
Maybe too many open files (default ulimit of 1024?) is sometimes reached and could produce this error?
I just did another scan, a verbose one, and this one actually completed without a hitch. (So the files missing before are automatically listed in the mirror database now.)
It is a bit strange. Could you tell me the ip number/site addres that you are calling from, and which time? The server is busy and that would help me locate the log entries. And, do your script still produce errors? best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi Keld, On Wed, Feb 25, 2009 at 12:53:53PM +0100, Keld Jørn Simonsen wrote: [snip]
It is a bit strange. Could you tell me the ip number/site addres that you are calling from, and which time? The server is busy and that would help me locate the log entries.
195.135.221.0/22 (195.135.221.2)
And, do your script still produce errors?
Looking more deeply at the scanner logs, I noticed more such error messages now, in fact many more than I would expect, with other FTP servers. Indeed, the 550 is an artifact. I no know that two things happen: 1) the used Perl module reports an error wrongly - what happens is a timeout (421), but Perl claims that a 550 happened, which is bollocks. Here's an example: sunsite.informatik.rwth-aachen.de: ftp dir: ftp://sunsite.informatik.rwth-aachen.de/pub/linux/opensuse/distribution/11.0/repo/oss/suse/i686 Net::FTP=GLOB(0x112a290)>>> CWD /pub/linux/opensuse/distribution/11.0/repo/oss/suse/i686 Net::FTP=GLOB(0x112a290)<<< 421 Timeout. sunsite.informatik.rwth-aachen.de: ftp status code 550, closing. Net::FTP=GLOB(0x112a290)>>> QUIT 2) the timeout happens simply because the server closes the connection after a minute or two (which is a completely normal and valid configuration), because our local scanning process takes too long dealing with the file list. So I'll have to fix this with a reconnect, and make the scan work faster to prevent the timeouts in the first place. Thank you very much for your help! And sorry for the false alarm. Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi Keld, On Wed, Feb 25, 2009 at 02:12:08PM +0100, Peter Poeml wrote:
Looking more deeply at the scanner logs, I noticed more such error messages now, in fact many more than I would expect, with other FTP servers.
Indeed, the 550 is an artifact.
I no know that two things happen:
1) the used Perl module reports an error wrongly - what happens is a timeout (421), but Perl claims that a 550 happened, which is bollocks. Here's an example:
sunsite.informatik.rwth-aachen.de: ftp dir: ftp://sunsite.informatik.rwth-aachen.de/pub/linux/opensuse/distribution/11.0/repo/oss/suse/i686 Net::FTP=GLOB(0x112a290)>>> CWD /pub/linux/opensuse/distribution/11.0/repo/oss/suse/i686 Net::FTP=GLOB(0x112a290)<<< 421 Timeout. sunsite.informatik.rwth-aachen.de: ftp status code 550, closing. Net::FTP=GLOB(0x112a290)>>> QUIT
2) the timeout happens simply because the server closes the connection after a minute or two (which is a completely normal and valid configuration), because our local scanning process takes too long dealing with the file list.
So I'll have to fix this with a reconnect, and make the scan work faster to prevent the timeouts in the first place.
I am working around with a reconnect in the scanner now. So the scanning should catch up with all files again, where it is done via FTP. Thanks again for your help, Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
On Thu, Feb 26, 2009 at 12:08:05PM +0100, Peter Poeml wrote:
Hi Keld,
On Wed, Feb 25, 2009 at 02:12:08PM +0100, Peter Poeml wrote:
Looking more deeply at the scanner logs, I noticed more such error messages now, in fact many more than I would expect, with other FTP servers.
Indeed, the 550 is an artifact.
I no know that two things happen:
1) the used Perl module reports an error wrongly - what happens is a timeout (421), but Perl claims that a 550 happened, which is bollocks. Here's an example:
sunsite.informatik.rwth-aachen.de: ftp dir: ftp://sunsite.informatik.rwth-aachen.de/pub/linux/opensuse/distribution/11.0/repo/oss/suse/i686 Net::FTP=GLOB(0x112a290)>>> CWD /pub/linux/opensuse/distribution/11.0/repo/oss/suse/i686 Net::FTP=GLOB(0x112a290)<<< 421 Timeout. sunsite.informatik.rwth-aachen.de: ftp status code 550, closing. Net::FTP=GLOB(0x112a290)>>> QUIT
2) the timeout happens simply because the server closes the connection after a minute or two (which is a completely normal and valid configuration), because our local scanning process takes too long dealing with the file list.
So I'll have to fix this with a reconnect, and make the scan work faster to prevent the timeouts in the first place.
I am working around with a reconnect in the scanner now. So the scanning should catch up with all files again, where it is done via FTP.
Thanks again for your help,
Good to know that the problem was solved. Best regards keld -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
participants (2)
-
Keld Jørn Simonsen
-
Peter Poeml