Mailinglist Archive: opensuse-bugs (15136 mails)
| < Previous | Next > |
[Bug 328845] New: using timestamp_file column for redirection?
- From: bugzilla_noreply@xxxxxxxxxx
- Date: Thu, 27 Sep 2007 03:59:26 -0600 (MDT)
- Message-id: <bug-328845-21960@xxxxxxxxxxxxxxxxxxxxxxxxx/>
https://bugzilla.novell.com/show_bug.cgi?id=328845
Summary: using timestamp_file column for redirection?
Product: openSUSE.org
Version: unspecified
Platform: Other
OS/Version: Other
Status: NEW
Severity: Enhancement
Priority: P5 - None
Component: Download Infrastructure
AssignedTo: jw@xxxxxxxxxx
ReportedBy: poeml@xxxxxxxxxx
QAContact: adrian@xxxxxxxxxx
CC: cthiel@xxxxxxxxxx
Found By: ---
We have this configuration for Factory and for the update tree:
# In the Factory tree(s), metadata changes frequently.
# Do not redirect for files which do not change their filename on a change
# and might contain references to stale files.
# Rule: if a filename contains no digit, it is not "versioned".
<DirectoryMatch "^/srv/ftp/pub/opensuse/distribution/.*[Ff]actory">
ZrkadloExcludeFileMask "/[^0-9]*$"
</DirectoryMatch>
There is a pretty large file (135MB, served with 1GB per hour currently) named
distribution/SL-OSS-factory/inst-source/boot/boot.iso which is not being
redirected due to this exception.
Our mirror scanner saves a timestamp actually, and we could redirect
based on a refined SQL query which takes the mtime of the file into
account.
However, not all mirrors show the same mtime, even though most have the
up to date file.
stat
/srv/ftp/pub/opensuse/distribution/SL-OSS-factory/inst-source/boot/boot.iso
File:
`/srv/ftp/pub/opensuse/distribution/SL-OSS-factory/inst-source/boot/boot.iso'
Size: 160917504 Blocks: 314296 IO Block: 4096 regular file
Device: 820h/2080d Inode: 2155185605 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1002/ mirror) Gid: ( 102/haldaemon)
Access: 2007-09-24 10:53:41.000000000 +0200
Modify: 2007-09-24 08:10:03.000000000 +0200
Change: 2007-09-25 17:26:44.941985833 +0200
mysql> select count(*) from file_server where fileid=319671 and
timestamp_file='2007-09-24 08:10:03';
+----------+
| count(*) |
+----------+
| 12 |
+----------+
1 row in set (0.01 sec)
mysql> select count(*) from file_server where fileid=319671;
+----------+
| count(*) |
+----------+
| 19 |
+----------+
1 row in set (0.00 sec)
Some have mtime 6:10:00 instead of 8:10:03:
mysql> SELECT file_server.serverid, server.baseurl, file_server.timestamp_file
FROM file_server LEFT JOIN server ON file_server.serverid = server.id WHERE
file_server.path_md5='KOPBSLwaAof/Gl6ZF1goaw' AND server.enabled=1 AND
server.status_baseurl=1 AND server.score > 0 order by timestamp_file;
+----------+------------------------------------------------------------------+---------------------+
| serverid | baseurl |
timestamp_file |
+----------+------------------------------------------------------------------+---------------------+
| 52 | http://mirrors.kernel.org/opensuse/ ;|
2007-09-05 20:55:53 |
| 36 | http://ftp.riken.jp/Linux/opensuse/ ;|
2007-09-19 20:19:57 |
| 29 | http://ftp.kddilabs.jp/Linux/packages/opensuse/ ;|
2007-09-19 20:19:57 |
| 22 | http://ftp.hosteurope.de/mirror/ftp.opensuse.org/ ;|
2007-09-24 06:10:00 |
| 33 | http://ftp.novell.hu/pub/mirrors/ftp.opensuse.org/opensuse/ ;|
2007-09-24 06:10:00 |
| 81 | http://mirror.internode.on.net/pub/opensuse/ ;|
2007-09-24 06:10:00 |
| 41 | http://download.uni-hd.de/ftp/pub/linux/opensuse/ ;|
2007-09-24 06:10:00 |
| 79 | http://ftp.halifax.rwth-aachen.de/opensuse/ ;|
2007-09-24 08:10:03 |
| 91 | http://rm.mirror.garr.it/mirrors/opensuse/ ;|
2007-09-24 08:10:03 |
| 39 | http://ftp.tu-chemnitz.de/pub/linux/opensuse/ ;|
2007-09-24 08:10:03 |
| 86 | http://ftp.linux.cz/pub/linux/opensuse/ ;|
2007-09-24 08:10:03 |
| 44 | http://ftp5.gwdg.de/pub/opensuse/ ;|
2007-09-24 08:10:03 |
| 21 | http://ftp.heanet.ie/mirrors/ftp.opensuse.org/opensuse/ ;|
2007-09-24 08:10:03 |
| 53 | http://opensuse.mirrors.skynet.be/pub/ftp.opensuse.org/opensuse/ ;|
2007-09-24 08:10:03 |
| 32 | http://ftp.novell.co.jp/pub/opensuse/ ;|
2007-09-24 08:10:03 |
| 65 | http://ftp.estpak.ee/pub/suse/opensuse/ ;|
2007-09-24 08:10:03 |
| 97 | http://widehat.opensuse.org/ ;|
2007-09-24 08:10:03 |
| 13 | http://ftp-stud.fht-esslingen.de/pub/Mirrors/ftp.opensuse.org/ ;|
2007-09-24 08:10:03 |
+----------+------------------------------------------------------------------+---------------------+
This shows two problems.
Let's take this example:
% curl -sI
'http://ftp.novell.hu/pub/mirrors/ftp.opensuse.org/opensuse/distribution/SL-OSS-factory/inst-source/boot/boot.iso'
Last-Modified: Mon, 24 Sep 2007 06:10:03 GMT
ftp.novell.hu is scanned via FTP protocol.
a) The time is shifted by two hours. This is also the case when the file
mtime is viewed via Apache. The shift may already occur when mirroring
from us, I don't know.
b) The scanner sometimes (when scanning via FTP?) doesn't see (and
store) the seconds (03 in this case).
Questions:
1) can this be "fixed"? If fixable.
2) might granularity on minutes be sufficient, so we could get
rid of the seconds?
3) can we fix (or compensate for) the two hours shift?
It seems systematic, and could possibly be determined by an automatic
probe.
Anyway, right now, it is very good that we have the "/[^0-9]*$"
redirection exclusion for Factory, because otherwise we would redirect
to a very obsolete copy on mirrors.kernel.org, which has not been synced
since three weeks... although this means that we serve a pretty large
file ourselves.
Juergen, what do you think?
--
Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.
Summary: using timestamp_file column for redirection?
Product: openSUSE.org
Version: unspecified
Platform: Other
OS/Version: Other
Status: NEW
Severity: Enhancement
Priority: P5 - None
Component: Download Infrastructure
AssignedTo: jw@xxxxxxxxxx
ReportedBy: poeml@xxxxxxxxxx
QAContact: adrian@xxxxxxxxxx
CC: cthiel@xxxxxxxxxx
Found By: ---
We have this configuration for Factory and for the update tree:
# In the Factory tree(s), metadata changes frequently.
# Do not redirect for files which do not change their filename on a change
# and might contain references to stale files.
# Rule: if a filename contains no digit, it is not "versioned".
<DirectoryMatch "^/srv/ftp/pub/opensuse/distribution/.*[Ff]actory">
ZrkadloExcludeFileMask "/[^0-9]*$"
</DirectoryMatch>
There is a pretty large file (135MB, served with 1GB per hour currently) named
distribution/SL-OSS-factory/inst-source/boot/boot.iso which is not being
redirected due to this exception.
Our mirror scanner saves a timestamp actually, and we could redirect
based on a refined SQL query which takes the mtime of the file into
account.
However, not all mirrors show the same mtime, even though most have the
up to date file.
stat
/srv/ftp/pub/opensuse/distribution/SL-OSS-factory/inst-source/boot/boot.iso
File:
`/srv/ftp/pub/opensuse/distribution/SL-OSS-factory/inst-source/boot/boot.iso'
Size: 160917504 Blocks: 314296 IO Block: 4096 regular file
Device: 820h/2080d Inode: 2155185605 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1002/ mirror) Gid: ( 102/haldaemon)
Access: 2007-09-24 10:53:41.000000000 +0200
Modify: 2007-09-24 08:10:03.000000000 +0200
Change: 2007-09-25 17:26:44.941985833 +0200
mysql> select count(*) from file_server where fileid=319671 and
timestamp_file='2007-09-24 08:10:03';
+----------+
| count(*) |
+----------+
| 12 |
+----------+
1 row in set (0.01 sec)
mysql> select count(*) from file_server where fileid=319671;
+----------+
| count(*) |
+----------+
| 19 |
+----------+
1 row in set (0.00 sec)
Some have mtime 6:10:00 instead of 8:10:03:
mysql> SELECT file_server.serverid, server.baseurl, file_server.timestamp_file
FROM file_server LEFT JOIN server ON file_server.serverid = server.id WHERE
file_server.path_md5='KOPBSLwaAof/Gl6ZF1goaw' AND server.enabled=1 AND
server.status_baseurl=1 AND server.score > 0 order by timestamp_file;
+----------+------------------------------------------------------------------+---------------------+
| serverid | baseurl |
timestamp_file |
+----------+------------------------------------------------------------------+---------------------+
| 52 | http://mirrors.kernel.org/opensuse/ ;|
2007-09-05 20:55:53 |
| 36 | http://ftp.riken.jp/Linux/opensuse/ ;|
2007-09-19 20:19:57 |
| 29 | http://ftp.kddilabs.jp/Linux/packages/opensuse/ ;|
2007-09-19 20:19:57 |
| 22 | http://ftp.hosteurope.de/mirror/ftp.opensuse.org/ ;|
2007-09-24 06:10:00 |
| 33 | http://ftp.novell.hu/pub/mirrors/ftp.opensuse.org/opensuse/ ;|
2007-09-24 06:10:00 |
| 81 | http://mirror.internode.on.net/pub/opensuse/ ;|
2007-09-24 06:10:00 |
| 41 | http://download.uni-hd.de/ftp/pub/linux/opensuse/ ;|
2007-09-24 06:10:00 |
| 79 | http://ftp.halifax.rwth-aachen.de/opensuse/ ;|
2007-09-24 08:10:03 |
| 91 | http://rm.mirror.garr.it/mirrors/opensuse/ ;|
2007-09-24 08:10:03 |
| 39 | http://ftp.tu-chemnitz.de/pub/linux/opensuse/ ;|
2007-09-24 08:10:03 |
| 86 | http://ftp.linux.cz/pub/linux/opensuse/ ;|
2007-09-24 08:10:03 |
| 44 | http://ftp5.gwdg.de/pub/opensuse/ ;|
2007-09-24 08:10:03 |
| 21 | http://ftp.heanet.ie/mirrors/ftp.opensuse.org/opensuse/ ;|
2007-09-24 08:10:03 |
| 53 | http://opensuse.mirrors.skynet.be/pub/ftp.opensuse.org/opensuse/ ;|
2007-09-24 08:10:03 |
| 32 | http://ftp.novell.co.jp/pub/opensuse/ ;|
2007-09-24 08:10:03 |
| 65 | http://ftp.estpak.ee/pub/suse/opensuse/ ;|
2007-09-24 08:10:03 |
| 97 | http://widehat.opensuse.org/ ;|
2007-09-24 08:10:03 |
| 13 | http://ftp-stud.fht-esslingen.de/pub/Mirrors/ftp.opensuse.org/ ;|
2007-09-24 08:10:03 |
+----------+------------------------------------------------------------------+---------------------+
This shows two problems.
Let's take this example:
% curl -sI
'http://ftp.novell.hu/pub/mirrors/ftp.opensuse.org/opensuse/distribution/SL-OSS-factory/inst-source/boot/boot.iso'
Last-Modified: Mon, 24 Sep 2007 06:10:03 GMT
ftp.novell.hu is scanned via FTP protocol.
a) The time is shifted by two hours. This is also the case when the file
mtime is viewed via Apache. The shift may already occur when mirroring
from us, I don't know.
b) The scanner sometimes (when scanning via FTP?) doesn't see (and
store) the seconds (03 in this case).
Questions:
1) can this be "fixed"? If fixable.
2) might granularity on minutes be sufficient, so we could get
rid of the seconds?
3) can we fix (or compensate for) the two hours shift?
It seems systematic, and could possibly be determined by an automatic
probe.
Anyway, right now, it is very good that we have the "/[^0-9]*$"
redirection exclusion for Factory, because otherwise we would redirect
to a very obsolete copy on mirrors.kernel.org, which has not been synced
since three weeks... although this means that we serve a pretty large
file ourselves.
Juergen, what do you think?
--
Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.
| < Previous | Next > |