https://bugzilla.novell.com/show_bug.cgi?id=328845 Summary: using timestamp_file column for redirection? Product: openSUSE.org Version: unspecified Platform: Other OS/Version: Other Status: NEW Severity: Enhancement Priority: P5 - None Component: Download Infrastructure AssignedTo: jw@novell.com ReportedBy: poeml@novell.com QAContact: adrian@novell.com CC: cthiel@novell.com Found By: --- We have this configuration for Factory and for the update tree: # In the Factory tree(s), metadata changes frequently. # Do not redirect for files which do not change their filename on a change # and might contain references to stale files. # Rule: if a filename contains no digit, it is not "versioned". ZrkadloExcludeFileMask "/[^0-9]*$" </DirectoryMatch> There is a pretty large file (135MB, served with 1GB per hour currently) named distribution/SL-OSS-factory/inst-source/boot/boot.iso which is not being redirected due to this exception. Our mirror scanner saves a timestamp actually, and we could redirect based on a refined SQL query which takes the mtime of the file into account. However, not all mirrors show the same mtime, even though most have the up to date file. stat /srv/ftp/pub/opensuse/distribution/SL-OSS-factory/inst-source/boot/boot.iso File: `/srv/ftp/pub/opensuse/distribution/SL-OSS-factory/inst-source/boot/boot.iso' Size: 160917504 Blocks: 314296 IO Block: 4096 regular file Device: 820h/2080d Inode: 2155185605 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1002/ mirror) Gid: ( 102/haldaemon) Access: 2007-09-24 10:53:41.000000000 +0200 Modify: 2007-09-24 08:10:03.000000000 +0200 Change: 2007-09-25 17:26:44.941985833 +0200 mysql> select count(*) from file_server where fileid=319671 and timestamp_file='2007-09-24 08:10:03'; +----------+ | count(*) | +----------+ | 12 | +----------+ 1 row in set (0.01 sec) mysql> select count(*) from file_server where fileid=319671; +----------+ | count(*) | +----------+ | 19 | +----------+ 1 row in set (0.00 sec) Some have mtime 6:10:00 instead of 8:10:03: mysql> SELECT file_server.serverid, server.baseurl, file_server.timestamp_file FROM file_server LEFT JOIN server ON file_server.serverid = server.id WHERE file_server.path_md5='KOPBSLwaAof/Gl6ZF1goaw' AND server.enabled=1 AND server.status_baseurl=1 AND server.score > 0 order by timestamp_file; +----------+------------------------------------------------------------------+---------------------+ | serverid | baseurl | timestamp_file | +----------+------------------------------------------------------------------+---------------------+ | 52 | http://mirrors.kernel.org/opensuse/ | 2007-09-05 20:55:53 | | 36 | http://ftp.riken.jp/Linux/opensuse/ | 2007-09-19 20:19:57 | | 29 | http://ftp.kddilabs.jp/Linux/packages/opensuse/ | 2007-09-19 20:19:57 | | 22 | http://ftp.hosteurope.de/mirror/ftp.opensuse.org/ | 2007-09-24 06:10:00 | | 33 | http://ftp.novell.hu/pub/mirrors/ftp.opensuse.org/opensuse/ | 2007-09-24 06:10:00 | | 81 | http://mirror.internode.on.net/pub/opensuse/ | 2007-09-24 06:10:00 | | 41 | http://download.uni-hd.de/ftp/pub/linux/opensuse/ | 2007-09-24 06:10:00 | | 79 | http://ftp.halifax.rwth-aachen.de/opensuse/ | 2007-09-24 08:10:03 | | 91 | http://rm.mirror.garr.it/mirrors/opensuse/ | 2007-09-24 08:10:03 | | 39 | http://ftp.tu-chemnitz.de/pub/linux/opensuse/ | 2007-09-24 08:10:03 | | 86 | http://ftp.linux.cz/pub/linux/opensuse/ | 2007-09-24 08:10:03 | | 44 | http://ftp5.gwdg.de/pub/opensuse/ | 2007-09-24 08:10:03 | | 21 | http://ftp.heanet.ie/mirrors/ftp.opensuse.org/opensuse/ | 2007-09-24 08:10:03 | | 53 | http://opensuse.mirrors.skynet.be/pub/ftp.opensuse.org/opensuse/ | 2007-09-24 08:10:03 | | 32 | http://ftp.novell.co.jp/pub/opensuse/ | 2007-09-24 08:10:03 | | 65 | http://ftp.estpak.ee/pub/suse/opensuse/ | 2007-09-24 08:10:03 | | 97 | http://widehat.opensuse.org/ | 2007-09-24 08:10:03 | | 13 | http://ftp-stud.fht-esslingen.de/pub/Mirrors/ftp.opensuse.org/ | 2007-09-24 08:10:03 | +----------+------------------------------------------------------------------+---------------------+ This shows two problems. Let's take this example: % curl -sI 'http://ftp.novell.hu/pub/mirrors/ftp.opensuse.org/opensuse/distribution/SL-O...' Last-Modified: Mon, 24 Sep 2007 06:10:03 GMT ftp.novell.hu is scanned via FTP protocol. a) The time is shifted by two hours. This is also the case when the file mtime is viewed via Apache. The shift may already occur when mirroring from us, I don't know. b) The scanner sometimes (when scanning via FTP?) doesn't see (and store) the seconds (03 in this case). Questions: 1) can this be "fixed"? If fixable. 2) might granularity on minutes be sufficient, so we could get rid of the seconds? 3) can we fix (or compensate for) the two hours shift? It seems systematic, and could possibly be determined by an automatic probe. Anyway, right now, it is very good that we have the "/[^0-9]*$" redirection exclusion for Factory, because otherwise we would redirect to a very obsolete copy on mirrors.kernel.org, which has not been synced since three weeks... although this means that we serve a pretty large file ourselves. Juergen, what do you think? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.