[Bug 307098] New: zypper request flood
https://bugzilla.novell.com/show_bug.cgi?id=307098 Summary: zypper request flood Product: openSUSE 10.3 Version: Beta 2 Platform: Other OS/Version: Other Status: NEW Severity: Blocker Priority: P5 - None Component: libzypp AssignedTo: kkaempf@novell.com ReportedBy: poeml@novell.com QAContact: kkaempf@novell.com Found By: --- % zypper sa http://download.opensuse.org/repositories/home:/poeml/openSUSE:Factory/ home:poeml 84.44.247.44 - - [03/Sep/2007:09:44:07 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml HTTP/1.1" 206 2 "-" "Novell ZYPP Installer" "-" zypper issues a lot of redundant requests: % zypper refresh 84.44.247.44 - - [03/Sep/2007:09:44:12 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml HTTP/1.1" 200 1198 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:13 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml HTTP/1.1" 200 1198 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:13 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml.key HTTP/1.1" 206 2 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:13 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml.asc HTTP/1.1" 206 2 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:13 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml.key HTTP/1.1" 200 893 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:13 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml.asc HTTP/1.1" 200 189 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:14 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml.key HTTP/1.1" 200 893 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:14 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml.asc HTTP/1.1" 200 189 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:14 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml HTTP/1.1" 200 1198 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:15 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/filelists.xml.gz HTTP/1.1" 200 1862 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:15 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/primary.xml.gz HTTP/1.1" 200 5294 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:15 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/patterns.test2.xml.gz HTTP/1.1" 200 99 "-" "Novell ZYPP Installer" "-" It should look like this, when downloading metadata for the first time: 84.44.247.44 - - [03/Sep/2007:09:44:12 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml HTTP/1.1" 200 1198 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:14 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml.key HTTP/1.1" 200 893 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:14 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml.asc HTTP/1.1" 200 189 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:15 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/filelists.xml.gz HTTP/1.1" 200 1862 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:15 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/primary.xml.gz HTTP/1.1" 200 5294 "-" "Novell ZYPP Installer" "-" 84.44.247.44 - - [03/Sep/2007:09:44:15 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/patterns.test2.xml.gz HTTP/1.1" 200 99 "-" "Novell ZYPP Installer" "-" To increase the survival chance of our download infrastructure, this should be fixed before 10.3 release. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c1
Klaus Kämpf
https://bugzilla.novell.com/show_bug.cgi?id=307098#c2
--- Comment #2 from Peter Poeml
https://bugzilla.novell.com/show_bug.cgi?id=307098#c3
--- Comment #3 from Peter Poeml
https://bugzilla.novell.com/show_bug.cgi?id=307098#c4
Cristian Rodriguez
To increase the survival chance of our download infrastructure, this should be fixed before 10.3 release.
yes otherwise if this stuf makes too much (redundant request * number of users) the redirector will die in the battle field :( -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c5
Christoph Thiel
https://bugzilla.novell.com/show_bug.cgi?id=307098#c6
--- Comment #6 from Stanislav Visnovsky
https://bugzilla.novell.com/show_bug.cgi?id=307098#c7
--- Comment #7 from Cristian Rodriguez
I've played a bit with the debugger:
** check if we need to refresh **
84.44.247.44 - - [03/Sep/2007:09:44:12 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml HTTP/1.1" 200 1198 "-" "Novell ZYPP Installer" "-"
** start refresh **
** does the file exist? ** (MediaCurl::getDoesFileExist)
84.44.247.44 - - [03/Sep/2007:09:44:13 +0200] "GET /repositories/home:/poeml/openSUSE_Factory/repodata/repomd.xml HTTP/1.1" 200 1198 "-" "Novell ZYPP Installer" "-"
huh ? I am missing something ? why does it donwloads the file @_O ? a ¿does the file exist? check sounds like HTTP HEAD request NOT HTTP GET !! if HTTP HEAD returns 200 then the file exists, for local/NFS repos. stat()..., via ftp ..MTDM will return the filemtime in case the file exists or error in case it does not.. that's weird. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c8
--- Comment #8 from Cristian Rodriguez
huh ? I am missing something ?
I have read the source , it fortunately downloads only a tiny bit of data. ;) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c9
Duncan Mac-Vicar
https://bugzilla.novell.com/show_bug.cgi?id=307098#c10
Peter Poeml
https://bugzilla.novell.com/show_bug.cgi?id=307098#c11
--- Comment #11 from Peter Poeml
MDTM README < 213 20040224153912 Last-Modified: Tue, 24 Feb 2004 15:39:12 GMT [...] SIZE README < 213 622 Content-Length: 622
Sorry Cristian, I overlooked your hint to MDTM, which you previously gave :-) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c12
--- Comment #12 from Cristian Rodriguez
curl knows well how to do that:
Exactly that was my point, curl knows exactlt what to do , if you issue a HEAD request to an FTP server, it will give you an error if the file does not exists, or will show you the last modified time, it is handled transparently.
2) if you ever need to re-download for a file, do it the right way and m>ake it a conditional GET, so you need only one request as well -- with If-Modified-Since header
gotcha Peter, that 's the exact point. the server will return 304 "Not Modified" in case you already have your file up to date, once again, Im pretty sure curl knows how to do this the right way. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c13
--- Comment #13 from Cristian Rodriguez
https://bugzilla.novell.com/show_bug.cgi?id=307098#c14
--- Comment #14 from Cristian Rodriguez
https://bugzilla.novell.com/show_bug.cgi?id=307098
Stephan Kulow
https://bugzilla.novell.com/show_bug.cgi?id=307098#c15
Ján Kupec
https://bugzilla.novell.com/show_bug.cgi?id=307098#c16
--- Comment #16 from Stephan Kulow
https://bugzilla.novell.com/show_bug.cgi?id=307098#c18
Lukas Ocilka
https://bugzilla.novell.com/show_bug.cgi?id=307098#c19
--- Comment #19 from Michal Marek
curl_easy_setopt(_curl, CURLOPT_TIMECONDITION, CURL_TIMECOND_IFMODSINCE); curl_easy_setopt(_curl, CURLOPT_TIMEVALUE, timestamp);
where timestamp is either the timestamp stored in the last refresh or the timestamp of the last version stored in disk(if any) being the curl options that matches the correct friendly behaviuor suggested by both Peter and me.
Right, just one note in case it's not obvious: If you're going to use this feature, you need to defer opening of the local file to the first call of the CURLOPT_WRITEFUNCTION callback, otherwise you'd truncate it. curl_easy_getinfo(_curl, CURLINFO_RESPONSE_CODE, pointer_to_long) will then tell you the status code (200 if the file was downloaded, 304 if not). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c20
--- Comment #20 from Michal Marek
Right, just one note in case it's not obvious: If you're going to use this feature, you need to defer opening of the local file to the first call of the CURLOPT_WRITEFUNCTION callback, otherwise you'd truncate it.
Actually libzypp downloads to a tmp file, which is then moved to the target destination, so this is not an issue. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098
Lukas Ocilka
https://bugzilla.novell.com/show_bug.cgi?id=307098#c21
Stanislav Visnovsky
https://bugzilla.novell.com/show_bug.cgi?id=307098#c22
--- Comment #22 from Ján Kupec
(In reply to comment #19 from Michal Marek)
Right, just one note in case it's not obvious: If you're going to use this feature, you need to defer opening of the local file to the first call of the CURLOPT_WRITEFUNCTION callback, otherwise you'd truncate it.
Actually libzypp downloads to a tmp file, which is then moved to the target destination, so this is not an issue.
It seems it is at last - the tmp file is 0 lenght if not modified and thus the target is overwritten with an empty file on copy. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c23
Ján Kupec
https://bugzilla.novell.com/show_bug.cgi?id=307098#c24
--- Comment #24 from Ján Kupec
https://bugzilla.novell.com/show_bug.cgi?id=307098#c27
Michal Marek
(In reply to comment #20 from Michal Marek)
Actually libzypp downloads to a tmp file, which is then moved to the target destination, so this is not an issue.
It seems it is at last - the tmp file is 0 lenght if not modified and thus the target is overwritten with an empty file on copy.
Yeah, but at that time (after the transfer) you already have the response code, so you of course move only if it's not 304 (as you do in the patch) (In reply to comment #23 from Jan Kupec)
Just one more thing: is the timestamp of the file the only thing curl checks when CURL_TIMECOND_IFMODSINCE is used? So if i DO modify the file and reset the timestamp to the original, will curl still think it has not been changed? If so, is there a way to check for such situation?
You pass the timestamp (not some filename) stored in a long. The curl library doesn't care where you got the timestamp from and it assumes it's valid. (In reply to comment #24 from Jan Kupec)
Created an attachment (id=175753) --> (https://bugzilla.novell.com/attachment.cgi?id=175753) [details] CURL_TIMECOND_IFMODSINCE for doProvideFileCopy()
+ if (modified || infoRet != CURLE_OK) + { + // apply umask + if ( ::fchmod( ::fileno(file), filesystem::applyUmaskTo( 0644 ) ) ) + { + ERR << "Failed to chmod file " << destNew << endl; + } + ::fclose( file ); + + // move the temp file into dest + if ( rename( destNew, dest ) != 0 ) { + ERR << "Rename failed" << endl; + ZYPP_THROW(MediaWriteException(dest)); + } } + DBG << "done: " << PathInfo(dest) << endl;
Shouldn't the fclose() call be unconditional? Otherwise you could run out of filedescriptors. Also, in the not-modified case you can remove the zero-length tmp file. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c28
--- Comment #28 from Ján Kupec
Shouldn't the fclose() call be unconditional? Otherwise you could run out of
Sure. Thanx. In the future, i will wrap the fopen and fclose into some object's c-tor and d-tor, so that i don't have to care.
filedescriptors. Also, in the not-modified case you can remove the zero-length tmp file.
Yes, done. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c29
Ján Kupec
https://bugzilla.novell.com/show_bug.cgi?id=307098#c30
--- Comment #30 from Michael Andres
Sure. Thanx. In the future, i will wrap the fopen and fclose into some object's c-tor and d-tor, so that i don't have to care.
It's easier as you may think (see zypp/AutoDispose.h)
Replace
FILE *file = ::fopen( "/dev/null", "w" );
with
AutoDispose
https://bugzilla.novell.com/show_bug.cgi?id=307098#c31
Ján Kupec
https://bugzilla.novell.com/show_bug.cgi?id=307098#c32
--- Comment #32 from Cristian Rodriguez
Michael, thanx for the hint, i'll use it in the 11.0 code.
As for the patch, i tested it and it at least gives some 304 response codes when refreshing yum repos:
Just in case, remember that FTP will NOT return 304 error code, but success,d oes ftp works ok after this patch ?
It's not much of an improvement but at least some.
from the server point of view, it is an excellent step forward.
maybe rewriting the getDoesFileExist()
hrmm..in case that work will be done for 11, AFAICS: 1. file_exists should only be used in cases where the precence of the file is critical to take other decisions different to download or update. 2. It should not download content. 3. for dowload(or update) a file, conditional requests should be done always, this saves significant work in the server side and makes zypp a good netizen. This is of course a sugeesttion purely from the server side point of view, note that Im not expert on the zypp and your mileage may vary. and remember you can always ask Peter, darix or me in case of doubt. we are good enough on this topic ;) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c33
--- Comment #33 from Ján Kupec
Just in case, remember that FTP will NOT return 304 error code, but success,d oes ftp works ok after this patch ?
The question is whether the file gets truncated when CURL_TIMECOND_IFMODSINCE is set. I'll check.
and remember you can always ask Peter, darix or me in case of doubt. we are good enough on this topic ;)
thanx :O) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c35
Stephan Kulow
https://bugzilla.novell.com/show_bug.cgi?id=307098#c36
--- Comment #36 from Ján Kupec
Just in case, remember that FTP will NOT return 304 error code, but success,d oes ftp works ok after this patch ?
Just checked, works OK, but of course with the overhead of downloading the same files multiple times. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098#c37
Peter Poeml
https://bugzilla.novell.com/show_bug.cgi?id=307098#c38
--- Comment #38 from Ján Kupec
If, however, the fix applies to _larger_ files as well (I don't know if it does!), then it would be valuable.
It doesn't. Major change is needed to make use of the raw metadata cache in order to make a significant difference. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=307098
User dmacvicar@novell.com added comment
https://bugzilla.novell.com/show_bug.cgi?id=307098#c41
Duncan Mac-Vicar
participants (1)
-
bugzilla_noreply@novell.com