https://bugzilla.novell.com/show_bug.cgi?id=381280 User dmacvicar@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=381280#c2 Duncan Mac-Vicar changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jkupec@novell.com Version|Beta 1 |Beta 3 --- Comment #2 from Duncan Mac-Vicar 2008-05-21 09:37:44 MST --- I made some progress here. By adding a repo the _first time_ we don't now the type yet, so on the frst refresh, only the probing is still doing a head request. Those are the 3 first requests, but 4..n now don't have head request, and the 404 s inmediately handled as don't exist. 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:02:14 +0200] "HEAD //repodata/repomd.xml HTTP/1.1" 200 0 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:02:14 +0200] "GET //repodata/repomd.xml HTTP/1.1" 200 1240 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:02:14 +0200] "HEAD //repodata/repomd.xml HTTP/1.1" 200 0 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:02:14 +0200] "GET //repodata/repomd.xml.asc HTTP/1.1" 200 189 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:02:14 +0200] "GET //repodata/repomd.xml.key HTTP/1.1" 200 2173 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:02:14 +0200] "GET //repodata/repomd.xml HTTP/1.1" 200 1240 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:02:14 +0200] "GET /repodata/primary.xml.gz HTTP/1.1" 200 84998 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:02:15 +0200] "GET /repodata/updateinfo.xml.gz HTTP/1.1" 200 4534 "-" "ZYpp 4.25.0 (curl 7.16.4)" The first and following times the repo changes completely (or we simulate a forced refresh), we need don't need to probe anymore, so you can see that repomd.xml is get once to see if the repo changed. Then the request are the ones expected (except for repomd.xml) 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:03:05 +0200] "GET //repodata/repomd.xml HTTP/1.1" 200 1240 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:03:05 +0200] "GET //repodata/repomd.xml.asc HTTP/1.1" 200 189 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:03:05 +0200] "GET //repodata/repomd.xml.key HTTP/1.1" 200 2173 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:03:05 +0200] "GET //repodata/repomd.xml HTTP/1.1" 200 1240 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:03:06 +0200] "GET /repodata/primary.xml.gz HTTP/1.1" 200 84998 "-" "ZYpp 4.25.0 (curl 7.16.4)" 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:03:06 +0200] "GET /repodata/updateinfo.xml.gz HTTP/1.1" 200 4534 "-" "ZYpp 4.25.0 (curl 7.16.4)" The next times the repo does not change. You only get one request. 2001:780:101:a00:20f:feff:fe5d:ca52 piscola.suse.de - [21/May/2008:17:04:51 +0200] "GET //repodata/repomd.xml HTTP/1.1" 200 1240 "-" "ZYpp 4.25.0 (curl 7.16.4)" So 1) I consider the head request on probing unimportant, as only happens once in the lifetime of the repository, and only when adding urls of unknown type. 2) I think the duplicate retrieval of the repomd.xml can be optimized, but I am not sure if is possible, will look further. I will commit the current progress for now. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=381280 User dmacvicar@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=381280#c4 Duncan Mac-Vicar changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |jkupec@novell.com --- Comment #4 from Duncan Mac-Vicar 2008-05-26 07:00:34 MDT --- Ok, after breaking libzypp with attempt one, I have a completely different proposal/patch. I am looking the sponsorship from Jano again :-) The original attempt broke because checkonly flag is completely undefined, it is used somewhere, and somewhere else is not. Therefore 404 popup to the user interface before we can do it anything about it. There is no (sane) way to do a 404 without getting a callback that is handled directly by YaST. However, MediaCurl has the ability to look if the target file exists, and set the correct If-Modified-Since header. Therefore what I did was, expose the localRoot() option from MediaManager into MediaSetAccess. localRoot(medianr) is empty if the media is not attached, so I made MediaSetAccess::localRoot(medianr) to attach that media if it wasn't before. Then in RepoManager, I assumed that, the first time you download repomd.xml on a full cache refresh, even if it is a forced one, may be the same as the one in the old cache. So for both checkIfToRefreshMetadata I hardlinked the old cache repomd.xml into the MediaManager::localRoot(1), if it is not the same, Mediamanager will download it and then move it over the hardlink, replacing it (but not the original in the old cache), the old one will not be replaced until the compelte temp directory is moved over the old. If it succeeds, then there is no file downloaded. The .key and .asc file can also be hardlinked and injected into the media root. This provides two optimizations. * If the repomd.xml is not changed, nothing is downloaded. A kind of Fetcher cached without knowing the checksums of the files, this includes repomd.xml and the keys/signatures. If repomd.xml is not changed, 0 bytes are downloaded while checking for changes. If it is changed, then it is downloaded. Additionally if checkIfToRefreshMetadata is refactored / splitted in 2 parts. One that takes an additional MediaSetAccess, and the public version exposed in the API, which in turn could instanciate the MediaSetAccess and call the other one, could allow refreshMetadata to share the MediaSetAccess object with checkIfToRefreshMetadata, thus also avoiding the re-transfer of repomd.xml between checking and actually refreshing. So if repomd.xml changed, it will be downloaded when checking if it is changed, but when the refresh job starts, the repomd.xml will be set the right if-modified-since and will not be transfered again. Both changes only require us to implement the localRoot() exposure before GM, which is only adding API. The actual changes can be delivered in an online update as they don't change anything else than the implementation. I have a patch which implements the first part, it works very well, and the changes are very simple and uninstrusive. Jano, what do you think? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=381280 User coolo@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=381280#c6 --- Comment #6 from Stephan Kulow 2008-05-31 13:47:48 MDT --- Duncan, how confident are you? I set it to !ship_stopper because I think we should rather be sure the fix works, but if you guys are confident, we still have some time to test the change and according to Peter it's very important to have this in. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=381280 Duncan Mac-Vicar changed: What |Removed |Added ---------------------------------------------------------------------------- Component|libzypp |libzypp Product|openSUSE 11.0 |openSUSE 11.2 Version|Beta 3 |unspecified -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.