[zypp-devel] Read all patterns in 'one go'
Hi, rev 8309 of libzypp now supports to read all patterns in one go. Instead of downloading (and parsing) zillions of little .pat files, one can con'cat'enate all .pat files to a single 'patterns.pat' (resp. 'patterns.pat.gz') file. This fix will hopefully be obsoleted by the new .solv files anytime soon, but until then I hope it might be useful. Klaus --- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Thursday 17 January 2008 12:08:32 Klaus Kaempf wrote:
Hi,
rev 8309 of libzypp now supports to read all patterns in one go. Instead of downloading (and parsing) zillions of little .pat files, one can con'cat'enate all .pat files to a single 'patterns.pat' (resp. 'patterns.pat.gz') file.
This fix will hopefully be obsoleted by the new .solv files anytime soon, but until then I hope it might be useful.
thanks! Michael, we have to keep track to merge this changes, and ones Ladislav did, into the solv branch. Duncan -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
* Duncan Mac-Vicar Prett
Michael, we have to keep track to merge this changes, and ones Ladislav did, into the solv branch.
Btw, the susetags parser from sat-solver already supports reading multiple patterns within one file. Just like 'packages' contains multiple packages. Klaus --- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Thu, Jan 17, Klaus Kaempf wrote:
Hi,
rev 8309 of libzypp now supports to read all patterns in one go. Instead of downloading (and parsing) zillions of little .pat files, one can con'cat'enate all .pat files to a single 'patterns.pat' (resp. 'patterns.pat.gz') file.
This fix will hopefully be obsoleted by the new .solv files anytime soon, but until then I hope it might be useful.
Are you shure this patch is needed? We already download all .pat files listed in the content file, and we parse them. Why is a special case for a "patterns.pat" needed? -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Hi, On Thu, 17 Jan 2008, Michael Andres wrote:
Are you shure this patch is needed?
We already download all .pat files listed in the content file, and we parse them. Why is a special case for a "patterns.pat" needed?
Downloading many small files over http is much worse then downloading just one bigger one (and of course the bigger one compresses better). This can also be seen with downloading the patch information with all the many small (and big but uncompressed) patch.xml files. This all will be obsolete once we use onle .solv files, but hey. Ciao, Michael. -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Thursday 17 January 2008 15:59:18 Michael Matz wrote:
Downloading many small files over http is much worse then downloading just one bigger one (and of course the bigger one compresses better).
This can also be seen with downloading the patch information with all the many small (and big but uncompressed) patch.xml files.
This all will be obsolete once we use onle .solv files, but hey.
Michael's queston is related of if the patch does any difference. Current libzypp downloads allf iles listed in content file (patterns do) and parses all .pat files as patterns, and it should work for more than 1 pattern perl file. If it did not work is because autobuild did not do it that way. Note that this breaks architecture filtering before downloading, unless the patterns are still splitted by arch, and named correctly. Duncan -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Hi, On Thu, 17 Jan 2008, Duncan Mac-Vicar Prett wrote:
On Thursday 17 January 2008 15:59:18 Michael Matz wrote:
Downloading many small files over http is much worse then downloading just one bigger one (and of course the bigger one compresses better).
This can also be seen with downloading the patch information with all the many small (and big but uncompressed) patch.xml files.
This all will be obsolete once we use onle .solv files, but hey.
Michael's queston is related of if the patch does any difference.
Current libzypp downloads allf iles listed in content file (patterns do) and parses all .pat files as patterns, and it should work for more than 1 pattern perl file.
Ah, now I get it. The capability to parse multiple patches in one file was already there, so zypp might already do the right thing without the patch. Sorry for the confusion :) Ciao, Michael. -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Am Donnerstag 17 Januar 2008 schrieb Michael Matz:
Hi,
On Thu, 17 Jan 2008, Michael Andres wrote:
Are you shure this patch is needed?
We already download all .pat files listed in the content file, and we parse them. Why is a special case for a "patterns.pat" needed?
Downloading many small files over http is much worse then downloading just one bigger one (and of course the bigger one compresses better).
I changed the patterns now, so that factory only has one pat file: suse/setup/descr/dvd-11.0-14.i586.pat.gz It's 123K and from everyone says, it should simply work. Let's see :) Greetings, Stephan -- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Michael Matz escribió:
Downloading many small files over http is much worse then downloading just one bigger one (and of course the bigger one compresses better).
and is even worst in the current situation because zypp floods the servers with unneded requests or even (partial|complete) downloads instead of using CURL_TIMECOND_IFMODSINCE , when I analyzed this issue I was unpleasantly surprised with this misbehaviuor and I still dont know why such a basic thing like being friendly with the mirrors and use less network resources as possible was not considered in the current implementation ;( i.e last time I checked, a file exists check in zypp consisted in a partial,small file download.. and almost any protocol supported by curl have a way to do such operation without downloading any file.. -- "The only thing that interferes with my learning is my education." - Albert Einstein Cristian Rodríguez R. Platform/OpenSUSE - Core Services SUSE LINUX Products GmbH Research & Development http://www.opensuse.org/ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Michael Matz escribió:
Downloading many small files over http is much worse then downloading just one bigger one (and of course the bigger one compresses better).
and is even worst in the current situation because zypp floods the servers with unneded requests or even (partial|complete) downloads instead of using CURL_TIMECOND_IFMODSINCE , when I analyzed this issue I was unpleasantly surprised with this misbehaviuor and I still dont know why such a basic thing like being friendly with the mirrors and use less network resources as possible was not considered in the current implementation ;(
i.e last time I checked, a file exists check in zypp consisted in a partial,small file download.. and almost any protocol supported by curl have a way to do such operation without downloading any file.. I suggest you get your head together with Peter Poeml and write some layer on top of libcurl that redirector users can use and then we replace
Am Dienstag 22 Januar 2008 schrieb Cristian Rodríguez: libcurl usage with that code. I don't think libzypp is the only code that could use some nice to use wrapper that knows about the implications of our infrastructure. Greetings, Stephan -- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
* Michael Andres
Are you shure this patch is needed?
Hmm, maybe not. I haven't tried.
We already download all .pat files listed in the content file, and we parse them. Why is a special case for a "patterns.pat" needed?
It might be sufficient to just generate one big .pat file in autobuild. I just wonder why this wasn't done before. Klaus --- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
participants (6)
-
Cristian Rodríguez -
Duncan Mac-Vicar Prett -
Klaus Kaempf -
Michael Andres -
Michael Matz -
Stephan Kulow