Feature changed by: Michal Marek (michal-m) Feature #309167, revision 4 Title: lzma compression for primary.xml Buildservice: Evaluation Priority Requester: Mandatory openSUSE-11.3: Evaluation Priority Requester: Mandatory Requested by: Michal Marek (michal-m) Description: Providing all exported kernel symbols as individual dependencies (fate 305945) resulted in a DoS against download.o.o, because the primary.xml. gz file in the update repository grows with each kernel update (bnc 582907 and 586960). The least intrusive fix to this seems to be: Sort the entries in primary.xml and provide a lzma compressed version in addition to primary.xml.gz. Testing shows that this brings the metadata growth back to the 11.1 symset approach (about 100k for each subsequent kernel update) and also reduces the size, so it would be beneficial also for normal repositories. The repomd.xml format extension could look like: <data type="primary"> <location href="repodata/primary.xml.gz"/> <checksum type="sha">1233456...</checksum> <timestamp>1268305225</timestamp> <open-checksum type="sha">abcdef...</open-checksum> </data> <data type="primary-lzma"> <location href="repodata/primary.xml.lzma"/> <checksum type="sha">56789a...</checksum> <timestamp>1268305245</timestamp> <open-checksum type="sha">abcdef...</open-checksum> </data> With the meaning that if there is a primary.xml.lzma with the same open- checksum, download this one instead of primary.xml.gz, decompress it with lzma (in default install since 11.1) and then continue as usual. Old clients would continue working with the primary.xml.gz file. The feature primarily targets libzypp, but also extending yum should not be hard. Relations: - Get rid of "symsets" in kernel package dependencies (feature/id: 305945) - zsync support for update channel refresh (feature/id: 309088) - fate #305945 causes extreme update channel metadata overhead (novell/bugzilla/id: 586960) https://bugzilla.novell.com/show_bug.cgi?id=586960 + Discussion: + #2: Michal Marek (michal-m) (2010-05-05 11:05:36) + patches for createrepo: + http://gitorious.org/mmarek-misc/createrepo/commits/lzma-support libzypp + patches: http://gitorious.org/~mmarek/opensuse/libzypp-lzma/commits/lzma-support + sat-solver patches: http://gitorious.org/~mmarek/opensuse/sat-solver-lzma/commits/lzma-support + createrepo is submitted to system:packagemanager as sr 39243. Libzypp and sat- + solver patches were posted to the zypp-devel mailing list: + http://lists.opensuse.org/zypp-devel/2010-05/msg00005.html (http://lists.opensuse.org/zypp-devel/2010-05/msg00005.html) + . -- openSUSE Feature: https://features.opensuse.org/309167

Feature changed by: Michal Marek (michal-m) Feature #309167, revision 6 Title: lzma compression for primary.xml - Buildservice: Evaluation + Buildservice: Rejected by (michal-m) + reject date: 2010-10-08 14:28:45 + reject reason: Obsoleted by fate 309561. Priority Requester: Mandatory - openSUSE-11.3: Evaluation + openSUSE-11.3: Rejected by (michal-m) + reject date: 2010-10-08 14:29:03 + reject reason: Obsoleted by fate 309561 Priority Requester: Mandatory Requested by: Michal Marek (michal-m) Description: Providing all exported kernel symbols as individual dependencies (fate 305945) resulted in a DoS against download.o.o, because the primary.xml. gz file in the update repository grows with each kernel update (bnc 582907 and 586960). The least intrusive fix to this seems to be: Sort the entries in primary.xml and provide a lzma compressed version in addition to primary.xml.gz. Testing shows that this brings the metadata growth back to the 11.1 symset approach (about 100k for each subsequent kernel update) and also reduces the size, so it would be beneficial also for normal repositories. The repomd.xml format extension could look like: <data type="primary"> <location href="repodata/primary.xml.gz"/> <checksum type="sha">1233456...</checksum> <timestamp>1268305225</timestamp> <open-checksum type="sha">abcdef...</open-checksum> </data> <data type="primary-lzma"> <location href="repodata/primary.xml.lzma"/> <checksum type="sha">56789a...</checksum> <timestamp>1268305245</timestamp> <open-checksum type="sha">abcdef...</open-checksum> </data> With the meaning that if there is a primary.xml.lzma with the same open- checksum, download this one instead of primary.xml.gz, decompress it with lzma (in default install since 11.1) and then continue as usual. Old clients would continue working with the primary.xml.gz file. The feature primarily targets libzypp, but also extending yum should not be hard. Relations: - Get rid of "symsets" in kernel package dependencies (feature/id: 305945) - zsync support for update channel refresh (feature/id: 309088) - fate #305945 causes extreme update channel metadata overhead (novell/bugzilla/id: 586960) https://bugzilla.novell.com/show_bug.cgi?id=586960 Discussion: #2: Michal Marek (michal-m) (2010-05-05 11:05:36) patches for createrepo: http://gitorious.org/mmarek-misc/createrepo/commits/lzma-support libzypp patches: http://gitorious.org/~mmarek/opensuse/libzypp-lzma/commits/lzma-support sat-solver patches: http://gitorious.org/~mmarek/opensuse/sat-solver-lzma/commits/lzma-support createrepo is submitted to system:packagemanager as sr 39243. Libzypp and sat- solver patches were posted to the zypp-devel mailing list: http://lists.opensuse.org/zypp-devel/2010-05/msg00005.html (http://lists.opensuse.org/zypp-devel/2010-05/msg00005.html) . #3: Jan Engelhardt (jengelh) (2010-05-06 22:57:05) Should use xz these days. -- openSUSE Feature: https://features.opensuse.org/309167