Marcus Meissner schrieb:

You are getting progress bars, right?

Its likely just your slow network connection.

But it's a deadlock problem and a must-fix for the next release. There is _no_way_ to get the configuration fixed if it's broken (e.g. by a bad and slow mirror, bad connection or the like). - If the connection to a mirror is slow, there is no way to disable it because YaST doesn't give the user access to it before the metadata are downloaded, which can take up to 24 hours if an unlucky user managed to get a _really_ slow mirror like ftp.opensuse.org. - If the user tries to trick YaST into giving him access by going offline, it doesn't work either because then YaST thinks it's a good idea to hide the remote sources. The only way I'm aware of is a brute-force attack, going into /var/lib/zypp/db/sources and deleting the XML file directly. It's broken and can't stay this way. Andreas --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On 2006/11/30 16:13 (GMT+0100) Marcus Meissner apparently typed:

On Thu, Nov 30, 2006 at 10:12:02AM -0500, Felix Miata wrote:

...

I just booted in order to update to the latest factory ftp tree. I opened YaST to check and see which installation source was set, because I already found out that my usual mirror is out of sync.

...

Couldn't do it. zypp-updater was running and preventing access to the DB. So, I killed zypp-updater and tried again. Now Yast is stuck downloading God only know what files, when all I wanted to know was what installation source was set.

...

Simply ludicrous software management system.

You are getting progress bars, right?

Sure, little narrow ones that show less than half the URL that I have to keep widening by more than 100% each time they close and reopen. The bigger question is why does anything need to be downloaded just so that I can check to see what my installation source is set to? Also, why didn't the updater quit when I brought it up from the taskbar and selected to quit? Why does it have to lock the database so tight I can't even look at it?

Its likely just your slow network connection.

My network connection is about as fast as a network connection can be. The problem is mirrors in sync are in such short supply that it's pointless to try anywhere but ftp.gwdg.de for factory ftp, which is hopelessly slow nearly always. Now while I was waiting for a response the downloading has halted with this message in a window: ERROR: There were errors when restoring the source configuration. Not all sources are available for configuration. http://ftp.gwdg.de/pub/linux/suse/opensuse/distribution/SL-OSS-factory/inst-...: Cannot create the installation source. Do you want to immediately remove these sources? Yes No :-( -- "Rejoice and be glad, because great is your reward in heaven." Matthew 5:12 NIV Team OS/2 ** Reg. Linux User #211409 Felix Miata *** http://mrmazda.no-ip.com/ --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Marcus Meissner schrieb:

Is there a bugreport for this?

It's a combination of multiple things. (1) Installation sources in offline mode https://bugzilla.novell.com/show_bug.cgi?id=223600 (2) Metadata shouldn't be refreshed when starting yast2 inst_source Currently not reported (AFAIK). (3) download.opensuse.org often redirects to ftp.opensuse.org Currently not reported (AFAIK). --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Hi, A suggestion for openSUSE10.3. I've noticed that is refreshed by Yast YOU if it has changed on the server. It takes me about 4 minutes to download from and is currently 2.9MB in size. I had a look at this compressed text file and observed that it contains a very large proportion of redundant data. /a/b/c/d/e/file1 /a/b/c/d/e/file2 etc If this file could be compressed further by omitting the redundant data the compressed version would download much faster and may improve the end users experience of updating their system. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Hi, On Thu, 30 Nov 2006, Felix Miata wrote:

On 2006/11/30 16:39 (GMT+0100) Eberhard Moenkeberg apparently typed:

...

Use http://ftp-1.gwdg.de/pub/opensuse/distribution/SL-OSS-factory/inst-source

...

ftp-1 is my best server these days for all suse and opensuse directories, both regarding performance and actuality.

I switched to this for this update. I started over 5 hours ago. It still shows >2 hours remaining. I can't tell exactly how dismal this is because yast doesn't bother to tell the download rate, but I suspect it's in the modem speed range. From good performing mirrors I can download an average CD iso in less than 20 minutes. I should have been done >4 hours ago.

Believe me, this is the best you can do. I have a bandwidth shaping of 800 MBit/sec since 8 minutes (500 MBit/sec before), and about 2200 sessions currently. The server is delivering 73 MByte/sec at the moment - so please calculate if "your part" is fair. BTW: the remaining 27 MByte/sec get delivered by ftp.gwdg.de. ;-)) I can only guarantee the server stability, not the individual throughput. But the summary. Yes, I can. ;->> Cheers -e -- Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

succes of RC1 is incredible. My bittorent azureus has already 13 clients and I served 8 times my download of the 5 cd last week, for beta, 4 days after the release I had no more clients... jdd -- http://www.dodin.net http://dodin.org/mediawiki/index.php/GPS_Lowrance_GO --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On 2006/11/30 23:16 (GMT+0100) Eberhard Moenkeberg apparently typed:

On Thu, 30 Nov 2006, Felix Miata wrote:

...

On 2006/11/30 16:39 (GMT+0100) Eberhard Moenkeberg apparently typed:

...

...

Use http://ftp-1.gwdg.de/pub/opensuse/distribution/SL-OSS-factory/inst-source

...

...

ftp-1 is my best server these days for all suse and opensuse directories, both regarding performance and actuality.

...

I switched to this for this update. I started over 5 hours ago. It still shows >2 hours remaining. I can't tell exactly how dismal this is because yast doesn't bother to tell the download rate, but I suspect it's in the modem speed range. From good performing mirrors I can download an average CD iso in less than 20 minutes. I should have been done >4 hours ago.

Believe me, this is the best you can do. I have a bandwidth shaping of 800 MBit/sec since 8 minutes (500 MBit/sec before), and about 2200 sessions currently. The server is delivering 73 MByte/sec at the moment - so please calculate if "your part" is fair.

I'm not sure how to calculate. ETA is now down to about 20 minutes, which means the total time for almost 800 packages will have amounted to almost 7 hours by the time it finishes. By comparison, 800 is probably not a lot less than how many packages were required yesterday on a fresh install from mirrors.kernel.org that took less than 2 hours for the whole installation.

BTW: the remaining 27 MByte/sec get delivered by ftp.gwdg.de. ;-))

I have no doubt you do the best you can given the demand and what you have to work with. I think the real problem is professed development mirrors whose sync frequency is just fine for releases, but inadequate for development trees. The closest mirror to me is ftp.cise.ufl.edu. It's performance is great, if it has what I need, but it doesn't carry factory. Mirrors.kernel.org usually has good download performance, but it's sync behavior for factory is bad. I wanted the newest kernel this AM and went there to find it. I had no problem fetching it, but it had a dep on some newer perl package. I went to find that package, and there was _no_ version of it on that mirror. That's when I decided to try ftp-1.gwdg.de. We need testing caliber mirrors only for factory, but don't have it. This is what drives people to overload gwdg. :-( -- "Rejoice and be glad, because great is your reward in heaven." Matthew 5:12 NIV Team OS/2 ** Reg. Linux User #211409 Felix Miata *** http://mrmazda.no-ip.com/ --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Hi, On Thu, 30 Nov 2006, Felix Miata wrote:

On 2006/11/30 23:16 (GMT+0100) Eberhard Moenkeberg apparently typed:

...

On Thu, 30 Nov 2006, Felix Miata wrote:

...

On 2006/11/30 16:39 (GMT+0100) Eberhard Moenkeberg apparently typed:

...

...

...

Use http://ftp-1.gwdg.de/pub/opensuse/distribution/SL-OSS-factory/inst-source

...

...

...

ftp-1 is my best server these days for all suse and opensuse directories, both regarding performance and actuality.

...

...

I switched to this for this update. I started over 5 hours ago. It still shows >2 hours remaining. I can't tell exactly how dismal this is because yast doesn't bother to tell the download rate, but I suspect it's in the modem speed range. From good performing mirrors I can download an average CD iso in less than 20 minutes. I should have been done >4 hours ago.

...

Believe me, this is the best you can do. I have a bandwidth shaping of 800 MBit/sec since 8 minutes (500 MBit/sec before), and about 2200 sessions currently. The server is delivering 73 MByte/sec at the moment - so please calculate if "your part" is fair.

I'm not sure how to calculate. ETA is now down to about 20 minutes, which means the total time for almost 800 packages will have amounted to almost 7 hours by the time it finishes.

By comparison, 800 is probably not a lot less than how many packages were required yesterday on a fresh install from mirrors.kernel.org that took less than 2 hours for the whole installation.

Keep reserved about YasT's time estimations. It is good they are present, but they are most of the time of low value. At the end they are good. ;-)

...

BTW: the remaining 27 MByte/sec get delivered by ftp.gwdg.de. ;-))

I have no doubt you do the best you can given the demand and what you have to work with. I think the real problem is professed development mirrors whose sync frequency is just fine for releases, but inadequate for development trees. The closest mirror to me is ftp.cise.ufl.edu. It's performance is great, if it has what I need, but it doesn't carry factory.

It is planned since long to distribute factory via push service (like repositories currently) - this will give the chance for every mirror to be as actual as possible. The opensuse server would initiate the rsync run as soon as any package has finished compiling successfully. Let's pray to get it coming soon, or let's shout very very nasty to the SUSE guys who are responsible for it. Looking at the time frame, I would prefer to shout.

Mirrors.kernel.org usually has good download performance, but it's sync behavior for factory is bad. I wanted the newest kernel this AM and went there to find it. I had no problem fetching it, but it had a dep on some newer perl package. I went to find that package, and there was _no_ version of it on that mirror. That's when I decided to try ftp-1.gwdg.de.

We need testing caliber mirrors only for factory, but don't have it. This is what drives people to overload gwdg. :-(

If you have patience, you will stand it and it will succeed. It will perform better at European night and weekend. In summary, the GWDG servers are delivering almost 200 TB per month. This rate will get doubled (theoretically - I will have to see if the servers will perform accordingly) in October 2007. Cheers-e -- Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Keith Goggin schrieb:

I've noticed that is refreshed by Yast YOU if it has changed on the server. It takes me about 4 minutes to download from and is currently 2.9MB in size.

It's really insane. This XML stuff has made SUSE distros basically unusable without a broadband connection. 3MB before the distro is even released - crazy!

I had a look at this compressed text file and observed that it contains a very large proportion of redundant data.

/a/b/c/d/e/file1 /a/b/c/d/e/file2 etc

If this file could be compressed further by omitting the redundant data the compressed version would download much faster and may improve the end users experience of updating their system.

Yes, there are much more efficient ways to store these data than repomd does. But hey, XML is so much cooler. Maybe one of those talented persons who designed repomd should have a look at /var/lib/locatedb. It's able to store equivalent information about 2 Linuxes, 1 Windows system and a lot of user files in just 2.5MB. Thanks to not using XML. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On Fri, 01 Dec 2006 02:54:41 +0100, Andreas Hanke wrote:

It's really insane. This XML stuff has made SUSE distros basically unusable without a broadband connection. 3MB before the distro is even released - crazy!

My diy-linux w/uclibc uses 20MB of disk, including perl. My SUSE 10.2 "minimal" install used 900MB of disk. That's some load of #@!$%! --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Hanke wrote:

Keith Goggin schrieb:

...

I've noticed that is refreshed by Yast YOU if it has changed on the server. It takes me about 4 minutes to download from and is currently 2.9MB in size.

It's really insane. This XML stuff has made SUSE distros basically unusable without a broadband connection. 3MB before the distro is even released - crazy!

Let's have a look at some numbers ;) On my repository for 10.1: repo type | size (bytes) | MB | files - ----------+--------------+-------+--------------------------- yast2 | 1831446 | 1.74 | packages, packages.en rpm-md | 1454322 | 1.37 | primary.xml.gz, filelists.xml.gz yast2 repositories make up a somewhat larger download (21%), actually. When you look at Factory: repo type | size (bytes) | MB | files - ----------+--------------+-------+--------------------------- yast2 | 17876184 | 17.04 | packages, packages.en rpm-md | 20263138 | 19.32 | primary.xml.gz, filelists.xml.gz Here rpm-md is a little larger (12%).

...

I had a look at this compressed text file and observed that it contains a very large proportion of redundant data. /a/b/c/d/e/file1 /a/b/c/d/e/file2 etc

It's not redundant data. Those are file paths. Part of it are redundant, sure (e.g. /usr/share, /usr/lib, ...). As in yast2 repository files (packages, packages.en) -- the yast2 repository files have a lot more redundant data as they're not even compressed.

...

If this file could be compressed further by omitting the redundant data the compressed version would download much faster and may improve the end users experience of updating their system.

That's exactly what gzip is already doing on those rpm-md XML files.

Yes, there are much more efficient ways to store these data than repomd does. But hey, XML is so much cooler.

XML has its advantages. It's very easy to parse (but not necessarily easy to parse with good performance, SAX/DOM/StAX/...). Using DOM for such files uses insane amounts of memory. Parsing yast2 metadata isn't that complex but you definitely have to write your own error-prone parser instead of just using StAX. It took quite a few attempts and fixes to write the yast2 parser in smart properly, as the various "tags" are not documented (and suddenly new tags appeared, etc...).

Maybe one of those talented persons who designed repomd should have a look at /var/lib/locatedb. It's able to store equivalent information about 2 Linuxes, 1 Windows system and a lot of user files in just 2.5MB. Thanks to not using XML.

One might argue that xml.gz _is_ a binary format ;) gzip should already optimize away the XML tags and attributes. filelists.xml.gz contains all files that are in Factory in 19.32 MB and files of RPMs for 1023 different projects in 1.37 MB in my repository. Note that using bzip2 would be more effective (with filelists.xml.gz from Factory): Compression | File size | diff | ratio - --------------------+-----------+------+------- uncompressed | 155920703 | +92% | 0.00 original (gzip) | 12908161 | 0 | 12.07 gzip -9 | 12871908 | -1% | 12.11 gzip -9 --rsyncable | 13158574 | +2% | 11.84 bzip2 | 10848980 | -16% | 14.37 lrzip -M | 4879263 | -63% | 31.95 (diff is compressed compared to original .gz) (ratio is compressed compared to uncompressed) lrzip performs best (to say the least) but takes a long time, especially compared to gzip, and a lot of RAM (-M uses all available RAM, performed on a 2GB box), even on my dual-core AMD64. The ratio is pretty impressive though (see http://ck.kolivas.org/apps/lrzip/README). bzip2 is also significantly slower than gzip and uses somewhat more CPU but still gives a 16% gain. Now with the yast2 "packages" file: Compression | File size | diff | ratio - --------------------+-----------+------+------- original (no compr.)| 15547367 | 0 | 0.00 gzip -9 | 2340627 | -85% | 6.64 bzip2 -9 | 1797400 | -89% | 8.64 lrzip -M | 1393749 | -92% | 11.15 Note: you cannot directly compare the yast2 and rpm-md files (except for overall size) as they contain somewhat different subsets of data of a repository. Obviously there's still a lot to gain from compressing yast2 repository files -- even with a quick gzip -9 the file is 85% smaller. Who files the bug for requesting that feature ? ;) But to summarize, the compression already optimizes away the redundant data, especially the XML tags and attributes. It comes pretty much at no cost as of the download size. Of course, when it is parsed, it first has to be uncompressed (or streamed). The parsing performance highly depends on the method (SAX, DOM, StAX, ...). An interesting approach is to store that parsed data into a relational database, which is what ZMD is doing (sqlite DB). A database can use indexes to speed up queries a lot, as compared to having to load the whole data into memory and processing the query yourself. But not don't ask me why ZMD is a lot slower than zypp or smart... maybe the cost of inserting all that data into the sqlite database ? Or the synchronization with zypp/yast ? cheers - -- -o) Pascal Bleser http://linux01.gwdg.de/~pbleser/ /\\ _\_v The more things change, the more they stay insane. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFFb92/r3NMWliFcXcRAlAIAJ9Zd5EVozn2gLWZbcxC+yRflvEnXQCfSxbn fS1AZmOlw0VeDTUfSzD1FGI= =LNhD -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On Friday 01 December 2006 09:46, Pascal Bleser wrote:

Let's have a look at some numbers ;)

[...] From your numbers it is pretty clear that there are some straightforward optimization ways: - use bzip2 for repodata: downloading 2 MB of extra data is still slower than uncompressing a file with bzip2 compared to gzip - use compression for yast repositories - suggest using yast repositories for SUSE packages I know this is not that easy to do for repo-md, if other distributions should agree with it as well, but for yast repositories it is something that should be considered right from the start for 10.3. About XML parsing performance, here are some numbers (AMD64, 3200+): - parsing a 17MB (!) HTML file in Quanta Plus is 19seconds. Own parser, but I'm sure it is more complex and does much more things than an XML parser needed for YaST. - reading 2MB of XML file from several hunderds of files using Qt's DOM functions takes 328ms. So let's say 16MB would take 2 seconds. I did not messure this in YaST, but if it's slower, there is a big problem as Qt's DOM implementation is not considered to be fast. ;-) Andras -- Quanta Plus developer - http://quanta.kdewebdev.org K Desktop Environment - http://www.kde.org

Op vrijdag 1 december 2006 08:56, schreef Marcus Meissner:

...

Maybe one of those talented persons who designed repomd should have a look at /var/lib/locatedb. It's able to store equivalent information about 2 Linuxes, 1 Windows system and a lot of user files in just 2.5MB. Thanks to not using XML.

Yeah, I saw this too and thought about at least using some kind of directory prefix or similar handling.

XML really makes it easy to be stupid in data design :/

What about md5sum-ming the filelist, and use the md5sum as key? If a new version of a package is released with the same filelist only the md5sum needs to be transferred. For big packages the compression might be around 100% ;) -- Richard Bos Without a home the journey is endless --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Pascal Bleser schrieb:

...

What about md5sum-ming the filelist, and use the md5sum as key? If a new version of a package is released with the same filelist only the md5sum needs to be transferred. For big packages the compression might be around 100% ;)

Already done.

No, that's not what was asked for. Just as an example for repomd: You have a repository with package A-1 and B-1. createrepo writes the filelists of both packages into filelists.xml. Now A gets upgraded to A-1.1. Which is a great thing, because the package manager has to download the whole filelists.xml again, even though package B has not been touched at all. What you describe covers only the fact that the metadata aren't downloaded again if nothing changed. But if even the slightest thing changed, everything is downloaded from scratch, even the parts that have not been changed. Solutions: - Use a smarter protocol that can "fix" this design problem, e.g. rsync instead of http/ftp. - Think about a smarter metadata format. SUSE had one (the old plain-text patchinfos) and threw it away in favour of repomd. - Extend the repomd format to suck less, e.g. by splitting filelists.xml into filelists-dec06.xml, filelists-jan07.xml, filelists-feb07.xml so that at least the unchanged filelists from previous months aren't downloaded over and over again. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Op zaterdag 2 december 2006 01:11, schreef Andreas Hanke:

No, that's not what was asked for.

Just as an example for repomd:

You have a repository with package A-1 and B-1. createrepo writes the filelists of both packages into filelists.xml.

Now A gets upgraded to A-1.1. Which is a great thing, because the package manager has to download the whole filelists.xml again, even though package B has not been touched at all.

What you describe covers only the fact that the metadata aren't downloaded again if nothing changed. But if even the slightest thing changed, everything is downloaded from scratch, even the parts that have not been changed.

Solutions:

- Use a smarter protocol that can "fix" this design problem, e.g. rsync instead of http/ftp.

- Think about a smarter metadata format. SUSE had one (the old plain-text patchinfos) and threw it away in favour of repomd.

- Extend the repomd format to suck less, e.g. by splitting filelists.xml into filelists-dec06.xml, filelists-jan07.xml, filelists-feb07.xml so that at least the unchanged filelists from previous months aren't downloaded over and over again.

That's why I proposed to md5summing the filelist. A package with multi versions has often the same files. Especially in the released distribution as only security or important bug fixes are applied to packages. That means that the filelist of packages A-1.1 and A-1.2 is the same. It also means that the md5sum of the filelist of those 2 packages are the same. This could even be valid for multiple version of distribution. So when the client has a database with the md5sum of the filelist as key and as "payload" the filelist, the client can easily determine the filelist belonging to a package. How this should work in details, I have not worked out, but it might save a lot of bandwith. -- Richard Bos Without a home the journey is endless --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On Friday 01 December 2006 01:27, Lukas Ocilka wrote:

John Kelly wrote:

...

On Fri, 01 Dec 2006 02:54:41 +0100, Andreas Hanke

wrote:

...

It's really insane. This XML stuff has made SUSE distros basically unusable without a broadband connection. 3MB before the distro is even released - crazy!

My diy-linux w/uclibc uses 20MB of disk, including perl. My SUSE 10.2 "minimal" install used 900MB of disk. That's some load of #@!$%!

It seems you might be interested in the MiniSuSE project: http://en.opensuse.org/MiniSUSE

Lukas

That's for sure, and even more for MicroSUSE http://en.opensuse.org/MicroSUSE Although I would like to see John making MiniSUSE project to take off, his micro linux seems to be closer to MicroSUSE that is in the moment probably in greater need for people that can adapt present source to use uclibc. -- Regards, Rajko M. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org