[zypp-devel] DiskUsage calculation status (libzypp-3.13.12)
DiskUsage status (libzypp-3.13.12) - Provided ResObject interface: + /** + * \short Disk usage per directory + * A common attribute, although mostly packages require + * noticeable disk space. An e.g product could try to reserve + * a certain ammount of diskspace by providing DiskUsage data. + */ + const DiskUsage & diskusage() const; - Prepared classes zypp::repo::cached::PackageImpl zypp::repo::cached::SrcPackageImpl Both overload diskusage(), using a lazy initialized DiskUsage object. By now the object is empty. @Duncan: Please provide a ResolvableQuery to retrieve the du-data from the DB and store them in _diskusage (see #warning FILL DU DATA FROM DB). - Susetags DiskUsage parser is done, and feeds the data to the CacheStore via CacheStore::consumeDiskUsage. Provided CacheStore::consumeDiskUsage stores the data correctly and Duncan prrovides the ResolvableQuery, DiskUsage calculation should work again. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Sat, Aug 04, Michael Andres wrote:
- Susetags DiskUsage parser is done, and feeds the data to the CacheStore via CacheStore::consumeDiskUsage.
Parsing ALL metadata file for 10.3 takes about 4.32 sec. But when writing them to the database, CacheStore::consumeDiskUsage is too slow. The whole packages files takes about 40 sec., but I aborted the packages.DU after 10 minutes. Duncan, please check consumeDiskUsage. Some du-enties take more than a minute. I disabled the DU parsing in zypp/parser/susetags/RepoParser.cc, line 429. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Saturday 04 August 2007 01:34:00 Michael Andres wrote:
On Sat, Aug 04, Michael Andres wrote:
- Susetags DiskUsage parser is done, and feeds the data to the CacheStore via CacheStore::consumeDiskUsage.
Parsing ALL metadata file for 10.3 takes about 4.32 sec.
But when writing them to the database, CacheStore::consumeDiskUsage is too slow.
The whole packages files takes about 40 sec., but I aborted the packages.DU after 10 minutes.
Duncan, please check consumeDiskUsage. Some du-enties take more than a minute.
I disabled the DU parsing in zypp/parser/susetags/RepoParser.cc, line 429.
I just implemented some "smartness" for the parser. First, it assumes YaST would set the system partitions, if they are empty, it tries to detect them: -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Saturday 04 August 2007 01:34:00 Michael Andres wrote:
On Sat, Aug 04, Michael Andres wrote:
- Susetags DiskUsage parser is done, and feeds the data to the CacheStore via CacheStore::consumeDiskUsage.
Parsing ALL metadata file for 10.3 takes about 4.32 sec.
But when writing them to the database, CacheStore::consumeDiskUsage is too slow.
The whole packages files takes about 40 sec., but I aborted the packages.DU after 10 minutes.
Duncan, please check consumeDiskUsage. Some du-enties take more than a minute.
I disabled the DU parsing in zypp/parser/susetags/RepoParser.cc, line 429.
I just implemented some "smartness" for the parser. First, it assumes YaST would set the system partitions, if they are empty, it tries to detect them: _mounts = getZYpp()->getPartitions(); if (_mounts.empty() ) _mounts = DiskUsageCounter::detectMountPoints(); Then it parses the disk usage file and it only takes in counts entries that are not below (or are the same) than the user partitions. Appart of that, a quick already-seen in-memory cache for directory entries while inserting them, I can get stable cache to build in 1 minute and 20 seconds which is only a litte more than ussual. Now the question is. During installation, YaST sets the partitions to ZYpp after repos are cached. - would DiskUsageCounter::detectMountPoints(); work before that? - can we set them before? or are repositories set before partitioning is done? - any workaround here? I am committing this code, and see how can we find a workaround for the second part. I have to implement the retrieval of the disk usage too. Duncan -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Duncan Mac-Vicar Prett napsal(a): [...]
Now the question is. During installation, YaST sets the partitions to ZYpp after repos are cached.
- would DiskUsageCounter::detectMountPoints(); work before that?
It would detect the current partitioning in inst-sys which is not used for installation (the target is not known and the root is in ramdisk).
- can we set them before? or are repositories set before partitioning is done?
The repositories are registered before the installation proposal is created (this includes also partitioning).
- any workaround here?
The software proposal is called after partitioning proposal so if any change (add/remove a partition) in the partitioner would trigger DU rebuild it should work. At the begining it can be set to the current partitioning (inst-sys), it will be changed to the proposed partitioning before requesting any DU data anyway... BTW How much time does it take to parse DU file from e.g. Alpha7 DVD? Ladislav -- Best Regards Ladislav Slezák Yast Developer ------------------------------------------------------------------------ SUSE LINUX, s.r.o. e-mail: lslezak@suse.cz Lihovarská 1060/12 tel: +420 284 028 960 190 00 Prague 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz/ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Tuesday 07 August 2007 15:41:04 Ladislav Slezak wrote:
The software proposal is called after partitioning proposal so if any change (add/remove a partition) in the partitioner would trigger DU rebuild it should work. At the begining it can be set to the current partitioning (inst-sys), it will be changed to the proposed partitioning before requesting any DU data anyway...
If that works. The parser only takes care that getZYpp()->getPartitions() is not empty, otherwise it will fallback to try to detect them. Note: reparsing DU data right now means reparsing the repository. We don't support per-file rebuild of cache (now, but it wll be easier to add than before).
BTW How much time does it take to parse DU file from e.g. Alpha7 DVD?
Depends on how much partitions you have. If you have the 100.000 partitions for which our packages.DU file was (over)engineered, it takes infinitum. For a number of parttions that can be counted with your hand it should take just a couple of seconds. Duncan -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
participants (3)
-
Duncan Mac-Vicar Prett -
Ladislav Slezak -
Michael Andres