[opensuse] Corrupted RPM database
Hi all, I wonder if any of you had a similar experience and/or can help me solve this. I had an ext3 root filesystem on top of a soft-raid RAID0. One of the hard drives started to issue media errors. I immediately stopped all work, booted with a rescue system, and was able to tar all files from the filesystem, except two ( cannot remember, sorry ) belonging to /var/lib/rpm. tar warned that it did include the file with padding zeros. My first thought was that it not a big deal, as I could rebuild the RPM database. When I booted into the system again I tried fixing the mentioned database but, I'm experiencing a lot of errors. I'll enumerate the attempts and errors: 1) rpmdb --rebuilddb Error: error: rpmdbNextIterator: skipping h# 881 blob size(21356): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 913 blob size(13820): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 945 blob size(67712): BAD, 8 + 16 * il(0) + dl(0) ... error: rpmdbNextIterator: skipping h# 5086 blob size(16084): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 5118 blob size(28752): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 5135 blob size(4740): BAD, 8 + 16 * il(0) + dl(0) Segmentation fault 2) db_verify fails on the files /var/lib/rpm/{Basenames,Filemd5s,Packages} 3) Tried ( on all databases ): mv Basename Basename.original; db_dump Basename.original | db_load Basename db_verify now succeeds . After this I issued rpmdb --rebuilddb: It also issued errors but, from rpmdb, the database is now OK. Problem: After all this operations the databases size has shrunk dramatically. eg: Basenames 21MB -> 2.6MB. Filemd5s: 21MB -> 432KB. Packages: 77MB -> 14MB. My first thought was "bummer - there goes my rpm database." My questions are: 1) Is the supposedly corrected database, OK? How can I verify it? 2) Is it possible to build an RPM database from scratch ? Thanks for you help, -- Rui Santos http://www.ruisantos.com/ Veni, vidi, Linux! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Jul 16, 2008 at 04:35:09PM +0100, Rui Santos wrote:
When I booted into the system again I tried fixing the mentioned database but, I'm experiencing a lot of errors. I'll enumerate the attempts and errors:
1) rpmdb --rebuilddb Error: error: rpmdbNextIterator: skipping h# 881 blob size(21356): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 913 blob size(13820): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 945 blob size(67712): BAD, 8 + 16 * il(0) + dl(0) ... error: rpmdbNextIterator: skipping h# 5086 blob size(16084): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 5118 blob size(28752): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 5135 blob size(4740): BAD, 8 + 16 * il(0) + dl(0) Segmentation fault
2) db_verify fails on the files /var/lib/rpm/{Basenames,Filemd5s,Packages}
There's only one critical Database, it's "Packages". All other databases are just indices and get regenerated by 'rpm --rebuildb'. If Packages is corrupt (like in your case), you're in big trouble. The only way to recover is to use db_dump/db_load to salvage all undamaged parts of the old database. Then do 'rpm --rebuilddb'. The result will be that some packages are missing from the database, i.e. rpm no longer knows that the package is installed. There's no way to get the old data back, but you can run "zypper verify" to reinstall missing packages. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Michael Schroeder wrote:
On Wed, Jul 16, 2008 at 04:35:09PM +0100, Rui Santos wrote:
When I booted into the system again I tried fixing the mentioned database but, I'm experiencing a lot of errors. I'll enumerate the attempts and errors:
1) rpmdb --rebuilddb Error: error: rpmdbNextIterator: skipping h# 881 blob size(21356): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 913 blob size(13820): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 945 blob size(67712): BAD, 8 + 16 * il(0) + dl(0) ... error: rpmdbNextIterator: skipping h# 5086 blob size(16084): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 5118 blob size(28752): BAD, 8 + 16 * il(0) + dl(0) error: rpmdbNextIterator: skipping h# 5135 blob size(4740): BAD, 8 + 16 * il(0) + dl(0) Segmentation fault
2) db_verify fails on the files /var/lib/rpm/{Basenames,Filemd5s,Packages}
There's only one critical Database, it's "Packages". All other databases are just indices and get regenerated by 'rpm --rebuildb'.
If Packages is corrupt (like in your case), you're in big trouble. The only way to recover is to use db_dump/db_load to salvage all undamaged parts of the old database. Then do 'rpm --rebuilddb'.
The result will be that some packages are missing from the database, i.e. rpm no longer knows that the package is installed. There's no way to get the old data back, but you can run "zypper verify" to reinstall missing packages.
You mean that "zypper verify" tries to check all dependencies on the few installed rpm's and goes from there in the hope that it will reverse check all others ? Even so, I'm using openSUSE 10.3, witch does not contain the verify option. I'm trying to install the Backported 11.0 version of zypper but the dependencies problems of packages that are installed but not reported as such is beginning. Well, wish me luck :) Thanks for you quick reply Michael. I'll post the results later on...
Cheers, Michael.
-- Rui Santos http://www.ruisantos.com/ Veni, vidi, Linux! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Well, wish me luck :)
Which is exactly the reason I backup /var/lib/rpm every once in a while (usually after system update). I had one too many corrupt RPM databases so found this to be most painless method of preserving RPM database. Good luck in your quest for missing packages ;) -- Best regards, Nick Zeljkovic -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Jul 16, 2008 at 05:14:09PM +0100, Rui Santos wrote:
You mean that "zypper verify" tries to check all dependencies on the few installed rpm's and goes from there in the hope that it will reverse check all others ?
Even so, I'm using openSUSE 10.3, witch does not contain the verify option.
I think 10.3 YaST also has a "verify" button in the UI. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Wednesday 2008-07-16 at 17:47 +0200, Michael Schroeder wrote:
There's only one critical Database, it's "Packages". All other databases are just indices and get regenerated by 'rpm --rebuildb'.
If Packages is corrupt (like in your case), you're in big trouble. The only way to recover is to use db_dump/db_load to salvage all undamaged parts of the old database. Then do 'rpm --rebuilddb'.
There is another way: in "/var/adm/backup/rpmdb" you have daily bakcups of that database made by cron (run-crons: suse.de-backup-rpmdb). By default it keeps 5 copies, you can increase the number somewhere in /etc/sysconfig. Once you get a good copy, you only need to worry about what you did the last days(s). - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) iD8DBQFIfil3tTMYHG2NR9URAnmtAJ0WthvQueQKXpxAqK2xoJqxMFpMLQCeJzCe 8W6ak3WGARnAW26MmYuNoNU= =PPl5 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Jul 16, 2008 at 07:01:31PM +0200, Carlos E. R. wrote:
There is another way: in "/var/adm/backup/rpmdb" you have daily bakcups of that database made by cron (run-crons: suse.de-backup-rpmdb). By default it keeps 5 copies, you can increase the number somewhere in /etc/sysconfig.
Once you get a good copy, you only need to worry about what you did the last days(s).
Good point, I had forgotten that backup. So there's hope for Rui... Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Michael Schroeder wrote:
On Wed, Jul 16, 2008 at 07:01:31PM +0200, Carlos E. R. wrote:
There is another way: in "/var/adm/backup/rpmdb" you have daily bakcups of that database made by cron (run-crons: suse.de-backup-rpmdb). By default it keeps 5 copies, you can increase the number somewhere in /etc/sysconfig.
Once you get a good copy, you only need to worry about what you did the last days(s).
Good point, I had forgotten that backup. So there's hope for Rui...
More than hope, a solution :) I restored the Backup from the place Carlos indicated. It was from 2008-07-14, just one day before the crash. Thanks for your help Guys... My system is now up and running...
Cheers, Michael.
-- Rui Santos http://www.ruisantos.com/ Veni, vidi, Linux! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Wednesday 2008-07-16 at 19:08 +0200, Michael Schroeder wrote:
On Wed, Jul 16, 2008 at 07:01:31PM +0200, Carlos E. R. wrote:
There is another way: in "/var/adm/backup/rpmdb" you have daily bakcups of that database made by cron (run-crons: suse.de-backup-rpmdb). By default it keeps 5 copies, you can increase the number somewhere in /etc/sysconfig.
Once you get a good copy, you only need to worry about what you did the last days(s).
Good point, I had forgotten that backup. So there's hope for Rui...
It's very easy to forget that one, it is almost "hidden". I remember simply because I was bitten by rpmdb corruption and that backup saved me. But I had to search for it for my email, I didn't remember where it was. There is also a backup of some configuration files in there. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) iD8DBQFIfjcltTMYHG2NR9URAmTKAKCVs2XHtRu8lrkFINmUXBYV7mI7AwCeJJ3z xDdR/JpqX6eLB1x3g2id3Vg= =By/e -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
The Wednesday 2008-07-16 at 17:47 +0200, Michael Schroeder wrote:
There's only one critical Database, it's "Packages". All other databases are just indices and get regenerated by 'rpm --rebuildb'.
If Packages is corrupt (like in your case), you're in big trouble. The only way to recover is to use db_dump/db_load to salvage all undamaged parts of the old database. Then do 'rpm --rebuilddb'.
There is another way: in "/var/adm/backup/rpmdb" you have daily bakcups of that database made by cron (run-crons: suse.de-backup-rpmdb). By default it keeps 5 copies, you can increase the number somewhere in /etc/sysconfig.
Once you get a good copy, you only need to worry about what you did the last days(s).
Yes. Thanks for the TIP Carlos. I was just giving up on the dependencies issues, and were preparing to reinstall the system. Once again, thanks a lot.
-- Cheers, Carlos E. R.
-- Rui Santos http://www.ruisantos.com/ Veni, vidi, Linux! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (4)
-
Carlos E. R. -
Michael Schroeder -
Nick Zeljkovic -
Rui Santos