[Bug 1175637] New: Found uKmail/akonadictl : unreferenced external file: /home/<userid>/.local/share/akonadi/file_db_data/11/80611_r0
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 Bug ID: 1175637 Summary: Found uKmail/akonadictl : unreferenced external file: /home/<userid>/.local/share/akonadi/file_db_data/11/80 611_r0 Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.2 Hardware: x86-64 OS: openSUSE Leap 15.2 Status: NEW Severity: Normal Priority: P5 - None Component: KDE Applications Assignee: opensuse-kde-bugs@opensuse.org Reporter: ndordea@gmail.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- Hello, I deleted some old emails via Move to Trash .... [ i.e. select the emails to delete then Move to Trash . At the beginning kmail/akonadi worked fine ... but when I start deleting hundred of emails ... the process came to a still and I let the laptom over night to finish the delete . In the morning the delete was done and the trash folder was empty , then I did akonadictl fsck I got a lot of messages ... Found unreferenced external file: /home/<userid>/.local/share/akonadi/file_db_data/11/80611_r0 and that 100 files were moved to lost+found . Is there any way/proc/etc to get rid of those messages ? My understanding is that kmail/akonadi deletes all database records related to the deleted emails . Those messages point to an unsynchroned database after deletion . Please have a look at this issue and tell me what/how to do to get rid of those messages . Please find below akonadictl fsck output . Thank you. Best regards, Nick ---------------------------------- akonadictl fsck Looking for resources in the DB not matching a configured resource... Looking for collections not belonging to a valid resource... Checking collection tree consistency... Looking for items not belonging to a valid collection... Looking for item parts not belonging to a valid item... Looking for item flags not belonging to a valid item... Looking for overlapping external parts... Verifying external parts... Found 101 external files. Found 0 external parts. Found unreferenced external file: /home/<userid>/.local/share/akonadi/file_db_data/11/80611_r0 Found unreferenced external file: /home/<userid>/.local/share/akonadi/file_db_data/01/275601_r0 Found unreferenced external file: /home/<userid>/.local/share/akonadi/file_db_data/69/278169_r0 Found unreferenced external file: /home/<userid>/.local/share/akonadi/file_db_data/70/272670_r0 ...... 0 Found unreferenced external file: /home/<userid>/.local/share/akonadi/file_db_data/94/273694_r0 Found unreferenced external file: /home/<userid>/.local/share/akonadi/file_db_data/91/273891_r0 Found unreferenced external file: /home/<userid>/.local/share/akonadi/file_db_data/24/273724_r0 Found unreferenced external file: /home/<userid>/.local/share/akonadi/file_db_data/92/276492_r0 Moved 101 unreferenced files to lost+found. Checking size treshold changes... Found 0 parts to be moved to external files Found 0 parts to be moved to database Looking for dirty objects... Collection "Search" (id: 1) has no RID. Collection "OpenInvitations" (id: 151) has no RID. Collection "DeclinedInvitations" (id: 152) has no RID. Collection "Import_from_archive" (id: 160) has no RID. Collection "Last Search" (id: 513) has no RID. Found 5 collections without RID. Found 0 items without RID. Found 0 dirty items. Looking for rid-duplicates not matching the content mime-type of the parent collection Checking Local Folders Checking Notes Checking Personal Calendar Checking Search ..... Checking Sent Checking SentMail Checking Trash Migrating parts to new cache hierarchy... Checking search index consistency... Skipping virtual Collection 1 Checking Collection 2 search index... Checking Collection 3 search index... Checking Collection 4 search index... Checking Collection 5 search index... ...... Checking Collection 648 search index... Checking Collection 649 search index... Checking Collection 724 search index... Flushing collection statistics memory cache... Making sure virtual search resource and collections exist Consistency check done. ---------------------------------- -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 Nick Dordea <ndordea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ndordea@gmail.com -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c1 --- Comment #1 from Nick Dordea <ndordea@gmail.com> --- Hello, I checked what records are still saved in /$HOME/.local/share/akonadi/file_db_data directory ..... Guess what ? All the records akonadictl fsck says that were MOVED to /$HOME/.local/share/akonadi/file_lost+found/ reside on the /$HOME/.local/share/akonadi/file_db_data subdirecories !!! So akonadictl says that it moves a file but in fact it copies it ... and the original file remains in /$HOME/.local/share/akonadi/file_db_data subdirectories . It seems that akonadictl fsck needs to be modifies to do what it doing !!!! I hope that this change will be implemented soon, saving users'time of researching why the so called moved files are reported over and over again . Due to the fact that other similar issues might clobber database consistency , it is there any database repair app/command/etc. so check and reapair the internal relationships between database objects ? That will help the users a lot !!! Please let me know if KDE team will fix akonadi/kmail programs to remove the above bug . Thanks, Nick Dordea -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c3 Wolfgang Bauer <wbauer@tmo.at> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wbauer@tmo.at --- Comment #3 from Wolfgang Bauer <wbauer@tmo.at> --- Perhaps it should be added that those "unreferenced external files" do *not* cause any problem. They are just unused and only take hard disk space unnecessarily. And another note: deleting mails is indeed unnecessarily slow, even without bug#1173759. This has been improved recently in the git master branch: https://invent.kde.org/pim/akonadi/-/commit/9eda8853c08b2b2eba6070ab16f27ffc... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c4 Nick Dordea <ndordea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ndordea@gmail.com | |) | --- Comment #4 from Nick Dordea <ndordea@gmail.com> --- Hello, Here are the answers as well as some of my questions . 1) What are the timestamps of the files in file_db_data and file_lost+found? the same 2) Are the deleted mails deleted in the source folder? No . 3) Are they also gone server side? I do not know 4) Any errors in ~/.local/share/akonadi/*.error? Unable to register service as "org.freedesktop.Akonadi.Control.lock" Maybe it's already running? Application '/usr/bin/akonadi_newmailnotifier_agent' exited normally... Application '/usr/bin/akonadi_maildispatcher_agent' exited normally... Application '/usr/bin/akonadi_maildir_resource' exited normally... Application '/usr/bin/akonadi_sendlater_agent' exited normally... Application '/usr/bin/akonadi_indexing_agent' exited normally... Application '/usr/bin/akonadi_ical_resource' exited normally... Application '/usr/bin/akonadi_contacts_resource' exited normally... Application '/usr/bin/akonadi_migration_agent' exited normally... Application '/usr/bin/akonadi_archivemail_agent' exited normally... Application '/usr/bin/akonadi_unifiedmailbox_agent' exited normally... Application '/usr/bin/akonadi_imap_resource' exited normally... Application '/usr/bin/akonadi_notes_agent' exited normally... Application '/usr/bin/akonadi_followupreminder_agent' exited normally... Application '/usr/bin/akonadi_akonotes_resource' exited normally... Application '/usr/bin/akonadi_mailfilter_agent' exited normally... Application 'akonadiserver' exited normally.. 5) If you manually delete one record from file_db_data/xx/ that is reported by akonadictl fsck as "unreferenced", akonadictl stop/start/fsck DOES NOT REPORT it as unreferenced . So akonadictl fsck says that is moves a record but it is only doing a copy . 6) Are the records in file_lost+found directory needed ? Can I delete them and save some hard disk space ? What about the un-necessary records in file_db_data ? Could I manually delete the "unreferenced" records from file_db_data ? That will clean-up akonadictl fsck report . 7) Does this akonadictl fsck modify any mysql/MariaDB structures/tables/relationships ? If so is there any way to check/repair akonadi database? Thank you , Nick -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c6 --- Comment #6 from Wolfgang Bauer <wbauer@tmo.at> --- (In reply to Fabian Vogt from comment #5)
Ok, so they should not actually be unreferenced. They may be if the cached entries are purged from the database (which is just a cache), but the referenced files in file_db_data were not/could not be deleted for some reason.
Just a thought: maybe akonadictl fsck cannot rename/move the files because they are actually still in use by some process? And maybe akonadi itself fails to delete them in the first place when it removes entries from the cache for the same reason? (resulting in those unreferenced files to exist) I suppose the strace should show if that's the case... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c7 --- Comment #7 from Wolfgang Bauer <wbauer@tmo.at> --- PS:
2) Are the deleted mails deleted in the source folder? No . That may indeed be a side-effect of bug#1173759.
AFAIK, deleting (or moving, which is actually a copy followed by a delete) mails is not really reliable if a folder refresh is running at the same time. And that bug causes (maildir) folders getting refreshed all the time, as I understand it. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c8 --- Comment #8 from Wolfgang Bauer <wbauer@tmo.at> --- And for the record: I just ran akonadictl fsck on my system (with latest KDEPIM and Qt, so not affected by bug#1173759). It did find 2 unreferenced files in file_db_data, but it successfully *moved* them to file_lost+found, i.e. they no longer existed in file_db_data afterwards. So this seems to be working in general. (unless it is a bug in 20.04.2 that got fixed in 20.08.0, but I don't remember seeing a corresponding change) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c9 Nick Dordea <ndordea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ndordea@gmail.com | |) | --- Comment #9 from Nick Dordea <ndordea@gmail.com> --- Hello, 1) Just a thought: maybe akonadictl fsck cannot rename/move the files because they are actually still in use by some process? And maybe akonadi itself fails to delete them in the first place when it removes entries from the cache for the same reason? (resulting in those unreferenced files to exist) ND) It may be the case . After upgrade to leap 15.2, when moving emails to trash, kmail/akonadi is very slow and it take a long of time until the move is done/completed . That may cause some mismatch between hardening the changes to the the logs vs. database consistency . 2) And for the record: I just ran akonadictl fsck on my system (with latest KDEPIM and Qt, so not affected by bug#1173759). It did find 2 unreferenced files in file_db_data, but it successfully *moved* them to file_lost+found, i.e. they no longer existed in file_db_data afterwards. So this seems to be working in general. (unless it is a bug in 20.04.2 that got fixed in 20.08.0, but I don't remember seeing a corresponding change) ND)My machine indicates that kmail/akonadi are at 20.04.2 level [ Main_Distribution_oss distro] Would the bug#1173759 change/update the kmail/akonadi 20.04.02 ? 3) Speaking of akonadictl ... are there any chances to enhance the akonadictl messages to provide more information to/for the users . Those collection numbers, record-ids, etc. do not point to what the users are dealing with ... --- folders --- email title/date So how could we tell KDE support teams what is the "source" of a particular email .... Thank you, Nick -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c10 Fabian Vogt <fabian@ritter-vogt.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fabian@ritter-vogt.de Flags| |needinfo?(ndordea@gmail.com | |) --- Comment #10 from Fabian Vogt <fabian@ritter-vogt.de> --- (In reply to Wolfgang Bauer from comment #6)
(In reply to Fabian Vogt from comment #5)
Ok, so they should not actually be unreferenced. They may be if the cached entries are purged from the database (which is just a cache), but the referenced files in file_db_data were not/could not be deleted for some reason.
Just a thought: maybe akonadictl fsck cannot rename/move the files because they are actually still in use by some process?
Fortunately this is not Windows, so unlinking succeeds regardless of open handles. Not even mandatory locks can prevent that apparently.
And maybe akonadi itself fails to delete them in the first place when it removes entries from the cache for the same reason? (resulting in those unreferenced files to exist) I suppose the strace should show if that's the case...
Yep, adding NEEDINFO back. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c11 --- Comment #11 from Nick Dordea <ndordea@gmail.com> --- Created attachment 841044 --> http://bugzilla.opensuse.org/attachment.cgi?id=841044&action=edit bug#1175637_strace_-efile_akonadictl_fsck.txt bug#1175637_strace_-efile_akonadictl_fsck.txt ; 2cases : kmail up / kmail down -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c12 Nick Dordea <ndordea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ndordea@gmail.com | |) | --- Comment #12 from Nick Dordea <ndordea@gmail.com> --- Hello, Performed strace -efile akonadictl fsck twice case 1. kmail up case 2, kmail down In both cases the "moved" unreferenced records are still in the file_db_data subdirectories . So akonadictl do not delete them . Thanks. Nick -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c14 Nick Dordea <ndordea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ndordea@gmail.com | |) | --- Comment #14 from Nick Dordea <ndordea@gmail.com> --- Hello, strace -fefile -p $(pidof akonadiserver) |& grep -A5 -B5 lost + akonadictl fsck done (kmail up) The trace file provided as an attachment . bug#1175637_strace_-fefile_-p(pidof akonadiserver)_grep_-A5_-B5_then_akonadictl_fsck.txt Thank you, Nick -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c15 --- Comment #15 from Nick Dordea <ndordea@gmail.com> --- Created attachment 841095 --> http://bugzilla.opensuse.org/attachment.cgi?id=841095&action=edit bug#1175637_strace_-fefile_-p(pidof akonadiserver)_grep_-A5_-B5_then_akonadictl_fsck.txt -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c16 --- Comment #16 from Nick Dordea <ndordea@gmail.com> --- Hello, akonadictl fsck output akonadictl fsck Looking for resources in the DB not matching a configured resource... Looking for collections not belonging to a valid resource... Checking collection tree consistency... Looking for items not belonging to a valid collection... Looking for item parts not belonging to a valid item... Looking for item flags not belonging to a valid item... Looking for overlapping external parts... Verifying external parts... Found 29 external files. Found 0 external parts. Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/11/272611_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/96/274296_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/02/277702_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/96/274096_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/99/275599_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/69/272369_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/11/80611_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/98/277298_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/98/273898_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/06/278606_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/70/272670_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/01/273901_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/09/275409_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/10/272610_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/01/277701_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/76/273676_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/69/278169_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/04/272704_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/70/272370_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/01/275601_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/05/278605_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/11/275411_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/76/274276_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/00/273900_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/03/273903_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/03/272703_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/77/277877_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/77/227366_r0 Found unreferenced external file: /home/rocco/.local/share/akonadi/file_db_data/98/274098_r0 Moved 29 unreferenced files to lost+found. Checking size treshold changes... Found 0 parts to be moved to external files Found 0 parts to be moved to database Looking for dirty objects... Collection "Search" (id: 1) has no RID. Collection "OpenInvitations" (id: 151) has no RID. Collection "DeclinedInvitations" (id: 152) has no RID. Collection "Import_from_archive" (id: 160) has no RID. Collection "Last Search" (id: 513) has no RID. Found 5 collections without RID. Found 0 items without RID. Found 0 dirty items. ---------------------- Thanks, Nick -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c18 Nick Dordea <ndordea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ndordea@gmail.com | |) | --- Comment #18 from Nick Dordea <ndordea@gmail.com> --- Hello, Performed the following : 1) empty file_lost+found (Please note that it contained 12,206 records ) 2) clone ~/.local/share/akonadi/file_db_data dir ~/file_db_dbata_before_strace 3) start strace 4) akonadictl fsck [ kmail up ] 5) ~/.local/share/akonadi/file_lost+found ]> ls -al * | wc -l 29 6) kdiff3 ~/.local/share/akonadi/file_db_data vs. ~/file_db_data_before_strace Directory Comparison Status Number of subdirectories: 100 Number of equal files: 2 Number of different files: 29 ----------------------------- It seems that the unreferenced records were moved from db_data to file_lost+found You are right ... akonadiserver does not overide/replace equal records in in file_lost+found . What about,that before fsck starts , all unreferenced records from ~/.local/share/akonadi/file_lost+found are moved to ~/.local/share/akonadi/file_lost+found_saved ? That would do the trick . The strace contents will be appended as file bug#1175637_empty_file_lost+found_strace_-fefile_-p(pidof akonadiserver)_grep_-A5_-B5_then_akonadictl_fsck.txt Thanks, Nick -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175637 http://bugzilla.opensuse.org/show_bug.cgi?id=1175637#c19 --- Comment #19 from Nick Dordea <ndordea@gmail.com> --- Created attachment 841143 --> http://bugzilla.opensuse.org/attachment.cgi?id=841143&action=edit bug#1175637_empty_file_lost+found_strace_-fefile_-p(pidof akonadiserver)_grep_-A5_-B5_then_akonadictl_fsck.txt -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com