-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I have a large mail folder, with 20000 mails, of which probably thousands are duplicates. I can see them with my eyes. I have a thunderbird extension to find duplicates. "Remove Duplicate Messages" by Eyal Rozenberg. https://github.com/eyalroz/removedupes/ But it claims there are no duplicates. I have read the FAQ. Subsequently, I saved to file a message and its duplicate, to do a compare, and yes, there is a difference: X-Spam-Status: No, score=-3.4 required=5.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, RCVD_IN_DNSWL_MED,RCVD_IN_ZEN_BLOCKED_OPENDNS,RDNS_NONE,SPF_HELO_NONE, SPF_PASS autolearn=disabled version=3.4.5 X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, RDNS_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=disabled version=3.4.5 The reason of that difference, is that each copy comes from a different run of my sorting script, the spam filter "thought differently". (note to myself: disable spam filtering in that script) Do you know some other tool to find and remove duplicates from a mail folder, where I can tell it to ignore some header? ("Delete" means moving to trash folder) Or maybe remove the spam header. X-Spam-Checker-Version: SpamAssassin 3.4.5 (2021-03-20) on Telcontar.valinor X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, RCVD_IN_DNSWL_MED,RCVD_IN_ZEN_BLOCKED_OPENDNS,RDNS_NONE,SPF_HELO_NONE, SPF_PASS autolearn=disabled version=3.4.5 Logic: IF X-Spam-Checker-Version contains Telcontar.valinor, then remove X-Spam-Level: and X-Spam-Status: headers. - -- Cheers Carlos E. R. (from 15.4 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCZKPZtBwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVBmIAn2w1CHy/BOtOr+1Dl+CV KNqXEDgvAKCPtbhf4Aa9d4Hh5F3M486zrJupSg== =mTAn -----END PGP SIGNATURE-----