[Bug 906589] New: multiple repos repeatedly/frequently reporting "signed with an unknown key"
http://bugzilla.opensuse.org/show_bug.cgi?id=906589 Bug ID: 906589 Summary: multiple repos repeatedly/frequently reporting "signed with an unknown key" Classification: openSUSE Product: openSUSE.org Version: unspecified Hardware: All OS: All Status: NEW Severity: Normal Priority: P5 - None Component: Infrastructure Assignee: mrueckert@suse.com Reporter: grantksupport@operamail.com QA Contact: lrupp@suse.com Found By: --- Blocker: --- We've upgraded all our opensuse instances to 13.2; currently several hundred across multiple sites. As we're doing post-upgrade cleanups, etc, on `zypper *` we're seeing LOTS of 'unknown key' messages for/from repositories, for example, @ refresh, zypper -v ref Verbosity: 1 Initializing Target Specified repositories: Checking whether to refresh metadata for Backup Retrieving: repomd.xml ................................................................................................................................................................................................................[done] Repository 'Backup' is up to date. Checking whether to refresh metadata for BaseSystem Retrieving: repomd.xml ................................................................................................................................................................................................................[done] Retrieving: repomd.xml ................................................................................................................................................................................................................[done] Retrieving: repomd.xml.asc ............................................................................................................................................................................................................[done] Retrieving: repomd.xml.key ............................................................................................................................................................................................................[done] Retrieving: repomd.xml ................................................................................................................................................................................................................[done] File 'repomd.xml' from repository 'BaseSystem' is signed with an unknown key '88EB5D66E2C0098C'. Continue? [yes/no] (no): That ^^^ is just ONE example; most, if not yet all, enabled repos have returned this error at least once recently -- typically more often. This is NEW/CHANGED behavior. We're not alone -- we're hearing about this from multiple clients, and are bumping into similar issues/comments/questions online, in IRC, etc. This is happening for a broad variety of repos -- home: repos, 'semi-official' repos, *AND* official release/distribution repos. In any one run, there can be none-to-many repos that return the "signed with an unknown key" And, it's happening repeatedly & frequently. If I force clean up zypper clean --all rpm -qa | grep gpg-pubkey | xargs rpm -e zypper -vvv --gpg-auto-import-keys --no-gpg-checks ref --force then, an IMMEDIATELY subsequent `ref` or `dup`, of course, has no issues with unknown keys -- until "some time later". After a seemingly random amount of time -- just minutes to hours -- re-exec of the zypper cmd gets another mix of "unknown key" reports. For the example above, cat /etc/zypp/repos.d/BaseSystem.repo [BaseSystem] name=BaseSystem enabled=1 autorefresh=1 baseurl=http://download.opensuse.org/repositories/Base:/System/openSUSE_13.2 gpgcheck=1 keeppackages=0 priority=30 type=rpm-md Checking @ http://download.opensuse.org/repositories/Base:/System/openSUSE_13.2/repodat... Index of /repositories/Base:/System/openSUSE_13.2/repodata Icon Name Last modified Size [DIR] Parent Directory - [ ] 0ebcac183295ce4d1fde2c8f614bbe0fc481804c7948418a9ac0613ad16a5efe-primary.xml.gz 20-Nov-2014 14:48 23K Details [ ] 488fb3091c6e475a247d1b10a6035dafb05519f9fbd6ddaa5265c2826517b5d0-other.xml.gz 20-Nov-2014 14:48 25K Details [ ] d5fc3d48a3aa46cf156ac47421ec3d979ba0d7849fc503437701384455726e4b-filelists.xml.gz 20-Nov-2014 14:48 47K Details [TXT] repomd.xml 20-Nov-2014 14:48 1.6K Details [ ] repomd.xml.asc 20-Nov-2014 14:48 481 Details [ ] repomd.xml.key 20-Nov-2014 14:48 1.1K Details Apache/2.2.12 (Linux/SUSE) Server at download.opensuse.org Port 80 MirrorBrain powered by Apache it's clear there's a recent "Last Modified" change to the repodata ... I do not yet know if there ae ACTUAL changes, or only timestamps are changing. At first glance, it appears that with each change to the repo's content -- specifically the filelists -- the ENTIRE file content of the /repodata dir is being re-timestamped. Including the repomd.xml.key ... which would be ONE cause of the "unkonwn key" issue. It's *possible* that multiple repos have been compromised, and that blackhats are changing keys at will -- but I *seriously* doubt it; pls correct me if I'm wrong. (1) Why are multiple repos' keys changing so frequently -- even for the same repo, sometimes multiple times within a day or so? (2) There appears to be no mechanism/source for VALIDATING the new/updated keys from within a zypper command -- That's a potential security issue. How are keys to be validated? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #1 from grant k
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #2 from grant k
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
Marcus Rückert
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
grant k
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #4 from Marcus Rückert
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
Marcus Rückert
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #5 from grant k
how did you install those systems?
The vast majority of systems in place are recent upgrades from 13.1 -> 13.2 Some are upgrades directly from 12.3 -> 13.2 A very few are new 13.2 installs. I'm seeing this repo mis-behavior across the entire sample set, above.
it seems your auto import keys is not functioning.
If that were the case, then this zypper clean --all rpm -qa | grep gpg-pubkey | xargs rpm -e zypper -vvv --gpg-auto-import-keys --no-gpg-checks ref --force zypper -vvv ref would immediately fail for each & every repo, no? Atm, it does not. After "Some Time" passes, only some repos fail. It cursorily appears that those that fail have seen a recent update of repo file content -- with corresponding new timestamps -- and that the keys themseleves are similarly, new-timestamped. I'll dig around to see if/how I can provie to myself that the auto-import is/isn't failing. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
grant k
prove to myself that the auto-import is/isn't failing
I haven't managed that, but I am a bit confused by the above. Is there a specific test that'd answer the question -- does zypper's key auto-import work correctly? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
grant k
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #8 from grant k
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #9 from grant k
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #10 from grant k
NOTE the change: pubring.gpg -> pubring.kbx
exploring that change -- incl 'why' and what effect is has on this ^^^ issue ... reading https://www.gnupg.org/documentation/manuals/gnupg/kbxutil.html "... 11.1.1 Scrutinizing a keybox file A keybox is a file format used to store public keys along with meta information and indices. The commonly used one is the file pubring.kbx in the .gnupg directory. It contains all X.509 certificates as well as OpenPGP keys[1]. ... Footnotes [1] Well, OpenPGP keys are not implemented, gpg still used the keyring file pubring.gpg ..." I'm simply confused by that info. Is/isn't pubring.gpg expected/required -- by GnuPG &.or by zypper? And is the apparent lack of a pubring.gpg, with only a pubring.kbx present, a problem? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
Michael Andres
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
Marcus Meissner
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #12 from grant k
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #13 from grant k
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #14 from grant k
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #15 from Michael Andres
$ grep Key test.zypper.log 2014-11-24 05:58:56 <1> gk1(21113) [zypp::KeyRing] KeyRing.cc(Impl):176 Current KeyRing::DefaultAccept: 0000000000 2014-11-24 05:58:56 <1> gk1(21113) [zypp] RpmDb.cc(syncTrustedKeys):965 Going to sync trusted keys... 2014-11-24 05:58:56 <1> gk1(21113) [zypp] RpmDb.cc(syncTrustedKeys):969 Rpm keys to export into zypp trusted keyring: 0 2014-11-24 05:58:56 <1> gk1(21113) [zypp] RpmDb.cc(syncTrustedKeys):970 Zypp trusted keys to import into rpm database: 0 2014-11-24 05:58:56 <1> gk1(21113) [zypp] RpmDb.cc(syncTrustedKeys):1019 Trusted keys synced.
It looks like there are no trusted keys stored in the rpm database (check with rpm -qa 'gpg-pubkey*').
2014-11-24 05:58:58 <1> gk1(21113) [zypp] PublicKey.cc(Impl):307 Taking pubkey from /var/cache/zypp/raw/OS13-updateLjSa8t/repodata/repomd.xml.key of size 988 and sha1 f7a25091290faa78bd47849f4b5238e4ee648236 2014-11-24 05:58:58 <1> gk1(21113) [zypp] PublicKey.cc(readFromFile):347 Reading pubkey from /var/tmp/TmpFile.UUPtIw of size 988 and sha1 f7a25091290faa78bd47849f4b5238e4ee648236 2014-11-24 05:58:58 <1> gk1(21113) [zypp] PublicKey.cc(readFromFile):397 Read pubkey from /var/tmp/TmpFile.UUPtIw: [B88B2FD43DBDC284-53674dd4] [openSUSE Project Signing Key
] [22C07BA534178CD02EFE22AAB88B2FD43DBDC284] [TTL 3446] 2014-11-24 05:58:59 <1> gk1(21113) [zypp::KeyRing] KeyRing.cc(verifyFileSignatureWorkflow):373 Going to verify signature for repomd.xml ( /var/cache/zypp/raw/OS13-updateLjSa8t/repodata/repomd.xml ) with /var/cache/zypp/raw/OS13-updateLjSa8t/repodata/repomd.xml.asc 2014-11-24 05:58:59 <1> gk1(21113) [zypp::KeyRing] KeyRing.cc(readSignatureKeyId):567 Determining key id if signature /var/cache/zypp/raw/OS13-updateLjSa8t/repodata/repomd.xml.asc 2014-11-24 05:58:59 <1> gk1(21113) [zypp::KeyRing] KeyRing.cc(readSignatureKeyId):612 Determined key id [B88B2FD43DBDC284] for signature /var/cache/zypp/raw/OS13-updateLjSa8t/repodata/repomd.xml.asc 2014-11-24 05:58:59 <1> gk1(21113) [zypp::KeyRing] KeyRing.cc(publicKeyExists):301 Searching key [B88B2FD43DBDC284] in keyring /var/tmp/zypp.8YO46w/zypp-trusted-krhw1kNd 2014-11-24 05:58:59 <1> gk1(21113) [zypp::KeyRing] KeyRing.cc(publicKeyExists):301 Searching key [B88B2FD43DBDC284] in keyring /var/tmp/zypp.8YO46w/zypp-general-krtrsBtU 2014-11-24 05:58:59 <1> gk1(21113) [zypp::KeyRing] KeyRing.cc(verifyFileSignatureWorkflow):477 File [/var/cache/zypp/raw/OS13-updateLjSa8t/repodata/repomd.xml] ( repomd.xml ) signed with unknown key [B88B2FD43DBDC284] 2014-11-24 05:59:04 <1> gk1(21113) [zypp::KeyRing] KeyRing.cc(verifyFileSignatureWorkflow):485 User does not want to accept unknown key B88B2FD43DBDC284
Trusted keys from the rpm database would have been imported into the 'zypp-trusted' keyring. Keys you temporarily accept would go to the 'zypp-general' keyring. Both keyrings are empty, so the key is unknown and needs to be confirmed.
2014-11-24 05:59:05 <1> gk1(21113) [zypp] PublicKey.cc(Impl):307 Taking pubkey from /var/cache/zypp/raw/ToolswnzCDp/repodata/repomd.xml.key of size 1003 and sha1 123d8c16811bd1cc3c68a467c3a92b2c8d2e3386 2014-11-24 05:59:05 <1> gk1(21113) [zypp] PublicKey.cc(readFromFile):347 Reading pubkey from /var/tmp/TmpFile.rhP5DU of size 1003 and sha1 123d8c16811bd1cc3c68a467c3a92b2c8d2e3386 2014-11-24 05:59:05 <1> gk1(21113) [zypp] PublicKey.cc(readFromFile):397 Read pubkey from /var/tmp/TmpFile.rhP5DU: [868EA2EB32567F38-50757861] [devel:tools OBS Project devel:tools@build.opensuse.org] [8BE2DD6EBB28D2578A36828C868EA2EB32567F38] [TTL 24] ... 2014-11-24 05:59:10 <1> gk1(21113) [zypp::KeyRing] KeyRing.cc(verifyFileSignatureWorkflow):485 User does not want to accept unknown key 868EA2EB32567F38
Same for 868EA2EB32567F38. Now it would be interesting to have a log where you permanently accept a key and we can see whether it's actually imported into the rpmdb. Or a log where the trusted key was in the rpmdb and you nevertheless were asked to accept it. If such logs are available, please attach them. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #16 from grant k
Now it would be interesting to have a log where you permanently accept a key and we can see whether it's actually imported into the rpmdb.
this ^^ , atm, I have repeating the test, forcing a clean, but accepting the 'Continue' <-- yes zypper clean --all zypper -v log output tail -f zypper.log is attached as test2.zypper.log
Or a log where the trusted key was in the rpmdb and you nevertheless were asked to accept it.
if still needed, will have to let some time pass -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #17 from Michael Andres
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #18 from grant k
a test/workaround could be:
I've v1429 ls -al /usr/lib64/libzypp* lrwxrwxrwx 1 root root 15 Nov 9 18:02 /usr/lib64/libzypp.so -> libzypp.so.1429* lrwxrwxrwx 1 root root 19 Nov 9 17:50 /usr/lib64/libzypp.so.1429 -> libzypp.so.1429.0.4* -rwxr-xr-x 1 root root 4.8M Oct 15 11:18 /usr/lib64/libzypp.so.1429.0.4* one instance strings /usr/lib64/libzypp.so.1429.0.4 | grep pubring pubring.gpg changed sed -i 's/pubring\.gpg/pubring.kbx/' /usr/lib64/libzypp.so.1429.0.4 strings /usr/lib64/libzypp.so.1429.0.4 | grep pubring pubring.kbx repeating with 2-repo test gpg --version gpg (GnuPG) 2.1.0 libgcrypt 1.6.2 ... zypper clean --all zypper -vvv ref Verbosity: 3 Initializing Target Specified repositories: Checking whether to refresh metadata for OS13-update Retrieving: http://download.opensuse.org/update/13.2/repodata/repomd.xml.asc ........................................................................................................................[done] Retrieving: http://download.opensuse.org/update/13.2/repodata/repomd.xml.key ........................................................................................................................[done] Retrieving repository 'OS13-update' metadata .........................................................................................................................................................................................[error] Repository 'OS13-update' is invalid. [OS13-update|http://download.opensuse.org/update/13.2] Valid metadata not found at specified URL History: - File /var/tmp/TmpFile.F05fw7 doesn't contain public key data Please check if the URIs defined for this repository are pointing to a valid repository. Skipping repository 'OS13-update' because of the above error. Checking whether to refresh metadata for Tools Retrieving: http://download.opensuse.org/repositories/devel:/tools/openSUSE_13.2/repodat... ............................................................................................[done] Retrieving: http://download.opensuse.org/repositories/devel:/tools/openSUSE_13.2/repodat... ............................................................................................[done] Retrieving repository 'Tools' metadata ...............................................................................................................................................................................................[error] Repository 'Tools' is invalid. [Tools|http://download.opensuse.org/repositories/devel:/tools/openSUSE_13.2] Valid metadata not found at specified URL History: - File /var/tmp/TmpFile.kJT0BC doesn't contain public key data Please check if the URIs defined for this repository are pointing to a valid repository. Skipping repository 'Tools' because of the above error. Could not refresh the repositories because of errors. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #19 from Michael Andres
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #20 from grant k
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #21 from grant k
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #22 from Michael Andres
so, WHY is that repo getting a "New repository or package signing key" ?
Authority for trusted keys on your system is the rpm databbase, namely the gpg-pubkey (pseudo) packages. (rpm -qai gpg-pubkey)
New repository or package signing key received: Repository: MediaApps Key Name: multimedia OBS Project .... Key Fingerprint: 01FB54D4 EAF87B0F 5624266B 5F3D540F 3A802234 Key Created: Wed 21 May 2014 02:04:08 PM PDT Key Expires: Fri 29 Jul 2016 02:04:08 PM PDT Rpm Name: gpg-pubkey-3a802234-537d14c8
The fingerprints last 8 byte make the gpg-pubkey packages version. The key is known and trusted if we have gpg-pubkey-3a802234 in the rpm database. If the rpm data base does not contain gpg-pubkey-3a802234, the key is NEW and you are asked to confirm it. If you accept the key it is imported into the rpm database. After this, gpg-pubkey-3a802234 is present and you don't have to confirm it again, until it is removed from the rpm database (rpm -e gpg-pubkey-3a802234). If the server side did not change, the key was most probably not in the rpm database (or you accidentally used the sed-patched libzypp with gpg-2.0.x). A zypper.log could tell... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
--- Comment #23 from grant k
or you accidentally used the sed-patched libzypp with gpg-2.0.x
yep. PEBKAC. my bad :-/ -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=906589
Michael Andres
participants (1)
-
bugzilla_noreply@novell.com