out of date certificates in caches
![](https://seccdn.libravatar.org/avatar/065b1b1774363493af32c0a6ce9ff07c.jpg?s=120&d=mm&r=g)
I just spent several days tracking down a problem on OpenSuSE 15.2 which I don't know who has the responsibility for reporting what failed, the operating system or the server/client applications. Anywise what happened was Certbot automatically updated some security certificates on one of my systems (Nova) and I wasn't aware of the consequences of it doing so. After that happened all of the sudden I lost the ability to connect to the services running on Nova. Autofs/Automount could not mount Samba shares from Nova, vncviewer could not connect to x11vnc service on Nova, ssh could not connect to sshd, knockd no longer opened ports, etc. None of the clients (except ssh but with a caveat) reported anything useful, just generalized crap like "No route to host" I hide a lot of the service's ports behind a port knock sequence which rendered the error messages from ssh invisible also. It was only when I directly opened the ssh port, on Nova, in firewalld and tried connecting that I finally learned what was going on. The cached certificates on my other systems were no longer valid and thus causing the failure. Only ssh actually offers a solution to the user to update the cached certificates with the new one from Nova. But again only if the server's port is not being handled by a portknocker. This was very difficult to track down and resolve! My question is, who should have the responsibility for detecting these out of date certificates, and offering a solution to the users when they are encountered? To my Object Oriented Design mind it seems the O.S. should have this responsibility and it should not be delegated to each individual client/server application. Maybe a distro is not the right place to start such a conversation, but I am only a poor user of OpenSuSE and not a member of any more centralized Linux group(s). After spending so much time tracking this down, I feel this is a deep design flaw in Linux itself but have no idea where to report it. Marc... -- *"The Truth is out there" - Spooky* *_ _ . . . . . . _ _ . _ _ _ _ . . . . _ . . . . _ _ . _ _ _ . . . . _ _ . _ . . _ . _ _ _ _ . _ . _ . _ . _ . * Computers: the final frontier. These are the voyages of the user Marc. His mission: to explore strange new hardware. To seek out new software and new applications. To boldly go where no Marc has gone before! (/This email is digitally signed. My public key for sending encrypted email to me can be found at - https://keys.openpgp.org/search?q=marc@marcchamberlin.com or just ask me for it and I will send it to you as an attachment. If you don't understand, no worries, just ignore it and/or ask me to explain it further./)
![](https://seccdn.libravatar.org/avatar/150bb68600b6f4527c14c79e81e90f53.jpg?s=120&d=mm&r=g)
On Tue, 22 Jun 2021 17:38:16 -0700 Marc Chamberlin <marc@marcchamberlin.com> wrote: I feel that your message is probably important, so I tried to understand it but failed :(
Certbot automatically updated some security certificates on one of my systems (Nova)
I started getting lost here. What is Nova? Googling nova certificate told me something about importing cars to the UK, followed by some old chrysler product. So perhaps some more explanation would help. Maybe just breaking the message into more easily followed paragraphs? Or maybe I'm simply the wrong person to understand what you wrote.
![](https://seccdn.libravatar.org/avatar/065b1b1774363493af32c0a6ce9ff07c.jpg?s=120&d=mm&r=g)
On 6/23/21 1:09 AM, Dave Howorth wrote:
On Tue, 22 Jun 2021 17:38:16 -0700 Marc Chamberlin <marc@marcchamberlin.com> wrote:
I feel that your message is probably important, so I tried to understand it but failed :(
Certbot automatically updated some security certificates on one of my systems (Nova) I started getting lost here. What is Nova? OH! Sorry!!! Nova is the name of one of my computers.
-- *"The Truth is out there" - Spooky* *_ _ . . . . . . _ _ . _ _ _ _ . . . . _ . . . . _ _ . _ _ _ . . . . _ _ . _ . . _ . _ _ _ _ . _ . _ . _ . _ . * Computers: the final frontier. These are the voyages of the user Marc. His mission: to explore strange new hardware. To seek out new software and new applications. To boldly go where no Marc has gone before! (/This email is digitally signed. My public key for sending encrypted email to me can be found at - https://keys.openpgp.org/search?q=marc@marcchamberlin.com or just ask me for it and I will send it to you as an attachment. If you don't understand, no worries, just ignore it and/or ask me to explain it further./)
![](https://seccdn.libravatar.org/avatar/7891b1b1a5767f4b9ac1cc0723cebdac.jpg?s=120&d=mm&r=g)
Marc Chamberlin wrote:
On 6/23/21 1:09 AM, Dave Howorth wrote:
On Tue, 22 Jun 2021 17:38:16 -0700 Marc Chamberlin <marc@marcchamberlin.com> wrote:
I feel that your message is probably important, so I tried to understand it but failed :(
Certbot automatically updated some security certificates on one of my systems (Nova) I started getting lost here. What is Nova?
OH! Sorry!!! Nova is the name of one of my computers.
I did get that bit, but I was thoroughly confused about the rest. I have difficulty imagining how certbot updating some certificates should cock up all kinds of services for you. I have certbot renewing customer certs twice a day, works very well for hundreds of them. -- Per Jessen, Zürich (21.4°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland.
![](https://seccdn.libravatar.org/avatar/065b1b1774363493af32c0a6ce9ff07c.jpg?s=120&d=mm&r=g)
On 6/23/21 12:09 PM, Per Jessen wrote:
Marc Chamberlin wrote:
On 6/23/21 1:09 AM, Dave Howorth wrote:
On Tue, 22 Jun 2021 17:38:16 -0700 Marc Chamberlin <marc@marcchamberlin.com> wrote:
I feel that your message is probably important, so I tried to understand it but failed :(
Certbot automatically updated some security certificates on one of my systems (Nova) I started getting lost here. What is Nova? OH! Sorry!!! Nova is the name of one of my computers. I did get that bit, but I was thoroughly confused about the rest. I have difficulty imagining how certbot updating some certificates should cock up all kinds of services for you. I have certbot renewing customer certs twice a day, works very well for hundreds of them.
Per - I am trying to dig into this further also because, for example, I am not sure why the automount service, should even care about certificates. And I will offer to say that this could all be a red herring of some kind and I may have a problem elsewhere. Anywise from what I am observing, let's say Computer A is running some services and it also has some security certificates from LetsEncrpyt that gets automatically updated. There is no problem as far as Computer A is concerned. Now Computer B comes along and has a client that wants to connect to a service on computer A. In order to establish a secure connection to Computer A, Computer B's client presents a cached copy of the certificate that Computer B has for Computer A, to Computer A. But the certificate from Computer B is now out of date and Computer A refuses it. Many, if not most of the clients on Computer B, that want an encrypted connection, such as the ones I previously mentioned in my first email, don't handle this condition very well. Only the SSH client, on Computer B, offers to tell you how to delete the old cached copy and update Computer B's cached copy with the new version of Computer A's new certificate. Other clients just drop the ball and spout off some generic B.S. error message complaining they cannot make the connection to Computer A. "No route to host" or even worse "AUTH method LOGIN failed from network" are a couple examples of such B.S. I have come across. How helpful is that? So my question is, who should have the responsibility for presenting a decent error message, the O.S. itself, or each client/server app? In other words who do I belly ache to? BTW IMHO the error message from the ssh client is pretty decent and I got no complaints with it! Empherical observations have shown a pattern where once I synchronize the cached copy of an outdated certificate with the updated version, the problems I was chasing down went away most of the time. I just thought I would bring this issue up as a topic for conversation, now that I have what appears to be a solution, (check with and use SSH to update cached certificates) this is not a pressing issue for me anymore and I will move on and monitor what happens if anyone want to run with this issue. Marc... -- *"The Truth is out there" - Spooky* *_ _ . . . . . . _ _ . _ _ _ _ . . . . _ . . . . _ _ . _ _ _ . . . . _ _ . _ . . _ . _ _ _ _ . _ . _ . _ . _ . * Computers: the final frontier. These are the voyages of the user Marc. His mission: to explore strange new hardware. To seek out new software and new applications. To boldly go where no Marc has gone before! (/This email is digitally signed. My public key for sending encrypted email to me can be found at - https://keys.openpgp.org/search?q=marc@marcchamberlin.com or just ask me for it and I will send it to you as an attachment. If you don't understand, no worries, just ignore it and/or ask me to explain it further./)
![](https://seccdn.libravatar.org/avatar/7891b1b1a5767f4b9ac1cc0723cebdac.jpg?s=120&d=mm&r=g)
Marc Chamberlin wrote:
Anywise from what I am observing, let's say Computer A is running some services and it also has some security certificates from LetsEncrpyt that gets automatically updated.
Right. A perfectly straight forward setup.
There is no problem as far as Computer A is concerned. Now Computer B comes along and has a client that wants to connect to a service on computer A. In order to establish a secure connection to Computer A, Computer B's client presents a cached copy of the certificate that Computer B has for Computer A, to Computer A. But the certificate from Computer B is now out of date and Computer A refuses it.
"A" is running a service using LE certificates, maybe https - "B" connects and only needs to check if the certificate presented by "A" is valid. "B" typically does not present anything.
So my question is, who should have the responsibility for presenting a decent error message, the O.S. itself, or each client/server app?
The latter. The operating system is not in charge. Example: When your Firefox browser tries to connect to an https website and cannot validate the certificate presented by the server (expired, revoked, wrong name, unsupported TLS version), it is up to the browser to inform you.
In other words who do I belly ache to? BTW IMHO the error message from the ssh client is pretty decent and I got no complaints with it!
This is where I get lost - what has ssh got to with your LE certificates ? -- Per Jessen, Zürich (19.2°C)
![](https://seccdn.libravatar.org/avatar/065b1b1774363493af32c0a6ce9ff07c.jpg?s=120&d=mm&r=g)
Marc Chamberlin wrote:
Anywise from what I am observing, let's say Computer A is running some services and it also has some security certificates from LetsEncrpyt that gets automatically updated. Right. A perfectly straight forward setup.
There is no problem as far as Computer A is concerned. Now Computer B comes along and has a client that wants to connect to a service on computer A. In order to establish a secure connection to Computer A, Computer B's client presents a cached copy of the certificate that Computer B has for Computer A, to Computer A. But the certificate from Computer B is now out of date and Computer A refuses it. "A" is running a service using LE certificates, maybe https - "B" connects and only needs to check if the certificate presented by "A" is valid. "B" typically does not present anything. Per - Again thanks for your reply. I will accept your explanation, mine was only an inference about what is going on when the certificate stored on Computer A is different from the one Computer B has for Computer A. This doesn't really change anything about what I have reported and the behavior I seem to be seeing. The bottom line is that there is a difference, between what is presented and what is cached, and the certificate is rejected, causing the connection to fail. And some clients apparently drop the ball on this problem, don't offer a solution, and simply spit out a useless and/or bogus error message that can only be interpreted as - "I'm sorry, I can't do that". NOT helpful and sometimes the bogus error message is extremely misleading.
So my question is, who should have the responsibility for presenting a decent error message, the O.S. itself, or each client/server app? The latter. The operating system is not in charge.
Example: When your Firefox browser tries to connect to an https website and cannot validate the certificate presented by the server (expired, revoked, wrong name, unsupported TLS version), it is up to the browser to inform you. OK, but a more unified approach to handling this error condition would
On 6/24/21 1:25 AM, Per Jessen wrote: provide a better user experience. Which is why I suggest that perhaps the O.S. should be more involve or a mandatory common library be used to make a secured connection between the client and the server. Right now it appears that the developers of each client app is handling this error condition in different ways, often badly, and I feel this model should be reconsidered.
In other words who do I belly ache to? BTW IMHO the error message from the ssh client is pretty decent and I got no complaints with it! This is where I get lost - what has ssh got to with your LE certificates ?
Ssh is simply another client application (of sshd) and handles the condition of certificate rejection far better that most other types of clients. I like it's handling of a certificate rejection because it actually presents a guide to a solution for the user, i.e. it tells the user on Computer B that he/she has to delete the cached version of the certificate for Computer A, and how to do it. Here is an example - marc@bigbang:~/bin> ssh marc@nova @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is SHA256:85Nz/w0T9weHNksY7etRqmV5Fmmctxt1p8U4cZ4yFuE. Please contact your system administrator. Add correct host key in /home/marc/.ssh/known_hosts to get rid of this message. Offending ECDSA key in /home/marc/.ssh/known_hosts:9 You can use following command to remove the offending key: ssh-keygen -R nova -f /home/marc/.ssh/known_hosts I simply have to follow the directions for removing the offending key and the next time I use ssh I get this - marc@bigbang:~/bin> ssh marc@nova The authenticity of host 'nova (192.168.10.10)' can't be established. ECDSA key fingerprint is SHA256:85Nz/w0T9weHNksY7etRqmV5Fmmctxt1p8U4cZ4yFuE. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 'nova,192.168.10.10' (ECDSA) to the list of known hosts. Password: Last login: Wed Jun 23 21:24:14 2021 from 192.168.10.100 Have a lot of fun... and VIOLA! I have been guided to the solution, by ssh, and am a happy camper on my way! Now if only all client apps wanting to make a secure connection to their server could handle this rejection of a certificate in a similar fashion, instead of presenting crap for an error message, wouldn't that be nice? Which is why I feel that normalizing connection requests, in either the O.S. itself, or in a required common library would be a wonderful OOD solution. Just an idea/thought. My guess is that when Firefox, or Thunderbird want to make a secure connection to their respective servers, when they encounter a mismatched set of certificates, or hold an out of date certificate for their server's host, they simply first assume the user wants to update their cached version of the certificate. So they simply attempt to go ahead and do so with minimal interactions with the user. If necessary, and I have seen Firefox do this, they too will pop up a dialog explaining the problem and guide the user to a solution. Also not a bad solution, IMHO of course. Marc...
-- *"The Truth is out there" - Spooky* *_ _ . . . . . . _ _ . _ _ _ _ . . . . _ . . . . _ _ . _ _ _ . . . . _ _ . _ . . _ . _ _ _ _ . _ . _ . _ . _ . * Computers: the final frontier. These are the voyages of the user Marc. His mission: to explore strange new hardware. To seek out new software and new applications. To boldly go where no Marc has gone before! (/This email is digitally signed. My public key for sending encrypted email to me can be found at - https://keys.openpgp.org/search?q=marc@marcchamberlin.com or just ask me for it and I will send it to you as an attachment. If you don't understand, no worries, just ignore it and/or ask me to explain it further./)
![](https://seccdn.libravatar.org/avatar/7891b1b1a5767f4b9ac1cc0723cebdac.jpg?s=120&d=mm&r=g)
Marc Chamberlin wrote:
On 6/24/21 1:25 AM, Per Jessen wrote:
"A" is running a service using LE certificates, maybe https - "B" connects and only needs to check if the certificate presented by "A" is valid. "B" typically does not present anything.
Per - Again thanks for your reply. I will accept your explanation, mine was only an inference about what is going on when the certificate stored on Computer A is different from the one Computer B has for Computer A.
Hi Marc Yes, but the issue is that B does not have a "certificate for A" stored or cached. A presents a certificate, and B just validates it.
This doesn't really change anything about what I have reported and the behavior I seem to be seeing. The bottom line is that there is a difference, between what is presented and what is cached,
Without more information, it is difficult to say, but I don't see where anything is being cached.
and the certificate is rejected, causing the connection to fail. And some clients apparently drop the ball on this problem, don't offer a solution, and simply spit out a useless and/or bogus error message that can only be interpreted as - "I'm sorry, I can't do that". NOT helpful and sometimes the bogus error message is extremely misleading.
Yes, that is quite likely correct - I have also had to debug such issues in postfix.
Example: When your Firefox browser tries to connect to an https website and cannot validate the certificate presented by the server (expired, revoked, wrong name, unsupported TLS version), it is up to the browser to inform you.
OK, but a more unified approach to handling this error condition would provide a better user experience. Which is why I suggest that perhaps the O.S. should be more involve or a mandatory common library be used to make a secured connection between the client and the server.
It is a common library, it is called libssl, by the OpenSSL project, https://www.openssl.org
Right now it appears that the developers of each client app is handling this error condition in different ways, often badly, and I feel this model should be reconsidered.
Many apps don't even have a user interface as such - postfix, apache, dovecot, openvpn - but there is no reason why apps should not try to be more user friendly.
My guess is that when Firefox, or Thunderbird want to make a secure connection to their respective servers, when they encounter a mismatched set of certificates, or hold an out of date certificate for their server's host,
I think you misunderstand - they simply don't "cache" anything. A mail server or an https server presents them with a certificate. The client (TB or FF) validates it - checks that the name matches, that it is not expired, that it is nort revoked - if all is good, the client permits the connection. If you want to influence development, this is probably more a topic for the factory list or maybe for those individual projects. -- Per Jessen, Zürich (18.1°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland.
![](https://seccdn.libravatar.org/avatar/065b1b1774363493af32c0a6ce9ff07c.jpg?s=120&d=mm&r=g)
On 6/24/21 10:30 AM, Per Jessen wrote:
Marc Chamberlin wrote:
On 6/24/21 1:25 AM, Per Jessen wrote:
"A" is running a service using LE certificates, maybe https - "B" connects and only needs to check if the certificate presented by "A" is valid. "B" typically does not present anything. Per - Again thanks for your reply. I will accept your explanation, mine was only an inference about what is going on when the certificate stored on Computer A is different from the one Computer B has for Computer A. Hi Marc Yes, but the issue is that B does not have a "certificate for A" stored or cached. A presents a certificate, and B just validates it.
This doesn't really change anything about what I have reported and the behavior I seem to be seeing. The bottom line is that there is a difference, between what is presented and what is cached, Without more information, it is difficult to say, but I don't see where anything is being cached. Hi Pers, and thanks again for your responses. OK I will accept your explanation, for the moment, but then I have to ask what is ssh doing? The ssh client clearly understands when a certificate from it's server (sshd) has changed and it will definitely whine about it. To me the only way the ssh client "knows" this is that it must have something to compare the new certificate against. I am going to continue this line of questions below because you have explained something I was unaware of.....
and the certificate is rejected, causing the connection to fail. And some clients apparently drop the ball on this problem, don't offer a solution, and simply spit out a useless and/or bogus error message that can only be interpreted as - "I'm sorry, I can't do that". NOT helpful and sometimes the bogus error message is extremely misleading. Yes, that is quite likely correct - I have also had to debug such issues in postfix.
Example: When your Firefox browser tries to connect to an https website and cannot validate the certificate presented by the server (expired, revoked, wrong name, unsupported TLS version), it is up to the browser to inform you. OK, but a more unified approach to handling this error condition would provide a better user experience. Which is why I suggest that perhaps
the O.S. should be more involve or a mandatory common library be used to make a secured connection between the client and the server. It is a common library, it is called libssl, by the OpenSSL project, https://www.openssl.org Now this makes sense and is exactly what I was asking for, a common library that servers and clients can use to establish a secure connection. So the remaining question is why doesn't it handle the error conditions and do the presentation of the error and solutions on how to handle the errors.
Right now it appears that the developers of each client app is handling this error condition in different ways, often badly, and I feel this model should be reconsidered. Many apps don't even have a user interface as such - postfix, apache, dovecot, openvpn - but there is no reason why apps should not try to be more user friendly. These are not client apps, rather these are services running on a server. I would not expect these to have a user interface, but I would expect client apps connecting to these services to have a user interface, or a way to communicate to a user when things go wrong, and supply the user with a way to fix the sort of problem like this one, a connection failure due to having a cached version of an invalid/out of date certificate.
My guess is that when Firefox, or Thunderbird want to make a secure connection to their respective servers, when they encounter a mismatched set of certificates, or hold an out of date certificate for their server's host, I think you misunderstand - they simply don't "cache" anything. A mail server or an https server presents them with a certificate. The client (TB or FF) validates it - checks that the name matches, that it is not expired, that it is nort revoked - if all is good, the client permits the connection. I disagree, while TB and FF don't cache certificates directly (I am not sure about this) I strongly suspect that they use libssl to establish a secure connection. And from the behavior of the ssh client, I strongly suspect that libssl IS caching certificates from servers that it was requested to make a secure connection to, in the past.
So I am going to hypothesize that automount/autofs, for example, is using libssl to establish a secure connection, which would explain why automount starts working after I use ssh to update the "cached copy" of the certificate for the host that the sshd service is running on. Automount is interesting in that I guess/suspect it is both a client and a service. As a client it can connect to file services, such as NFS or NMB/SMB etc., running on other machines, and uses libssl to create a secure connection that allows the transfer of data to/from mount points, and vice versa. And as a client it badly handles a connection error caused by invalid/out of date certificates, and just reports a bad generic error message in the system log files.
If you want to influence development, this is probably more a topic for the factory list or maybe for those individual projects.
Which brings us back to my original question, who SHOULD have the responsibility for reporting connection errors caused by an invalid/out of date certificate? I guess I can ask on the factory list also, assuming I can join that list. Marc....
-- *"The Truth is out there" - Spooky* *_ _ . . . . . . _ _ . _ _ _ _ . . . . _ . . . . _ _ . _ _ _ . . . . _ _ . _ . . _ . _ _ _ _ . _ . _ . _ . _ . * Computers: the final frontier. These are the voyages of the user Marc. His mission: to explore strange new hardware. To seek out new software and new applications. To boldly go where no Marc has gone before! (/This email is digitally signed. My public key for sending encrypted email to me can be found at - https://keys.openpgp.org/search?q=marc@marcchamberlin.com or just ask me for it and I will send it to you as an attachment. If you don't understand, no worries, just ignore it and/or ask me to explain it further./)
participants (3)
-
Dave Howorth
-
Marc Chamberlin
-
Per Jessen