[opensuse-factory] Leap 15.2 unusable due to SSL hell - Chromium, Firefox, Thunderbird broken
Hi, I've reported issues in SSL handling before which cause tools to hang and cause port 443 RSTs in the tcpdump. Since 2017 I maintain a custom osc SSL patch/fix package. Leap 15.0 didn't even install without a libcurl fix for zypper. But my patches fixing SSL handling in those tools haven't been accepted although they are correct. After the upgrade to Leap 15.2, I had so unfortunate timing with my fast LAN/cable modem based internet connection that Firefox and Chromium couldn't load any tabs any more (all of them hanging in SSL handling). I've removed the ad blocker and Firefox didn't even start any more. I've executed "firefox -d gdb" and noticed that it crashes with a SEGFAULT in SSL handling at startup. Then I wanted to write this email and I noticed that Thunderbird was not able to download any email and the email window where I started writing crashed. My employer forces me to use Chromium/Chrome due to the need for a special plugin. So I've installed the Epiphany browser to download latest Chrome. That one is really fast in loading https websites. I tested latest Chrome and that one is also not able to load any HTTPS website tab. Also RSTs in the tcpdump. So Leap 15.2 is unusable for me. I've rolled back to the full disk backup before upgrading. I cannot fix SSL handling in three complex tools at once all by myself. For me it looks like I have to reinstall a distro as a workaround and I would choose Ubuntu as I can easily skip a faulty release there and can stay longer on a proper one. Using wireless connections with unpredictable high latency and harmful pulsed microwave radiation as a workaround is no option for me. Any chance to extend Leap 15.1 support until 15.3 release? IMHO an SSL stability initiative is required - even independent of vendors. If anybody else noticed slow browser tabs, slow https file downloads, slow email downloads, or plain hanging, then I'd be glad to team up to join forces against SSL network state machine violations. Cheers, Sebastian -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
[sorry, first mail was only sent as personal reply :-(] Hi Sebastian, On 19.10.20 11:45, Sebastian Parschauer wrote:
IMHO an SSL stability initiative is required - even independent of vendors. If anybody else noticed slow browser tabs, slow https file downloads, slow email downloads, or plain hanging, then I'd be glad to team up to join forces against SSL network state machine violations. I have read your message but not understood a single word.
Would it be possible to describe your problems, preferably in bugreports, so that mere mortals are able to understand what you are trying to tell? Almost needless to say, that SSL works fine for me in Leap 15.2 with Chromium, Firefox, Thunderbird and curl. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 19.10.20 11:51, Stefan Seyfried wrote:
[sorry, first mail was only sent as personal reply :-(]
Hi Sebastian,
On 19.10.20 11:45, Sebastian Parschauer wrote:
IMHO an SSL stability initiative is required - even independent of vendors. If anybody else noticed slow browser tabs, slow https file downloads, slow email downloads, or plain hanging, then I'd be glad to team up to join forces against SSL network state machine violations. I have read your message but not understood a single word.
Would it be possible to describe your problems, preferably in bugreports, so that mere mortals are able to understand what you are trying to tell?
To say it in simple words: SSL can lead to hanging forever if 1) A connection is tried to be reestablished although the old one is still open. The server will react with connection reset for the new connection. The client will repeat its client hello and wait for the server hello which will never be received. 2) A connection is tried to be established which is directly closed by previous connection handling in the client again. The new connection can never be established (duplicate SSL_shutdown() calls - first one closes old connection, second one closes new connection about to be established). One of these variants is hitting me with Firefox, Chromium, and Thunderbird as well now. I don't have the time right now to debug this all by myself to the code part that causes the issues. Firefox is coming with an own SSL library. From previous bug handling I know that I would be on my own as nobody sees priority in this. If you would tcpdump those tools on your side, then you would also see port 443 RSTs for sure which indicate that something is wrong with SSL handling. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi, On Mon, Oct 19, 2020 at 11:45:51AM +0200, Sebastian Parschauer wrote:
Hi,
I've reported issues in SSL handling before which cause tools to hang and cause port 443 RSTs in the tcpdump. Since 2017 I maintain a custom osc SSL patch/fix package. Leap 15.0 didn't even install without a libcurl fix for zypper. But my patches fixing SSL handling in those tools haven't been accepted although they are correct.
After the upgrade to Leap 15.2, I had so unfortunate timing with my fast LAN/cable modem based internet connection that Firefox and Chromium couldn't load any tabs any more (all of them hanging in SSL handling). I've removed the ad blocker and Firefox didn't even start any more. I've executed "firefox -d gdb" and noticed that it crashes with a SEGFAULT in SSL handling at startup. Then I wanted to write this email and I noticed that Thunderbird was not able to download any email and the email window where I started writing crashed.
My employer forces me to use Chromium/Chrome due to the need for a special plugin. So I've installed the Epiphany browser to download latest Chrome. That one is really fast in loading https websites. I tested latest Chrome and that one is also not able to load any HTTPS website tab. Also RSTs in the tcpdump. So Leap 15.2 is unusable for me. I've rolled back to the full disk backup before upgrading.
I cannot fix SSL handling in three complex tools at once all by myself. For me it looks like I have to reinstall a distro as a workaround and I would choose Ubuntu as I can easily skip a faulty release there and can stay longer on a proper one. Using wireless connections with unpredictable high latency and harmful pulsed microwave radiation as a workaround is no option for me.
Any chance to extend Leap 15.1 support until 15.3 release?
IMHO an SSL stability initiative is required - even independent of vendors. If anybody else noticed slow browser tabs, slow https file downloads, slow email downloads, or plain hanging, then I'd be glad to team up to join forces against SSL network state machine violations.
To be very frank, what you are describing sounds like your employeer or Internet provider deploys a SSL man-in-the-middle proxy which is not fully standards compliant and causes troubles. Is this assumption correct? Ciao, Marcus -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 19.10.20 12:01, Marcus Meissner wrote:
IMHO an SSL stability initiative is required - even independent of vendors. If anybody else noticed slow browser tabs, slow https file downloads, slow email downloads, or plain hanging, then I'd be glad to team up to join forces against SSL network state machine violations.
To be very frank, what you are describing sounds like your employeer or Internet provider deploys a SSL man-in-the-middle proxy which is not fully standards compliant and causes troubles.
Is this assumption correct?
No, it isn't. Then SUSE would do that as well as issues started when I still worked at SUSE. Working from home in Berlin, just regular Vodafone cable modem connection. Tools doing proper SSL handling like e.g. wget work fine. Have you ever tcpdumped Firefox, Thunderbird, or Chromium and never seen port 443 RSTs? I was even able to reproduce them on SUSE test servers for osc and Firefox. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Montag, 19. Oktober 2020, 12:27:31 CEST schrieb Sebastian Parschauer:
On 19.10.20 12:01, Marcus Meissner wrote:
IMHO an SSL stability initiative is required - even independent of vendors. If anybody else noticed slow browser tabs, slow https file downloads, slow email downloads, or plain hanging, then I'd be glad to team up to join forces against SSL network state machine violations.
I cannot confirm such behavior in general, but I don't have any Leaps with GUIs left.. My Leap 15.2 servers perform a lot of SSL connections in all kinds of different scenarios without troubles.
To be very frank, what you are describing sounds like your employeer or Internet provider deploys a SSL man-in-the-middle proxy which is not fully standards compliant and causes troubles.
Is this assumption correct?
No, it isn't. Then SUSE would do that as well as issues started when I still worked at SUSE. Working from home in Berlin, just regular Vodafone cable modem connection. Tools doing proper SSL handling like e.g. wget work fine.
Now, this is interesting. I've seen tcpdumps, that have shown "likely" behavior of transparent HTTP/SSL proxies for this very provider as well. Since this is the only way for a decent connection to the world here, I've resisted to hang this on the big bell (and the related "careful" support request remained unanswered..). Hmm. I don't trust anything behind my router cascade anyway. Marcus might have hit the nail here. Sebastian, could you do us the flavor and install TW on a spare partition. TW comes with everything current in this respect, and gives you and us another data point with current OpenSSL(s), Chrome, FF... FYI, I've built Python 3.8 with openssl-1.1.1h for 15.1 and 15.2 here: https://build.opensuse.org/project/show/home:frispete:python The new openssl handling made this very easy. This repo is experimental, but might help you to get yet another data point. Cheers, Pete -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Montag, 19. Oktober 2020, 15:03:34 CEST schrieb Hans-Peter Jansen:
Am Montag, 19. Oktober 2020, 12:27:31 CEST schrieb Sebastian Parschauer:
On 19.10.20 12:01, Marcus Meissner wrote:
IMHO an SSL stability initiative is required - even independent of vendors. If anybody else noticed slow browser tabs, slow https file downloads, slow email downloads, or plain hanging, then I'd be glad to team up to join forces against SSL network state machine violations.
I cannot confirm such behavior in general, but I don't have any Leaps with GUIs left.. My Leap 15.2 servers perform a lot of SSL connections in all kinds of different scenarios without troubles.
To be very frank, what you are describing sounds like your employeer or Internet provider deploys a SSL man-in-the-middle proxy which is not fully standards compliant and causes troubles.
Is this assumption correct?
No, it isn't. Then SUSE would do that as well as issues started when I still worked at SUSE. Working from home in Berlin, just regular Vodafone cable modem connection. Tools doing proper SSL handling like e.g. wget work fine.
Now, this is interesting.
I've seen tcpdumps, that have shown "likely" behavior of transparent HTTP/SSL proxies for this very provider as well. Since this is the only way for a decent connection to the world here, I've resisted to hang this on the big bell (and the related "careful" support request remained unanswered..). Hmm. I don't trust anything behind my router cascade anyway.
I have some connections to AK Vorrat, should we sort of get ready to get them involved, and maybe the local privacy officer (or whatever "Datenschutzbeauftragter" is in english)? Or just right away Patrick Breyer? Cheers Mathias and yes i'm actually serious about this. If we can prove that Vodafone is running a man in the middle against SSL.... -- Mathias Homann Mathias.Homann@openSUSE.org OBS: lemmy04 Jabber (XMPP): lemmy@tuxonline.tech IRC: [Lemmy] on freenode and ircnet (bouncer active) telegram: https://telegram.me/lemmy98 keybase: https://keybase.io/lemmy gpg key fingerprint: 8029 2240 F4DD 7776 E7D2 C042 6B8E 029E 13F2 C102
On 19.10.20 15:24, Mathias Homann wrote:
Am Montag, 19. Oktober 2020, 15:03:34 CEST schrieb Hans-Peter Jansen:
Am Montag, 19. Oktober 2020, 12:27:31 CEST schrieb Sebastian Parschauer:
On 19.10.20 12:01, Marcus Meissner wrote:
IMHO an SSL stability initiative is required - even independent of vendors. If anybody else noticed slow browser tabs, slow https file downloads, slow email downloads, or plain hanging, then I'd be glad to team up to join forces against SSL network state machine violations.
I cannot confirm such behavior in general, but I don't have any Leaps with GUIs left.. My Leap 15.2 servers perform a lot of SSL connections in all kinds of different scenarios without troubles.
To be very frank, what you are describing sounds like your employeer or Internet provider deploys a SSL man-in-the-middle proxy which is not fully standards compliant and causes troubles.
Is this assumption correct?
No, it isn't. Then SUSE would do that as well as issues started when I still worked at SUSE. Working from home in Berlin, just regular Vodafone cable modem connection. Tools doing proper SSL handling like e.g. wget work fine.
Now, this is interesting.
I've seen tcpdumps, that have shown "likely" behavior of transparent HTTP/SSL proxies for this very provider as well. Since this is the only way for a decent connection to the world here, I've resisted to hang this on the big bell (and the related "careful" support request remained unanswered..). Hmm. I don't trust anything behind my router cascade anyway.
I have some connections to AK Vorrat, should we sort of get ready to get them involved, and maybe the local privacy officer (or whatever "Datenschutzbeauftragter" is in english)? Or just right away Patrick Breyer?
Cheers Mathias
and yes i'm actually serious about this. If we can prove that Vodafone is running a man in the middle against SSL....
Thanks, now I get it. So the fact that all browser tabs are affected at once although there should be different latencies to different servers, indicates that there is an SSL man-in-the-middle proxy at Vodafone. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Oct 19, 2020 at 12:01 PM Sebastian Parschauer <s.parschauer@gmx.de> wrote:
I have some connections to AK Vorrat, should we sort of get ready to get them w I get it. So the fact that all browser tabs are affected at once although there should be different latencies to different servers, indicates that there is an SSL man-in-the-middle proxy at Vodafone.
We are trying to tell you something is fishy in the network or equipment.,this does not precludes the existence of buggy SSL library users (I have seen suspicious or obviously buggy code already, you do not need to convince me of that) I would suspect all CPE first, sometimes there are configured unknown to you as multi function devices.. if vodaphone really runs a MITM proxy you have a news-coverage-worthy problem. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 19/10/2020 17.18, Cristian Rodríguez wrote:
On Mon, Oct 19, 2020 at 12:01 PM Sebastian Parschauer <s.parschauer@gmx.de> wrote:
I have some connections to AK Vorrat, should we sort of get ready to get them now I get it. So the fact that all browser tabs are affected at once although there should be different latencies to different servers, indicates that there is an SSL man-in-the-middle proxy at Vodafone.
We are trying to tell you something is fishy in the network or equipment.,this does not precludes the existence of buggy SSL library users (I have seen suspicious or obviously buggy code already, you do not need to convince me of that) I would suspect all CPE first, sometimes there are configured unknown to you as multi function devices.. if vodaphone really runs a MITM proxy you have a news-coverage-worthy problem.
Could it be some sort of a "proxy cache" as Telefónica (Spain) implemented years ago (for plain http), then they had to dismantle because of the complains? The purpose at the time was simply to save bandwidth for them, but it resulted in disaster. Webs would complain of too many connections coming from the same IP and block that IP, thus blocking all clients. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
On Monday 2020-10-19 17:01, Sebastian Parschauer wrote:
Thanks, now I get it. So the fact that all browser tabs are affected at once although there should be different latencies to different servers, indicates that there is an SSL man-in-the-middle proxy at Vodafone.
For an affected website, could you look at the certificate chain? » openssl s_client -connect www.picksomething.com:443 [...] Certificate chain 0 s:C = US, ST = California, L = Menlo Park, O = "Facebook, Inc.", CN = *.facebook.com i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA 1 s:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
19.10.2020 16:24, Mathias Homann пишет:
and yes i'm actually serious about this. If we can prove that Vodafone is running a man in the middle against SSL....
Is it possible without terminating SSL stream? In which case it has to generate certificate on the fly signed by CA that is trusted by client. I would start with checking server certificates in browser session.
On Monday 2020-10-19 11:45, Sebastian Parschauer wrote:
I've reported issues in SSL handling before which cause tools to hang and cause port 443 RSTs in the tcpdump. Since 2017 I maintain a custom osc SSL patch/fix package.
And this patch does what? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 19.10.20 12:09, Jan Engelhardt wrote:
On Monday 2020-10-19 11:45, Sebastian Parschauer wrote:
I've reported issues in SSL handling before which cause tools to hang and cause port 443 RSTs in the tcpdump. Since 2017 I maintain a custom osc SSL patch/fix package.
And this patch does what?
osc: Closing the SSL connection properly. See https://bugzilla.suse.com/show_bug.cgi?id=1068470 attachment osc-ssl-fix-hang-at-server-hello.patch: + def close(self): + self.sock.close() + So basically just adding the missing close() method to call SSL_shutdown(). The method is called but not implemented. You see RSTs in the tcpdump without this patch. The libcurl fix removes two superfluous SSL_shutdown() calls as it tries to close the connection three times. Not sure if this is still an issue as I only use http repos and wget instead of curl as a workaround. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Oct 19, 2020 at 7:20 AM Sebastian Parschauer <s.parschauer@gmx.de> wrote:
The libcurl fix removes two superfluous SSL_shutdown() calls as it tries to close the connection three times. Not sure if this is still an issue as I only use http repos and wget instead of curl as a workaroun
Right off the bat, let me say your analysis is correct and this is one of the common misuses of all the crypto libraries, most commonly found with openssl. Now ..I have SEEN this bugs in code..several times.. but NEVER practically experienced any ill effects. do you have anything special in your network or kernel configuration that makes you experience practical ill effects ? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 19.10.20 14:50, Cristian Rodríguez wrote:
On Mon, Oct 19, 2020 at 7:20 AM Sebastian Parschauer <s.parschauer@gmx.de> wrote:
The libcurl fix removes two superfluous SSL_shutdown() calls as it tries to close the connection three times. Not sure if this is still an issue as I only use http repos and wget instead of curl as a workaroun
Right off the bat, let me say your analysis is correct and this is one of the common misuses of all the crypto libraries, most commonly found with openssl. Now ..I have SEEN this bugs in code..several times.. but NEVER practically experienced any ill effects. do you have anything special in your network or kernel configuration that makes you experience practical ill effects ?
Thanks for confirming that I'm not crazy. ;-) I have a vanilla Leap distro - only one SSL fix package for osc. Kernel boot options "usbcore.autosuspend=-1 nosmt=force". USB autosuspend suspended an external USB HDD with mounted volumes. So I rather keep that one off. "nosmt=force" is a precaution due to Intel CPU design flaws. Got a Dell Latitude 5491 with the Dell WD15 USB-C dock. 1.5m CAT5e LAN cable to a Netgear router and from there to cable modem 0.5m CAT5e cable. I had similar issues with a Lenovo Thinkpad X250. It is rather the network setup than the CPU/kernel setup. Maybe I need a 100m LAN cable to modify latencies. ;-) Cheers, Sebastian -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 19.10.20 14:50, Cristian Rodríguez wrote:
On Mon, Oct 19, 2020 at 7:20 AM Sebastian Parschauer
Got a Dell Latitude 5491 with the Dell WD15 USB-C dock. 1.5m CAT5e LAN cable to a Netgear router
So..what's this netgear router doing (other than routing?) is its firmware up2date ? is it doing any sort of web filtering or is web filtering enabled but not filtering anything ? the behaviour you see is clearly as day as one of a man in middle, whatever the SSL code bugs might be, it should NOT behave like this. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 19.10.20 15:59, Cristian Rodríguez wrote:
Got a Dell Latitude 5491 with the Dell WD15 USB-C dock. 1.5m CAT5e LAN cable to a Netgear router
So..what's this netgear router doing (other than routing?) is its firmware up2date ? is it doing any sort of web filtering or is web filtering enabled but not filtering anything ? the behaviour you see is clearly as day as one of a man in middle, whatever the SSL code bugs might be, it should NOT behave like this.
Firmware is up2date. It uses MAC security. So unknown MAC addresses are blocked. For the osc bug I shortly connected my laptop with the SUSE firewall directly to cable modem and the issue persisted. The client retransmitted the client hello all the time and was waiting for the server hello which never came. And a lot more errors were visible like "TCP ACKed unseen segment". There is no man in middle issue. The tools just get stuck in the SSL network state machine. There are no atomic operations with Ethernet. Cheers, Sebastian -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Montag, 19. Oktober 2020, 16:13:15 CEST schrieb Sebastian Parschauer:
On 19.10.20 15:59, Cristian Rodríguez wrote:
Got a Dell Latitude 5491 with the Dell WD15 USB-C dock. 1.5m CAT5e LAN cable to a Netgear router
So..what's this netgear router doing (other than routing?) is its firmware up2date ?
is it doing any sort of web filtering or is web filtering enabled but
not filtering anything ? the behaviour you see is clearly as day as one of a man in middle, whatever the SSL code bugs might be, it should NOT behave like this.
Firmware is up2date. It uses MAC security. So unknown MAC addresses are blocked. For the osc bug I shortly connected my laptop with the SUSE firewall directly to cable modem and the issue persisted.
Is there any chance to take that laptop somewhere where you'll definitely NOT be on a vodafone connection, and see if the problem persists? And if it does NOT we should definitely talk to a lot of people. Cheers Mathias -- Mathias Homann Mathias.Homann@openSUSE.org Jabber (XMPP): lemmy@tuxonline.tech IRC: [Lemmy] on freenode and ircnet (bouncer active) telegram: https://telegram.me/lemmy98 keybase: https://keybase.io/lemmy gpg key fingerprint: 8029 2240 F4DD 7776 E7D2 C042 6B8E 029E 13F2 C102 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 10/19/20 8:32 AM, Mathias Homann wrote:
Is there any chance to take that laptop somewhere where you'll definitely NOT be on a vodafone connection, and see if the problem persists?
And if it does NOT we should definitely talk to a lot of people.
I believe that MITM interference can be detected by looking at a web site's certificate fingerprint. This link explains: https://www.grc.com/fingerprints.htm For example, if you visit that site, you can confirm that it's cert fingerprint is 7A:85:1C:F0:F6:9F:D0:CC:EA:EA:9A:88:01:96:BF:79:8C:E1:A8:33 If it's not, you're looking at that site through a MITM. Regards, Lew -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 19/10/2020 18.34, Lew Wolfgang wrote:
I believe that MITM interference can be detected by looking at a web site's certificate fingerprint. This link explains:
https://www.grc.com/fingerprints.htm
For example, if you visit that site, you can confirm that it's cert fingerprint is 7A:85:1C:F0:F6:9F:D0:CC:EA:EA:9A:88:01:96:BF:79:8C:E1:A8:33
If it's not, you're looking at that site through a MITM.
Thanks, I did not know this. To check this, go to a page in the list, say <https://www.facebook.com/>, which has fingerprint "14:54:7C:59:19:45:DD:42:40:C2:F6:5E:AC:A1:17:B7:20:F9:C4:38". Right click (firefox) on empty area of the page, to get "Page Info". Click on tab "security". Click on view certificate. Searching for "C4:38" should find the string - I don't. The SHA-1 I get is "D9:8F:D8:BB:5D:98:AA:06:03:50:50:AC:07:82:6C:2B:D0:1C:EB:9A" paypal - fail. grc.com - matches. Huh? Another method: click on the keylock at the left of the address bar. A box with information appears, and you can find the certificate information there as well. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
19.10.2020 20:07, Carlos E. R. пишет:
On 19/10/2020 18.34, Lew Wolfgang wrote:
I believe that MITM interference can be detected by looking at a web site's certificate fingerprint. This link explains:
https://www.grc.com/fingerprints.htm
For example, if you visit that site, you can confirm that it's cert fingerprint is 7A:85:1C:F0:F6:9F:D0:CC:EA:EA:9A:88:01:96:BF:79:8C:E1:A8:33
If it's not, you're looking at that site through a MITM.
Thanks, I did not know this.
To check this, go to a page in the list, say <https://www.facebook.com/>, which has fingerprint "14:54:7C:59:19:45:DD:42:40:C2:F6:5E:AC:A1:17:B7:20:F9:C4:38".
Right click (firefox) on empty area of the page, to get "Page Info". Click on tab "security". Click on view certificate. Searching for "C4:38" should find the string - I don't. The SHA-1 I get is "D9:8F:D8:BB:5D:98:AA:06:03:50:50:AC:07:82:6C:2B:D0:1C:EB:9A"
99.999% of information in Internet is valid only under very specific boundary conditions which are never described, mostly because authors are not aware of them themselves. Look at validity period of Facebook certificate. It was generated a week ago. I doubt the page in question follows every certificate refresh (if any). And even if page is updated now, certificate will be renewed in less than three months.
paypal - fail.
It is valid certificate which was issued to www.paypal.com. Who says only one single valid certificate may exist at any given time? And who knows how specific site selects certificate to use for connection request?
grc.com - matches.
Huh?
Another method: click on the keylock at the left of the address bar. A box with information appears, and you can find the certificate information there as well.
On 10/19/20 11:27 AM, Andrei Borzenkov wrote:
19.10.2020 20:07, Carlos E. R. пишет:
On 19/10/2020 18.34, Lew Wolfgang wrote:
I believe that MITM interference can be detected by looking at a web site's certificate fingerprint. This link explains:
https://www.grc.com/fingerprints.htm
For example, if you visit that site, you can confirm that it's cert fingerprint is 7A:85:1C:F0:F6:9F:D0:CC:EA:EA:9A:88:01:96:BF:79:8C:E1:A8:33
If it's not, you're looking at that site through a MITM. Thanks, I did not know this.
To check this, go to a page in the list, say <https://www.facebook.com/>, which has fingerprint "14:54:7C:59:19:45:DD:42:40:C2:F6:5E:AC:A1:17:B7:20:F9:C4:38".
Right click (firefox) on empty area of the page, to get "Page Info". Click on tab "security". Click on view certificate. Searching for "C4:38" should find the string - I don't. The SHA-1 I get is "D9:8F:D8:BB:5D:98:AA:06:03:50:50:AC:07:82:6C:2B:D0:1C:EB:9A"
99.999% of information in Internet is valid only under very specific boundary conditions which are never described, mostly because authors are not aware of them themselves.
Look at validity period of Facebook certificate. It was generated a week ago. I doubt the page in question follows every certificate refresh (if any). And even if page is updated now, certificate will be renewed in less than three months.
The GRC site is dynamic, it obtains the current fingerprint when accessed.
paypal - fail.
It is valid certificate which was issued to www.paypal.com. Who says only one single valid certificate may exist at any given time? And who knows how specific site selects certificate to use for connection request?
I think you're correct here, Andrei. I checked others on the GRC list from a couple of different networks and found three mismatches. facebook.com, blogger.com, and yahoo.com. Maybe it can be safely said that "if" a fingerprint matches there's no MITM, but if they don't match, something else "might" be going on that's not necessarily evil. Regards, Lew -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
19.10.2020 21:39, Lew Wolfgang пишет:
On 10/19/20 11:27 AM, Andrei Borzenkov wrote:
19.10.2020 20:07, Carlos E. R. пишет:
On 19/10/2020 18.34, Lew Wolfgang wrote:
I believe that MITM interference can be detected by looking at a web site's certificate fingerprint. This link explains:
https://www.grc.com/fingerprints.htm
For example, if you visit that site, you can confirm that it's cert fingerprint is 7A:85:1C:F0:F6:9F:D0:CC:EA:EA:9A:88:01:96:BF:79:8C:E1:A8:33
If it's not, you're looking at that site through a MITM. Thanks, I did not know this.
To check this, go to a page in the list, say <https://www.facebook.com/>, which has fingerprint "14:54:7C:59:19:45:DD:42:40:C2:F6:5E:AC:A1:17:B7:20:F9:C4:38".
Right click (firefox) on empty area of the page, to get "Page Info". Click on tab "security". Click on view certificate. Searching for "C4:38" should find the string - I don't. The SHA-1 I get is "D9:8F:D8:BB:5D:98:AA:06:03:50:50:AC:07:82:6C:2B:D0:1C:EB:9A"
99.999% of information in Internet is valid only under very specific boundary conditions which are never described, mostly because authors are not aware of them themselves.
Look at validity period of Facebook certificate. It was generated a week ago. I doubt the page in question follows every certificate refresh (if any). And even if page is updated now, certificate will be renewed in less than three months.
The GRC site is dynamic, it obtains the current fingerprint when accessed.
What makes you trust some random site to provide correct information? And this site does not even define what "Authentic Fingerprint" is or how it is computed. Assuming it is SHA-1 *certificate* fingerprint - certificate transparency logs have facebook certificates since 2014. The SHA-1 fingerprint listed on this site is not found by searching these lists. I'd rather believe that this site is wrong.
paypal - fail.
It is valid certificate which was issued to www.paypal.com. Who says only one single valid certificate may exist at any given time? And who knows how specific site selects certificate to use for connection request?
I think you're correct here, Andrei. I checked others on the GRC list from a couple of different networks and found three mismatches. facebook.com, blogger.com, and yahoo.com. Maybe it can be safely said that "if" a fingerprint matches there's no MITM, but if they don't match, something else "might" be going on that's not necessarily evil.
facebook, paypal, blogger issue new certificate almost every day (again, as long as I can believe certificate transparency lists). Certificates on this site seem newer than certificates I see when connecting from my location, but that's all. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
[line breaks disabled intentionally] Am Montag, 19. Oktober 2020, 18:34:17 CEST schrieb Lew Wolfgang:
On 10/19/20 8:32 AM, Mathias Homann wrote:
Is there any chance to take that laptop somewhere where you'll definitely NOT be on a vodafone connection, and see if the problem persists?
And if it does NOT we should definitely talk to a lot of people.
I believe that MITM interference can be detected by looking at a web site's certificate fingerprint. This link explains:
https://www.grc.com/fingerprints.htm
For example, if you visit that site, you can confirm that it's cert fingerprint is 7A:85:1C:F0:F6:9F:D0:CC:EA:EA:9A:88:01:96:BF:79:8C:E1:A8:33
If it's not, you're looking at that site through a MITM.
All tests are done with Firefox 81.0.1 as of TW 20201014. Always check, if the DNS-Name in the certificates match. Note, Firefox is *not* able to save these html certificate pages, you need to use the store links on that page. Vodafone Cable: SHA1 fingerprints for www.grc.com, www.linkedin.com and wordpress.com match, all other deviate: https://www.facebook.com: D9:8F:D8:BB:5D:98:AA:06:03:50:50:AC:07:82:6C:2B:D0:1C:EB:9A https://www.paypal.com: BD:DE:B3:95:6B:86:79:B8:27:86:DA:4D:75:3C:53:AA:04:1E:08:92 https://twitter.com: 86:CD:B1:CB:95:44:0C:F6:81:AB:C8:B2:29:F3:63:F8:9D:F9:11:89 https://www.blogger.com: 34:80:AE:BE:88:F3:37:CC:53:D5:99:F7:9E:F3:8A:A0:E0:A7:CA:28 https://de.yahoo.com: 5A:8F:B8:BA:9B:59:AD:84:72:28:54:70:74:82:A3:32:67:C8:53:FB Yahoo always redirects to de. Now, I repeated these tests with a Telekom Mobile connection: https://www.facebook.com: [same as above] D9:8F:D8:BB:5D:98:AA:06:03:50:50:AC:07:82:6C:2B:D0:1C:EB:9A https://www.paypal.com: [correct now] BC:5C:03:64:3B:FA:1A:5E:A8:51:9B:8E:7E:2D:7D:5E:30:AB:EA:30 https://twitter.com: [same as above] 86:CD:B1:CB:95:44:0C:F6:81:AB:C8:B2:29:F3:63:F8:9D:F9:11:89 https://www.blogger.com: [same as above] 34:80:AE:BE:88:F3:37:CC:53:D5:99:F7:9E:F3:8A:A0:E0:A7:CA:28 https://de.yahoo.com: [same as above] 5A:8F:B8:BA:9B:59:AD:84:72:28:54:70:74:82:A3:32:67:C8:53:FB Excluding the special paypal case, and given, that the two biggest provider in Germany are not totally penetrated, grc.com fingerprints seems wrong in some cases. The inclined reader should perhaps check this h{im,er}self. Here's the cert diff of paypal: --- certs@vodafone/www-paypal-com.txt 2020-10-19 22:05:30.441182094 +0200 +++ certs@telekom/www-paypal-com.txt 2020-10-19 22:05:43.158802852 +0200 @@ -2,44 +2,44 @@ Certificate: Data: Version: 3 (0x2) Serial Number: - 07:41:da:c6:19:b9:7b:b7:28:9a:a5:94:ce:26:c0:cd + 0e:c3:4e:77:02:57:00:4f:ad:cc:f4:a2:f5:19:d6:0c Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA Validity - Not Before: Mar 10 00:00:00 2020 GMT - Not After : Mar 15 12:00:00 2022 GMT + Not Before: Jan 9 00:00:00 2020 GMT + Not After : Jan 12 12:00:00 2022 GMT Subject: businessCategory = Private Organization, jurisdictionC = US, jurisdictionST = Delaware, serialNumber = 3014267, C = US, ST = California, L = San Jose, O = "PayPal, Inc.", OU = CDN Support, CN = www.paypal.com Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: - 00:cd:5f:3d:cd:ba:01:94:28:80:a0:e6:2f:22:ec: - 4d:a3:31:e9:41:81:91:ac:a2:36:7f:72:47:b8:92: - f8:c1:67:3e:bf:cd:22:d6:3b:d6:82:59:90:9c:8e: - e0:06:b0:3d:7a:80:79:a9:23:07:53:7f:9f:de:e6: - d7:ab:03:b9:b5:e4:a7:e5:46:1a:b1:e9:99:c3:8c: - 0f:a5:f5:39:c3:98:af:89:29:fb:c6:e0:ca:ef:8f: - 31:f4:84:57:bb:a9:32:08:c4:ff:36:72:cb:00:4f: - f0:b9:56:8d:9d:32:bc:3d:91:b9:d1:42:8d:89:c3: - 77:47:58:ba:9e:80:16:a6:13:49:7b:df:2c:b4:ee: - 02:8c:3d:58:5f:aa:94:90:22:03:d9:04:4d:69:ec: - 11:fe:ad:c6:3d:97:17:c9:81:a9:a2:f6:9b:c7:82: - 41:08:07:e8:ca:bc:17:ec:c3:12:6a:c8:57:fc:77: - 1a:a5:93:b7:c1:bb:05:fa:d3:b7:18:4d:59:22:e4: - f1:9b:74:07:60:3b:c3:4e:45:1b:3e:d7:ed:ed:2e: - 60:82:fe:56:b7:d5:14:60:2d:cf:89:e2:ed:24:50: - 44:0f:7c:c2:88:58:d6:36:ff:7b:1f:fa:33:65:d9: - 56:0b:6e:c6:1d:29:df:87:83:85:a8:dc:a7:3a:d6: - b8:93 + 00:ab:9a:c4:47:e6:4f:4b:f0:7a:29:6f:9a:1d:6b: + 4e:d5:04:0e:b2:02:ed:8d:d1:a9:ae:d5:da:20:8c: + 7e:8a:49:1a:3c:09:13:f7:72:ee:2e:40:e0:29:41: + 02:78:97:55:f8:06:0d:7b:2a:e3:e0:b3:e5:64:f2: + de:b8:b8:35:e1:c5:7c:eb:12:e3:68:47:74:6c:bc: + 04:25:33:09:17:28:e0:c9:3a:b2:4e:65:50:d0:4a: + e4:3b:b2:e1:2e:82:45:cb:52:05:3b:a4:b7:37:eb: + c8:29:fc:43:67:cc:66:a9:e5:9f:22:1b:1b:f7:86: + 36:35:9b:45:f5:0f:6c:3d:1d:15:55:5d:fe:ca:7d: + 5c:ef:1d:76:b7:f0:59:85:89:1a:c9:d2:bf:58:bc: + 26:9c:11:75:60:cb:59:e6:74:18:ee:0e:06:bc:54: + a1:47:f9:f5:b5:c0:be:ad:6d:ee:dd:99:b6:50:ed: + 85:33:f5:bd:93:4b:66:a9:08:f0:67:c7:bd:42:24: + c2:3b:e3:7f:f1:e2:51:62:b5:51:ae:21:a8:24:d9: + c9:ed:d3:b4:60:17:6a:0c:78:69:c1:96:ad:62:1d: + 18:11:a6:ea:f4:83:eb:2d:ae:b0:be:2e:56:9d:cf: + 9d:2c:70:5a:df:b2:1e:3a:c7:e4:23:e3:3b:58:e1: + fd:9d Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Authority Key Identifier: keyid:3D:D3:50:A5:D6:A0:AD:EE:F3:4A:60:0A:65:D3:21:D4:F8:F8:D6:0F X509v3 Subject Key Identifier: - A7:47:98:D1:12:78:DB:51:32:FA:8D:BF:1D:2E:6E:C3:0E:CD:CC:ED + F0:1E:F5:E3:EE:33:53:69:54:6A:27:40:E0:CE:84:B6:69:68:4B:9E X509v3 Subject Alternative Name: - DNS:www.paypal.com, DNS:www-st.paypal.com, DNS:history.paypal.com + DNS:www.paypal.com, DNS:login.paypal.com, DNS:history.paypal.com, DNS:www.paypalobjects.com, DNS:pics.paypal.com X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: @@ -66,95 +66,96 @@ Certificate: CT Precertificate SCTs: Signed Certificate Timestamp: Version : v1 (0x0) - Log ID : EE:4B:BD:B7:75:CE:60:BA:E1:42:69:1F:AB:E1:9E:66: - A3:0F:7E:5F:B0:72:D8:83:00:C4:7B:89:7A:A8:FD:CB - Timestamp : Mar 10 17:16:18.044 2020 GMT + Log ID : A4:B9:09:90:B4:18:58:14:87:BB:13:A2:CC:67:70:0A: + 3C:35:98:04:F9:1B:DF:B8:E3:77:CD:0E:C8:0D:DC:10 + Timestamp : Jan 9 00:41:33.309 2020 GMT Extensions: none Signature : ecdsa-with-SHA256 - 30:46:02:21:00:F0:BB:36:25:D8:23:AE:01:8E:8E:84: - E4:0E:90:A1:C5:C3:27:41:CC:01:EA:D2:77:EA:FA:C8: - F1:E2:93:12:86:02:21:00:CB:A9:16:94:FA:DC:F9:97: - 25:17:FD:C9:89:80:6B:01:C5:61:F2:46:E3:60:26:D4: - B2:88:7E:37:31:39:D4:50 + 30:46:02:21:00:F6:D8:70:6F:DE:F3:A1:DF:10:DF:94: + 78:E6:27:98:A9:7C:60:D1:C2:09:7D:39:DE:18:E6:4B: + D4:79:F7:FB:00:02:21:00:A5:B9:13:F3:F6:69:AB:70: + DC:D0:F3:AD:1F:EF:FA:4F:57:0E:38:00:6C:48:A8:78: + 99:9C:8C:32:94:97:21:24 Signed Certificate Timestamp: Version : v1 (0x0) Log ID : 56:14:06:9A:2F:D7:C2:EC:D3:F5:E1:BD:44:B2:3E:C7: 46:76:B9:BC:99:11:5C:C0:EF:94:98:55:D6:89:D0:DD - Timestamp : Mar 10 17:16:18.078 2020 GMT + Timestamp : Jan 9 00:41:33.535 2020 GMT Extensions: none Signature : ecdsa-with-SHA256 - 30:46:02:21:00:F6:C2:EA:CE:80:A5:3A:2A:ED:06:9D: - 07:8B:61:1B:14:F3:28:36:C6:A0:67:92:89:D6:53:66: - D4:56:6B:01:D7:02:21:00:DB:5E:24:47:FE:41:4E:A8: - D9:9A:B3:33:58:FC:25:42:79:96:9E:CC:7C:96:AC:38: - F7:A6:2A:EE:09:CE:15:10 + 30:45:02:21:00:FF:91:95:F6:47:8B:41:58:C0:BD:19: + 73:8B:9F:98:A0:5C:F2:9A:24:22:2A:F2:64:0F:48:B7: + DE:40:22:8D:DC:02:20:4B:9A:A9:F1:79:A3:01:65:10: + CA:BC:FC:24:F5:0A:9D:9A:1A:05:10:F0:2E:0C:EF:CC: + A9:AF:24:84:13:29:A0 Signed Certificate Timestamp: Version : v1 (0x0) - Log ID : BB:D9:DF:BC:1F:8A:71:B5:93:94:23:97:AA:92:7B:47: - 38:57:95:0A:AB:52:E8:1A:90:96:64:36:8E:1E:D1:85 - Timestamp : Mar 10 17:16:17.984 2020 GMT + Log ID : EE:4B:BD:B7:75:CE:60:BA:E1:42:69:1F:AB:E1:9E:66: + A3:0F:7E:5F:B0:72:D8:83:00:C4:7B:89:7A:A8:FD:CB + Timestamp : Jan 9 00:41:33.381 2020 GMT Extensions: none Signature : ecdsa-with-SHA256 - 30:45:02:20:30:08:34:13:7D:35:8D:A3:EE:B3:C8:D1: - 1C:40:B1:DC:40:78:76:6C:7D:8B:D6:06:9A:99:BF:7B: - 09:63:14:1A:02:21:00:ED:76:8A:E0:EC:80:23:E7:C6: - 4E:50:DB:5A:B2:C9:D8:16:3E:1C:B2:11:CF:F0:7A:80: - C7:47:6C:27:39:E0:CC + 30:44:02:20:08:FE:99:AB:2F:BE:95:36:08:E0:23:F7: + FA:0D:EE:50:A2:46:00:51:D3:F4:8A:75:8C:74:62:02: + A4:53:5C:8A:02:20:13:8A:4C:E6:E2:C7:0D:38:39:EB: + 49:29:D4:23:43:4C:FA:4B:01:8C:D2:DB:CB:7F:39:4C: + 84:14:E4:A4:DD:24 Signature Algorithm: sha256WithRSAEncryption - 2d:ec:38:8d:c0:a9:e7:95:69:72:73:e1:4b:31:d0:4a:93:95: - de:81:c2:bb:50:57:79:18:2f:1c:b9:36:b2:01:6c:f8:31:47: - 8f:24:e7:84:f9:68:cd:28:4a:86:f8:24:b0:f3:0e:dc:18:4d: - 18:2b:d8:a9:73:6e:6e:21:43:20:94:a7:33:d9:7c:a7:87:7c: - 25:8d:09:4d:5f:e4:b7:91:91:e5:2d:21:fb:3c:87:60:da:5f: - 0f:0f:b3:04:24:bc:4e:32:5f:e3:39:86:35:8d:55:ba:52:72: - f4:91:22:90:95:ce:aa:fc:c0:9f:eb:b2:ec:a2:97:66:78:d8: - 1e:83:cc:26:44:89:f2:21:fc:14:5e:42:de:cf:26:6d:f9:85: - 8d:83:35:45:64:92:e3:8a:5d:7b:34:df:24:9f:e6:9a:3a:52: - 33:27:25:8a:10:57:2b:5b:e6:15:3c:43:41:ac:6d:2e:18:ac: - 61:b3:5d:e6:1c:71:0f:0d:a8:65:36:22:6e:bb:9c:e0:42:c1: - 98:96:41:6e:bc:a2:71:a0:52:6e:60:78:a5:51:42:0b:92:5f: - ca:42:36:74:ff:d3:c9:30:b1:57:e5:b2:fb:e7:23:8c:05:1e: - 86:ad:7d:7b:f2:dd:43:6a:2b:82:87:ea:a4:01:4a:8e:a4:f5: - 9d:29:2d:78 + 95:06:8c:5c:1c:39:aa:19:33:0c:70:58:7f:94:2b:a4:71:be: + f7:13:4e:23:73:f0:a8:7c:35:16:0e:e2:be:59:56:3d:8c:5d: + fa:8c:fb:dd:de:8e:34:a3:6a:b3:86:5b:29:52:4a:5f:f7:cb: + 1f:33:b8:c8:60:3b:30:72:94:fc:61:df:5d:80:2f:f9:8f:ad: + f3:85:98:2a:e8:ed:6c:02:e1:3e:d0:cc:ef:68:36:4e:ae:51: + 6d:ca:2e:7b:ae:a3:79:3d:27:67:74:15:4d:cf:c9:1d:a1:f9: + 43:69:ce:66:b2:eb:ec:c4:31:48:27:d9:2d:e2:eb:f4:72:0a: + 73:23:d1:9c:5d:8e:34:b2:95:a3:a8:09:16:ce:2f:bf:d1:f8: + 47:bd:c1:6d:36:7e:3a:9c:58:c1:47:40:92:8e:b6:32:97:89: + 5e:fb:46:c3:3d:2c:06:46:23:86:2a:6c:d2:3a:18:3e:3a:2b: + fc:c3:3a:c0:17:6a:4c:32:f5:d2:a8:a9:a3:5f:2a:53:c9:bf: + 88:9f:0f:c6:74:63:7d:83:17:49:60:72:d2:cb:c9:b8:02:58: + f7:d9:f0:3c:fe:1f:4d:fb:eb:43:a0:fa:58:9e:19:1c:b7:6c: + 45:ec:0c:b9:0d:4a:09:be:76:68:35:48:62:5c:82:3c:80:e4: + e7:7b:66:f7 Obviously, both certificates appear valid, the major deviation is in the X509v3 Subject Alternative Names. Why their CDN delivers two different certificates from January and March this year escapes me. Claiming the same serial number in the Subject is even more suspicious, although the real serial numbers differ (I would be even more alarmed otherwise). The whole case stinks, but it's unclear, where the stink comes from. What do you guys think? Cheers, Pete -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Citeren Mathias Homann <Mathias.Homann@opensuse.org>:
Is there any chance to take that laptop somewhere where you'll definitely NOT be on a vodafone connection, and see if the problem persists?
And if it does NOT we should definitely talk to a lot of people.
Hold your horses. Vodafone has been doing this in various locations at least since 2014 as part of their Vodafone Content Control service. Chances are that even if a Vodafone customer doesn't subscribe to these services, their traffic still flows through the proxies (but without blocking content). Unless they redirect all HTTP(S) traffic through the proxies, chances are this is just done by DNS spoofing and should be trivial to circumvent by switching from the default Vodafone DNS resolvers to Google DNS (8.8.8.8 and 8.8.4.4) for instance. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 19.10.20 18:54, Arjen de Korte wrote:
Citeren Mathias Homann <Mathias.Homann@opensuse.org>:
Is there any chance to take that laptop somewhere where you'll definitely NOT be on a vodafone connection, and see if the problem persists?
And if it does NOT we should definitely talk to a lot of people.
Hold your horses. Vodafone has been doing this in various locations at least since 2014 as part of their Vodafone Content Control service. Chances are that even if a Vodafone customer doesn't subscribe to these services, their traffic still flows through the proxies (but without blocking content).
Unless they redirect all HTTP(S) traffic through the proxies, chances are this is just done by DNS spoofing and should be trivial to circumvent by switching from the default Vodafone DNS resolvers to Google DNS (8.8.8.8 and 8.8.4.4) for instance.
Good idea. I've changed the DNS servers, verified that the new ones are used, but this at least doesn't help against the osc issue. For Firefox there are also still lots of TCP and RST issues visible in the tcpdump although pages load rather fine. Needs more testing but I guess this is not a solution for me. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Montag, 19. Oktober 2020, 18:54:43 CEST schrieb Arjen de Korte:
Citeren Mathias Homann <Mathias.Homann@opensuse.org>:
Is there any chance to take that laptop somewhere where you'll definitely NOT be on a vodafone connection, and see if the problem persists?
And if it does NOT we should definitely talk to a lot of people.
Hold your horses. Vodafone has been doing this in various locations at least since 2014 as part of their Vodafone Content Control service. Chances are that even if a Vodafone customer doesn't subscribe to these services, their traffic still flows through the proxies (but without blocking content).
Unless they redirect all HTTP(S) traffic through the proxies, chances are this is just done by DNS spoofing and should be trivial to circumvent by switching from the default Vodafone DNS resolvers to Google DNS (8.8.8.8 and 8.8.4.4) for instance.
For the record, I did that a long time ago. The tests, I reported yesterday are done with the Google DNS servers as forwarders for the Vodafone tests. For obvious reasons, the Telekom Mobile DNS config is unchanged. Cheers, Pete -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2020-10-19 16:13:15 +0200, Sebastian Parschauer wrote:
On 19.10.20 15:59, Cristian Rodríguez wrote:
Got a Dell Latitude 5491 with the Dell WD15 USB-C dock. 1.5m CAT5e LAN cable to a Netgear router
So..what's this netgear router doing (other than routing?) is its firmware up2date ? is it doing any sort of web filtering or is web filtering enabled but not filtering anything ? the behaviour you see is clearly as day as one of a man in middle, whatever the SSL code bugs might be, it should NOT behave like this.
Firmware is up2date. It uses MAC security. So unknown MAC addresses are blocked. For the osc bug I shortly connected my laptop with the SUSE firewall directly to cable modem and the issue persisted.
The client retransmitted the client hello all the time and was waiting for the server hello which never came. And a lot more errors were visible like "TCP ACKed unseen segment". There is no man in middle issue. The tools just get stuck in the SSL network state machine. There are no atomic operations with Ethernet.
Hmm according to [1], I would rather say that there's probably an issue with the TCP connection. I really fail to see why - a RST is ACKed - the ACK is received on a new _to be established_ connection IMHO, this TCP behavior should not happen at all/is strange (regardless if a SSL connection is not properly closed or not). Any ideas? Marcus [1] https://bugzilla.suse.com/show_bug.cgi?id=1068470#c52 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Oct 19, 2020 at 09:32:49PM +0200, Marcus Hüwe wrote:
On 2020-10-19 16:13:15 +0200, Sebastian Parschauer wrote:
On 19.10.20 15:59, Cristian Rodríguez wrote:
Got a Dell Latitude 5491 with the Dell WD15 USB-C dock. 1.5m CAT5e LAN cable to a Netgear router
So..what's this netgear router doing (other than routing?) is its firmware up2date ? is it doing any sort of web filtering or is web filtering enabled but not filtering anything ? the behaviour you see is clearly as day as one of a man in middle, whatever the SSL code bugs might be, it should NOT behave like this.
Firmware is up2date. It uses MAC security. So unknown MAC addresses are blocked. For the osc bug I shortly connected my laptop with the SUSE firewall directly to cable modem and the issue persisted.
The client retransmitted the client hello all the time and was waiting for the server hello which never came. And a lot more errors were visible like "TCP ACKed unseen segment". There is no man in middle issue. The tools just get stuck in the SSL network state machine. There are no atomic operations with Ethernet.
Hmm according to [1], I would rather say that there's probably an issue with the TCP connection. I really fail to see why - a RST is ACKed - the ACK is received on a new _to be established_ connection
IMHO, this TCP behavior should not happen at all/is strange (regardless if a SSL connection is not properly closed or not).
Any ideas?
We could also consider a bad network card / card in a bad (offload?) mode in your machine if this problem is pinned to your machine. Ciao, Marcus -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 20.10.20 08:15, Marcus Meissner wrote:
Firmware is up2date. It uses MAC security. So unknown MAC addresses are blocked. For the osc bug I shortly connected my laptop with the SUSE firewall directly to cable modem and the issue persisted.
The client retransmitted the client hello all the time and was waiting for the server hello which never came. And a lot more errors were visible like "TCP ACKed unseen segment". There is no man in middle issue. The tools just get stuck in the SSL network state machine. There are no atomic operations with Ethernet.
Hmm according to [1], I would rather say that there's probably an issue with the TCP connection. I really fail to see why - a RST is ACKed - the ACK is received on a new _to be established_ connection
IMHO, this TCP behavior should not happen at all/is strange (regardless if a SSL connection is not properly closed or not).
Any ideas?
We could also consider a bad network card / card in a bad (offload?) mode in your machine if this problem is pinned to your machine.
Nope, multiple systems are affected. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Oct 20, 2020 at 4:51 AM Sebastian Parschauer <s.parschauer@gmx.de> wrote:
Nope, multiple systems are affected.
and those multiple systems are ultimately attached to the same CPEs right ? some network device still broken.. can you borrow different network equipment for example a different xDSL gateway..(heared good things about DrayTek Vigor160) (It would be great if kernel network maintainers could comment on this thread, something is violating the specs and we do not know what exactly) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 20.10.20 13:48, Cristian Rodríguez wrote:
On Tue, Oct 20, 2020 at 4:51 AM Sebastian Parschauer <s.parschauer@gmx.de> wrote:
Nope, multiple systems are affected.
and those multiple systems are ultimately attached to the same CPEs right ? some network device still broken.. can you borrow different network equipment for example a different xDSL gateway..(heared good things about DrayTek Vigor160)
(It would be great if kernel network maintainers could comment on this thread, something is violating the specs and we do not know what exactly)
Network equipment is fine. Just certain timing leads to SSL issues. I would rather go for a long network cable to modify latency. But this is nothing else than a workaround. With a 100m network cable I would also only be able to shift latency by 1 / ((2/3 c) / 100m) = 1 / (200,000,000 m/s / 100m) = 500 ns. I guess this is not enough. I could ask neighbors but those also use the same Vodafone cable Internet. So likely I hit the same issues. Last month a Telekom guy came to the house as others complained about slow Internet. He told me that everything goes through the same fiber optics no matter which vendor. But I know for sure that my neighbors are unable to debug their network and Internet issues themselves. So maybe we all hit unfortunate SSL timing. Maybe playing around with e1000e kernel module parameters could help but would also only be a workaround. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 20/10/2020 15.39, Sebastian Parschauer wrote:
On 20.10.20 13:48, Cristian Rodríguez wrote:
On Tue, Oct 20, 2020 at 4:51 AM Sebastian Parschauer <> wrote:
I could ask neighbors but those also use the same Vodafone cable Internet.
how about tethering to your phone? -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
Sebastian Parschauer píše v Po 19. 10. 2020 v 12:20 +0200:
On 19.10.20 12:09, Jan Engelhardt wrote:
On Monday 2020-10-19 11:45, Sebastian Parschauer wrote:
I've reported issues in SSL handling before which cause tools to hang and cause port 443 RSTs in the tcpdump. Since 2017 I maintain a custom osc SSL patch/fix package.
And this patch does what?
osc: Closing the SSL connection properly. See https://bugzilla.suse.com/show_bug.cgi?id=1068470 attachment osc-ssl-fix-hang-at-server-hello.patch: + def close(self): + self.sock.close()
Comments and reviews of https://gitlab.com/m2crypto/m2crypto/-/merge_requests/234 (and consider all comments on https://gitlab.com/m2crypto/m2crypto/-/merge_requests/188) are more than welcome … the issue is a bit more complicated that this. Best, Matěj -- https://matej.ceplovi.cz/blog/, Jabber: mcepl@ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 “Promise me you’ll look after yourself. … Stay out of trouble. …” “I always do, Mrs. Weasley,” said Harry. “I like a quiet life, you know me.” -- Harry Potter and the Half-Blood Prince, chapter 17
Am Montag, 19. Oktober 2020, 11:45:51 CEST schrieb Sebastian Parschauer:
Hi,
steps to reproduce? Needless to say, on two out of two Leap 15.2 systems here I can not reproduce your issues. Cheers Mathias -- Mathias Homann Mathias.Homann@openSUSE.org OBS: lemmy04 Jabber (XMPP): lemmy@tuxonline.tech IRC: [Lemmy] on freenode and ircnet (bouncer active) telegram: https://telegram.me/lemmy98 keybase: https://keybase.io/lemmy gpg key fingerprint: 8029 2240 F4DD 7776 E7D2 C042 6B8E 029E 13F2 C102
On 19.10.20 12:25, Mathias Homann wrote:
Am Montag, 19. Oktober 2020, 11:45:51 CEST schrieb Sebastian Parschauer:
Hi,
steps to reproduce?
Needless to say, on two out of two Leap 15.2 systems here I can not reproduce your issues.
Reproduction of the hanging is pretty impossible if it didn't hit before. If it hit before: An internet connection with stable latency is required (LAN). Then it will hit again and again. SSL can lead to hanging forever if 1) A connection is tried to be reestablished although the old one is still open. The server will react with connection reset for the new connection. The client will repeat its client hello and wait for the server hello which will never be received. 2) A connection is tried to be established which is directly closed by previous connection handling in the client again. The new connection can never be established (duplicate SSL_shutdown() calls - first one closes old connection, second one closes new connection about to be established). osc is type 1. libcurl is type 2. I guess Chromium, Firefox, and Thunderbird are type 2 as well or something else I haven't seen yet. I haven't seen the repeating of the SSL client hello but the port 443 RSTs. Time between "[FIN, ACK]" and "TCP ACKed unseen segment" was 246 ms, "[FIN, ACK]" and "Encrypted Alert" 1.334 ms, "Encrypted Alert" and "[RST, ACK]" 57 ns in the osc bug. All RSTs were gone after the fix. So this can be a matter of nanoseconds. Do you see port 443 RSTs as well in the tcpdump, when using Chromium, Firefox, and Thunderbird? Thanks in advance. Cheers, Sebastian -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, 2020-10-19 at 11:45 +0200, Sebastian Parschauer wrote:
Hi,
I've reported issues in SSL handling before which cause tools to hang and cause port 443 RSTs in the tcpdump. Since 2017 I maintain a custom osc SSL patch/fix package. Leap 15.0 didn't even install without a libcurl fix for zypper. But my patches fixing SSL handling in those tools haven't been accepted although they are correct.
After the upgrade to Leap 15.2, I had so unfortunate timing with my fast LAN/cable modem based internet connection that Firefox and Chromium couldn't load any tabs any more (all of them hanging in SSL handling). I've removed the ad blocker and Firefox didn't even start any more. I've executed "firefox -d gdb" and noticed that it crashes with a SEGFAULT in SSL handling at startup. Then I wanted to write this email and I noticed that Thunderbird was not able to download any email and the email window where I started writing crashed.
My employer forces me to use Chromium/Chrome due to the need for a special plugin. So I've installed the Epiphany browser to download latest Chrome. That one is really fast in loading https websites. I tested latest Chrome and that one is also not able to load any HTTPS website tab. Also RSTs in the tcpdump. So Leap 15.2 is unusable for me. I've rolled back to the full disk backup before upgrading.
I cannot fix SSL handling in three complex tools at once all by myself. For me it looks like I have to reinstall a distro as a workaround and I would choose Ubuntu as I can easily skip a faulty release there and can stay longer on a proper one. Using wireless connections with unpredictable high latency and harmful pulsed microwave radiation as a workaround is no option for me.
Any chance to extend Leap 15.1 support until 15.3 release?
openSUSE Leap 15.1 had release interlock where we've agreed on resources and lifecycle from openSUSE Leap 15.1 with the maintenance team. Both Leap 15.1 and Leap 15.2 follows the standard rule as described on https://en.opensuse.org/Lifetime Leap 15.3 did not go through interlock yet, therefore the maintenance plan is still an open topic. Unfortunately, Leap doesn't have the resources to provide such a service. If the 15.1 code stream is business-critical to you then I suggest you migrate to SLE 15 SP1. The fear of differences is something that 15.2 and 15.3 releases are trying to address.
IMHO an SSL stability initiative is required - even independent of vendors. If anybody else noticed slow browser tabs, slow https file downloads, slow email downloads, or plain hanging, then I'd be glad to team up to join forces against SSL network state machine violations.
Cheers, Sebastian -- Best regards
Luboš Kocman Release Manager openSUSE Leap SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nuremberg Germany (HRB 36809, AG Nürnberg) Managing Director: Felix Imendörffer
On 19.10.20 13:09, Lubos Kocman wrote:
Any chance to extend Leap 15.1 support until 15.3 release?
openSUSE Leap 15.1 had release interlock where we've agreed on resources and lifecycle from openSUSE Leap 15.1 with the maintenance team.
Both Leap 15.1 and Leap 15.2 follows the standard rule as described on https://en.opensuse.org/Lifetime
Leap 15.3 did not go through interlock yet, therefore the maintenance plan is still an open topic. Unfortunately, Leap doesn't have the resources to provide such a service.
If the 15.1 code stream is business-critical to you then I suggest you migrate to SLE 15 SP1. The fear of differences is something that 15.2 and 15.3 releases are trying to address.
Thanks. Will think about that. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (14)
-
Andrei Borzenkov -
Arjen de Korte -
Carlos E. R. -
Cristian Rodríguez -
Hans-Peter Jansen -
Jan Engelhardt -
Lew Wolfgang -
Lubos Kocman -
Marcus Hüwe -
Marcus Meissner -
Mathias Homann -
Matěj Cepl -
Sebastian Parschauer -
Stefan Seyfried