http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 Oliver Kurz changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |okurz@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 http://bugzilla.opensuse.org/show_bug.cgi?id=1200425#c1 Benjamin Zeller changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bzeller@suse.com --- Comment #1 from Benjamin Zeller --- I can reproduce it locally too, but I sometimes need more than a second attempt for it to work. According to the logs the server responds with a 404 for the primary.xml request: 2022-06-10 14:30:13 <3> 19e4e6e6aba1(367) [zypp::media] MediaCurl.cc(doGetFileCopyFile):1252 curl error: 22: The requested URL returned error: 404, temp file size 0 bytes. If the file can not be found that would be a invalid repodata situation, not sure if zypp can do anything about the server not correctly responding. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 http://bugzilla.opensuse.org/show_bug.cgi?id=1200425#c3 --- Comment #3 from Dominik Heidler --- Hm - could it be that we have a broken mirror server? And on the 2nd attempt we're lucky and get a mirror that has the desired file. That would also explain the issues I had when updating one of my machines from 15.3 to 15.4. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 http://bugzilla.opensuse.org/show_bug.cgi?id=1200425#c5 --- Comment #5 from Michael Andres --- Try to use `ZYPP_MULTICURL=0 zypper ...` and so disable using metalink. I see the same errors while plain `curl -L https://download.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_15.4/repodata/530faf950f7f3915ae2210e94624215278b6ac551e40514e05772277c976f244-primary.xml.gz` or access without metalink work. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 http://bugzilla.opensuse.org/show_bug.cgi?id=1200425#c7 Michael Andres changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrii.nikitin@suse.com Flags| |needinfo?(andrii.nikitin@su | |se.com) --- Comment #7 from Michael Andres --- Andrii, any idea what may cause those hiccups? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425
http://bugzilla.opensuse.org/show_bug.cgi?id=1200425#c9

--- Comment #9 from Michael Andres  ---
In case we can check the servers logs.
The failing request 2022-06-13 10:53:30 from 91.55.248.129:

> * Found bundle for host download.opensuse.org: 0x5604f43a9ce0 [can multiplex]
> * Re-using existing connection! (#0) with host download.opensuse.org
> * Connected to download.opensuse.org (195.135.221.134) port 443 (#0)
> * Using Stream ID: 7 (easy handle 0x5604f43b47b0)
> > GET /repositories/devel:/openQA/openSUSE_Leap_15.4/repodata/b137b90094f320a6254d1879cb7a5c30d3bbb8a4a1c8757084f4c09922779981-prim>
> > Host: download.opensuse.org
> > User-Agent: ZYpp 17.30.1 (curl 7.66.0) 
> > X-ZYpp-AnonymousId: 508ce7fd-36c7-4aad-ba2a-edef9fbd16f9
> > Accept: */*, application/metalink+xml, application/metalink4+xml
> * The requested URL returned error: 404 
> * stopped the pause stream!
> * Connection #0 to host download.opensuse.org left intact

10 seconds later 2022-06-13 10:53:30 the succeeding one:

> * Found bundle for host download.opensuse.org: 0x55ca297b34a0 [can multiplex]
> * Re-using existing connection! (#0) with host download.opensuse.org
> * Connected to download.opensuse.org (195.135.221.134) port 443 (#0)
> * Using Stream ID: 7 (easy handle 0x55ca296eca70)
> > GET /repositories/devel:/openQA/openSUSE_Leap_15.4/repodata/b137b90094f320a6254d1879cb7a5c30d3bbb8a4a1c8757084f4c09922779981-prim>
> > Host: download.opensuse.org
> > User-Agent: ZYpp 17.30.1 (curl 7.66.0) 
> > X-ZYpp-AnonymousId: 508ce7fd-36c7-4aad-ba2a-edef9fbd16f9
> > Accept: */*, application/metalink+xml, application/metalink4+xml
> < HTTP/2 302 
> < content-length: 0
> < server: Apache
> < location: https://mirrorcache-eu.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_15.4/repodata/b137b90094f320a6254d1879cb7a>
>s_curl):117 redirecting to location: https://mirrorcache-eu.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_15.4/repodata/b137>
> < date: Mon, 13 Jun 2022 08:53:30 GMT
> < cache-control: must-revalidate
> < expires: Mon, 13 Jun 2022 08:53:30 GMT
> < content-type: application/x-gzip
> * Connection #0 to host download.opensuse.org left intact
> * Issue another request to this URL: 'https://mirrorcache-eu.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_15.4/repodata/b1>
> *   Trying 195.135.221.140:443...
> * TCP_NODELAY set
> * Connected to mirrorcache-eu.opensuse.org (195.135.221.140) port 443 (#1)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> *   CAfile: none
> *   CApath: /etc/ssl/certs
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
> * TLSv1.3 (IN), TLS handshake, Certificate (11):
> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
> * TLSv1.3 (IN), TLS handshake, Finished (20):
> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
> * TLSv1.3 (OUT), TLS handshake, Finished (20):
> * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
> * ALPN, server accepted to use h2
> * Server certificate:
> *  subject: CN=opensuse.org
> *  start date: May 11 00:59:07 2022 GMT
> *  expire date: Aug  9 00:59:06 2022 GMT
> *  subjectAltName: host "mirrorcache-eu.opensuse.org" matched cert's "*.opensuse.org"
> *  issuer: C=US; O=Let's Encrypt; CN=R3
> *  SSL certificate verify ok.
> * Using HTTP2, server supports multi-use
> * Connection state changed (HTTP/2 confirmed)
> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
> * Using Stream ID: 1 (easy handle 0x55ca296eca70)
> > GET /repositories/devel:/openQA/openSUSE_Leap_15.4/repodata/b137b90094f320a6254d1879cb7a5c30d3bbb8a4a1c8757084f4c09922779981-prim>
> > Host: mirrorcache-eu.opensuse.org
> > User-Agent: ZYpp 17.30.1 (curl 7.66.0) 
> > X-ZYpp-AnonymousId: 508ce7fd-36c7-4aad-ba2a-edef9fbd16f9
> > Accept: */*, application/metalink+xml, application/metalink4+xml
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> * old SSL session ID is stale, removing
> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
> < HTTP/2 302 
> < location: https://downloadcontent.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_15.4/repodata/b137b90094f320a6254d1879cb7>
>s_curl):117 redirecting to location: https://downloadcontent.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_15.4/repodata/b13>
> < content-length: 0
> < date: Mon, 13 Jun 2022 08:53:30 GMT
> < x-frame-options: SAMEORIGIN
> < x-xss-protection: 1; mode=block
> < x-content-type-options: nosniff
> < referrer-policy: no-referrer-when-downgrade
> < strict-transport-security: max-age=15768000
> * Connection #1 to host mirrorcache-eu.opensuse.org left intact
> * Issue another request to this URL: 'https://downloadcontent.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_15.4/repodata/b>
> *   Trying 195.135.221.157:443...
> * TCP_NODELAY set
> * Connected to downloadcontent.opensuse.org (195.135.221.157) port 443 (#2)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> *   CAfile: none
> *   CApath: /etc/ssl/certs
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> * TLSv1.2 (IN), TLS handshake, Finished (20):
> * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
> * ALPN, server accepted to use h2
> * Server certificate:
> *  subject: CN=opensuse.org
> *  start date: May 11 00:58:24 2022 GMT
> *  expire date: Aug  9 00:58:23 2022 GMT
> *  subjectAltName: host "downloadcontent.opensuse.org" matched cert's "*.opensuse.org"
> *  issuer: C=US; O=Let's Encrypt; CN=R3
> *  SSL certificate verify ok.
> * Using HTTP2, server supports multi-use
> * Connection state changed (HTTP/2 confirmed)
> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
> * Using Stream ID: 1 (easy handle 0x55ca296eca70)
> > GET /repositories/devel:/openQA/openSUSE_Leap_15.4/repodata/b137b90094f320a6254d1879cb7a5c30d3bbb8a4a1c8757084f4c09922779981-prim>
> > Host: downloadcontent.opensuse.org
> > User-Agent: ZYpp 17.30.1 (curl 7.66.0) 
> > X-ZYpp-AnonymousId: 508ce7fd-36c7-4aad-ba2a-edef9fbd16f9
> > Accept: */*, application/metalink+xml, application/metalink4+xml
> * Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
> < HTTP/2 200 
> < server: nginx
> < date: Mon, 13 Jun 2022 08:53:30 GMT
> < content-type: application/octet-stream
> < content-length: 19465
> < last-modified: Mon, 13 Jun 2022 08:45:40 GMT
> < etag: "62a6f934-4c09"
> < strict-transport-security: max-age=31536000
> < accept-ranges: bytes

To me it looks like the server is too slow to lauch something. Subsequent
requests succeeded. 
But now after I took awhile to write this. It fails again (once), and after
having been primed succeeded 4 times in a row.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 Pavel Dost�l changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pdostal@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 Dominik Heidler changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Normal |Major -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 http://bugzilla.opensuse.org/show_bug.cgi?id=1200425#c13 --- Comment #13 from Andrii Nikitin --- It looks a bug was introduced in MirrorCache metalink handling last week and I want to sick leave afterward, so didn't notice it. I hotpatched the root cause, so please confirm if possible if all works reliably now. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 http://bugzilla.opensuse.org/show_bug.cgi?id=1200425#c15 --- Comment #15 from Benjamin Zeller --- (In reply to Oliver Kurz from comment #14)

(In reply to Benjamin Zeller from comment #10)

...

Here i constantly get errors and just sometimes a success, removing the metalink makes it work

How about making that a zypper feature to automatically fall back to `ZYPP_MULTICURL=0` after multiple failed attempts to work around broken mirroring infrastructure?

We are already working on a full async backend that will completely replace the current curl and multicurl backends, so we are not going to introduce big changes into the old backend. Auto Fallback also is a bit tricky, consider a codepath that is just checking if a file exists. IF we'd do auto fallback on errors or auto retries all file existance checks had to be run multiple times, making zypper slower. IMO the feasible solution would be to implement opt in switches for automatic retries on failed downloads. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 http://bugzilla.opensuse.org/show_bug.cgi?id=1200425#c16 --- Comment #16 from Nick Dordea --- Thank you for looking into zypper issues. (ND 1200458 ) Regards, ND -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 http://bugzilla.opensuse.org/show_bug.cgi?id=1200425#c18 --- Comment #18 from Michael Andres --- (In reply to Oliver Kurz from comment #14)

How about making that a zypper feature to automatically fall back to `ZYPP_MULTICURL=0` after multiple failed attempts to work around broken mirroring infrastructure?

Sorry - no. If the server says 404(file not found), we must not ask 'Really?'. If the mirroring infrastructure is that broken, d.o.o should turn it off. We can try to work around issues where silent reties make sense. If ZYPP_MULTICURL fails(!) we already fall back to plain curl. But we can not distrust everything d.o.o is telling us. Then we must not use it at all. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 Nick Dordea changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ndordea@gmail.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1200425 http://bugzilla.opensuse.org/show_bug.cgi?id=1200425#c20 Andrii Nikitin changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CONFIRMED |RESOLVED Resolution|--- |INVALID --- Comment #20 from Andrii Nikitin --- indeed, closing as invalid, because there was no problem with libzypp, and openSUSE infrastructure should be reported to admin@opensuse.org -- You are receiving this mail because: You are on the CC list for the bug.