Fwd: checksum fails for downloaded ISO
---------- Forwarded message ---------- From: Darryl Gregorash <raven@accesscomm.ca> Date: Mon, 12 Sep 2022 12:09:38 -0600 Subject: Re: checksum fails for downloaded ISO To: support@lists.opensuse.org On 2022-09-12 11:41, Andrei Borzenkov wrote:
On 12.09.2022 19:31, Darryl Gregorash wrote:
Is this stupid thing still an issue? WTH, openSUSE guys, can't you take about a minute and 13 seconds to correct this?
Why do not you step in and fix it if it is so simple? I'm pretty sure I don't have the necessary privileges to do that. Next?
On Montag, 12. September 2022 20:10:48 CEST Luna Jernberg wrote:
---------- Forwarded message ---------- From: Darryl Gregorash <raven@accesscomm.ca> Date: Mon, 12 Sep 2022 12:09:38 -0600 Subject: Re: checksum fails for downloaded ISO To: support@lists.opensuse.org
On 2022-09-12 11:41, Andrei Borzenkov wrote:
On 12.09.2022 19:31, Darryl Gregorash wrote:
Is this stupid thing still an issue? WTH, openSUSE guys, can't you take about a minute and 13 seconds to correct this?
Why do not you step in and fix it if it is so simple?
I'm pretty sure I don't have the necessary privileges to do that. Next?
Luna, can you please stop forwarding messages? If I were interested on the support ML, I would subscribe to it. Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen phone: +49 241 53809034 mobile: +49 151 50412019
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 El 2022-09-12 a las 20:13 +0200, Stefan Brüns escribió:
On Montag, 12. September 2022 20:10:48 CEST Luna Jernberg wrote:
---------- Forwarded message ---------- From: Darryl Gregorash <raven@accesscomm.ca> Date: Mon, 12 Sep 2022 12:09:38 -0600 Subject: Re: checksum fails for downloaded ISO To: support@lists.opensuse.org
On 2022-09-12 11:41, Andrei Borzenkov wrote:
On 12.09.2022 19:31, Darryl Gregorash wrote:
Is this stupid thing still an issue? WTH, openSUSE guys, can't you take about a minute and 13 seconds to correct this?
Why do not you step in and fix it if it is so simple?
I'm pretty sure I don't have the necessary privileges to do that. Next?
Luna,
can you please stop forwarding messages?
If I were interested on the support ML, I would subscribe to it.
Arguably, it is not a support question, but a bug somewhere. An infrastructure configuration issue, perhaps. - -- Cheers Carlos E. R. (from openSUSE 15.3 (Legolas)) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCYx+dzhwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVUfwAnijM3K4UhBr+cSjAUhZT UqI8g5MZAJ9PmENeOvdLiLTL/heXrR/0zwWaDQ== =JL5o -----END PGP SIGNATURE-----
Stefan Brüns wrote:
Luna, can you please stop forwarding messages? If I were interested on the support ML, I would subscribe to it.
+1. My thoughts exactly. Besides, the forwarding breaks the threading, making it difficult to follow. -- Per Jessen, Zürich (17.8°C) Member, openSUSE Heroes
On 12.09.22 22:59, Carlos E. R. wrote:
Arguably, it is not a support question, but a bug somewhere. An infrastructure configuration issue, perhaps.
It would indeed help if the heroes configured a redirect that masks the symlink, so that no one downloads the unversioned media. So that on users disk it's always openSUSE-Leap-15.4-DVD-x86_64-Build243.2-Media.iso and openSUSE-Leap-15.4-DVD-x86_64-Build243.2-Media.iso.sha256 no matter if they clicked the original file or the symlink. Greetings, Stephan
Stephan Kulow wrote:
On 12.09.22 22:59, Carlos E. R. wrote:
Arguably, it is not a support question, but a bug somewhere. An infrastructure configuration issue, perhaps.
It would indeed help if the heroes configured a redirect that masks the symlink, so that no one downloads the unversioned media.
If that symlink isn't useful, maybe just remove it ? -- Per Jessen, Zürich (19.5°C) Member, openSUSE Heroes
On 13.09.22 08:37, Per Jessen wrote:
Stephan Kulow wrote:
On 12.09.22 22:59, Carlos E. R. wrote:
Arguably, it is not a support question, but a bug somewhere. An infrastructure configuration issue, perhaps.
It would indeed help if the heroes configured a redirect that masks the symlink, so that no one downloads the unversioned media.
If that symlink isn't useful, maybe just remove it ?
It's useful to link to it and make it discoverable that you can link to it. Greetings, Stephan
Stephan Kulow wrote:
On 13.09.22 08:37, Per Jessen wrote:
Stephan Kulow wrote:
On 12.09.22 22:59, Carlos E. R. wrote:
Arguably, it is not a support question, but a bug somewhere. An infrastructure configuration issue, perhaps.
It would indeed help if the heroes configured a redirect that masks the symlink, so that no one downloads the unversioned media.
If that symlink isn't useful, maybe just remove it ?
It's useful to link to it and make it discoverable that you can link to it.
Okay - maybe it would be most easily solved by also having the checksums as symlinks? It seems to me to be the real issue - if you click on the unversioned link, you get a versioned file due to the symlink, but the checksum isn't correctly symlinked ? -- Per Jessen, Zürich (18.9°C) Member, openSUSE Heroes
On 13.09.22 09:09, Per Jessen wrote:
Stephan Kulow wrote:
Okay - maybe it would be most easily solved by also having the checksums as symlinks?
It seems to me to be the real issue - if you click on the unversioned link, you get a versioned file due to the symlink, but the checksum isn't correctly symlinked ?
No, the problem is the content of the checksum - because in there you have the versioned filename. No matter how you got there. Greetings, Stephan
Stephan Kulow wrote:
On 13.09.22 09:09, Per Jessen wrote:
Stephan Kulow wrote:
Okay - maybe it would be most easily solved by also having the checksums as symlinks?
It seems to me to be the real issue - if you click on the unversioned link, you get a versioned file due to the symlink, but the checksum isn't correctly symlinked ?
No, the problem is the content of the checksum - because in there you have the versioned filename. No matter how you got there.
From http://download.opensuse.org, I am however given a 302 redirect, to
Hmm - maybe the issue is elsewhere. If I download "openSUSE-Leap-15.4-DVD-x86_64-Media.iso" from http://rsync.o.o, that is what I get, i.e. "openSUSE-Leap-15.4-DVD-x86_64-Media.iso". I can verify the checksum with openSUSE-Leap-15.4-DVD-x86_64-Media.iso.sha256 which has the correct filename. Same thing from e.g. http://mirror.hostsuisse.net or http://ftp.gwdg.de the versioned file, but not for the checksum file. -- Per Jessen, Zürich (22.1°C) Member, openSUSE Heroes
On Sep 13 2022, Per Jessen wrote:
From http://download.opensuse.org, I am however given a 302 redirect, to the versioned file, but not for the checksum file.
With $ rsync rsync.opensuse.org::opensuse-full/opensuse/distribution/leap/15.4/iso/ you can see that all the unversioned *.iso files are symlinks, but none of the *.sha256 files. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
Andreas Schwab wrote:
On Sep 13 2022, Per Jessen wrote:
From http://download.opensuse.org, I am however given a 302 redirect, to the versioned file, but not for the checksum file.
With
$ rsync rsync.opensuse.org::opensuse-full/opensuse/distribution/leap/15.4/iso/
you can see that all the unversioned *.iso files are symlinks, but none of the *.sha256 files.
Exactly. If the sha256 files were also symlinks, maybe pontifex would treat them the same way and send redirects? I'll have to check the apache config, but it seems to me we are doing this on purpose. -- Per Jessen, Zürich (23.4°C) Member, openSUSE Heroes
On 2022-09-13 12:02, Per Jessen wrote:
Stephan Kulow wrote:
On 13.09.22 09:09, Per Jessen wrote:
Stephan Kulow wrote:
Okay - maybe it would be most easily solved by also having the checksums as symlinks?
It seems to me to be the real issue - if you click on the unversioned link, you get a versioned file due to the symlink, but the checksum isn't correctly symlinked ?
No, the problem is the content of the checksum - because in there you have the versioned filename. No matter how you got there.
Hmm - maybe the issue is elsewhere.
If I download "openSUSE-Leap-15.4-DVD-x86_64-Media.iso" from http://rsync.o.o, that is what I get, i.e. "openSUSE-Leap-15.4-DVD-x86_64-Media.iso".
I can verify the checksum with openSUSE-Leap-15.4-DVD-x86_64-Media.iso.sha256 which has the correct filename.
But if you do: wget http://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-D... wget http://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-D... cer@Elesar:~> sha256sum -c openSUSE-Leap-15.4-DVD-aarch64-Media.iso.sha256 sha256sum: openSUSE-Leap-15.4-DVD-aarch64-Media.iso: No such file or directory openSUSE-Leap-15.4-DVD-aarch64-Media.iso: FAILED open or read sha256sum: WARNING: 1 listed file could not be read cer@Elesar:~> even though openSUSE-Leap-15.4-DVD-x86_64-Media.iso exists and is correct. The needed behavior is that "wget http://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-D..." gets instead openSUSE-Leap-15.4-DVD-aarch64-Build243.2-Media.iso, with some trick on Apache or whatever (I don't know what). -- Cheers / Saludos, Carlos E. R. (from Elesar, using openSUSE Leap 15.3)
On 13.09.2022 13:42, Per Jessen wrote:
Andreas Schwab wrote:
On Sep 13 2022, Per Jessen wrote:
From http://download.opensuse.org, I am however given a 302 redirect, to the versioned file, but not for the checksum file.
With
$ rsync rsync.opensuse.org::opensuse-full/opensuse/distribution/leap/15.4/iso/
you can see that all the unversioned *.iso files are symlinks, but none of the *.sha256 files.
Exactly. If the sha256 files were also symlinks, maybe pontifex would treat them the same way and send redirects? I'll have to check the apache config, but it seems to me we are doing this on purpose.
The problem is that end result depends on client software (and client options) and there is no configuration that would work with any combination of links and clients. wget ... follows redirects but stores files under original name, so openSUSE-Leap-15.4-NET-x86_64-Media.iso is stored as openSUSE-Leap-15.4-NET-x86_64-Media.iso and linking openSUSE-Leap-15.4-NET-x86_64-Media.iso.sha256 to a file that contains openSUSE-Leap-15.4-NET-x86_64-Build243.2-Media.iso name will fail checksum. wget --trust-server-names ... will store file under redirected name, so linking checksum will work. Of course it fails now. curl -LO ... follows redirect but stores file under original name. See above for wget. curl does not have the equivalent of wget --trust-server-names. Both firefox and chromium store file under redirected name when downloading from download.o.o; of course when downloading from something like https://rsync.opensuse.org/ there is no redirect and so filename in .sha256 would not match. Using links simplifies download frontend (get.o.o) which may have static configuration pointing at ...-Media.iso which is magically resolved to the latest image. But it is never ending source of confusion as far as I remember.
Andrei Borzenkov wrote:
The problem is that end result depends on client software (and client options) and there is no configuration that would work with any combination of links and clients.
Yeah, that is also roughly the conclusion I have been getting to. Using my naive combination of wget and sha256sum, I was unable to reproduce the issue.
wget ... follows redirects but stores files under original name, so [snip] wget --trust-server-names ... will store file under redirected name, so linking checksum will work. Of course it fails now.
Interesting, I was not aware of that option.
Both firefox and chromium store file under redirected name when downloading from download.o.o; of course when downloading from something like https://rsync.opensuse.org/ there is no redirect and so filename in .sha256 would not match.
Yup. I was also going to ask the OP why he was (or appeared to be) using a mirror directly instead of going via download.o.o
Using links simplifies download frontend (get.o.o) which may have static configuration pointing at ...-Media.iso which is magically resolved to the latest image. But it is never ending source of confusion as far as I remember.
The topic certainly pops up every now and then. What if we simply told people to make sure they do their checking on : <filename>.iso <filename>.iso.sha256 What I wonder is if there is some gui or some wrapper that does this checking for them. -- Per Jessen, Zürich (20.9°C) Member, openSUSE Heroes
On 2022-09-13 20:55, Per Jessen wrote:
Andrei Borzenkov wrote:
What I wonder is if there is some gui or some wrapper that does this checking for them.
aria2c --check-integrity=true "URL" And it was documented in the wiki, at least when I checked it long ago. I might have written part of it. -- Cheers / Saludos, Carlos E. R. (from 15.3 x86_64 at Telcontar)
On 13.09.2022 23:24, Carlos E. R. wrote:
On 2022-09-13 20:55, Per Jessen wrote:
Andrei Borzenkov wrote:
What I wonder is if there is some gui or some wrapper that does this checking for them.
aria2c --check-integrity=true "URL"
It does not use checksum files on server at all, it using metalink. So it does not hit this issue.
And it was documented in the wiki, at least when I checked it long ago. I might have written part of it.
Hello Per, Am Dienstag, 13. September 2022, 20:55:06 CEST schrieb Per Jessen:
Both firefox and chromium store file under redirected name when downloading from download.o.o; of course when downloading from something like https://rsync.opensuse.org/ there is no redirect and so filename in .sha256 would not match.
Yup. I was also going to ask the OP why he was (or appeared to be) using a mirror directly instead of going via download.o.o
I just verified, the download gets redirected to the *Build243.2-Media.iso file. *no* direct use of Mirror. Cheers Axel
Axel Braun wrote:
Hello Per,
Am Dienstag, 13. September 2022, 20:55:06 CEST schrieb Per Jessen:
Both firefox and chromium store file under redirected name when downloading from download.o.o; of course when downloading from something like https://rsync.opensuse.org/ there is no redirect and so filename in .sha256 would not match.
Yup. I was also going to ask the OP why he was (or appeared to be) using a mirror directly instead of going via download.o.o
I just verified, the download gets redirected to the *Build243.2-Media.iso file. *no* direct use of Mirror.
Yes, that is to be expected, but the name of the downloaded file should _not_ contain the versioning when the request doesn't. -- Per Jessen, Zürich (24.5°C) Member, openSUSE Heroes
Am Mittwoch, 14. September 2022, 11:47:03 CEST schrieb Per Jessen:
Axel Braun wrote:
Hello Per,
Am Dienstag, 13. September 2022, 20:55:06 CEST schrieb Per Jessen:
Both firefox and chromium store file under redirected name when downloading from download.o.o; of course when downloading from something like https://rsync.opensuse.org/ there is no redirect and so filename in .sha256 would not match.
Yup. I was also going to ask the OP why he was (or appeared to be) using a mirror directly instead of going via download.o.o
I just verified, the download gets redirected to the *Build243.2-Media.iso file. *no* direct use of Mirror.
Yes, that is to be expected, but the name of the downloaded file should _not_ contain the versioning when the request doesn't.
But it does - do we make a ticket with admin@o.o out of this? Cheers Axel
On Wed, Sep 14, 2022 at 12:47 PM Per Jessen <per@opensuse.org> wrote:
Yes, that is to be expected, but the name of the downloaded file should _not_ contain the versioning when the request doesn't.
Why? I already tried to explain that name of downloaded file depends on client and using most popular web browsers with get.o.o results in file name containing version because they are using file name from redirection. I just tried once more using Edge.
Andrei Borzenkov wrote:
On Wed, Sep 14, 2022 at 12:47 PM Per Jessen <per@opensuse.org> wrote:
Yes, that is to be expected, but the name of the downloaded file should _not_ contain the versioning when the request doesn't.
Why? I already tried to explain that name of downloaded file depends on client
Sorry, yes, it _does_ depend on the client, but I still say when you request one file, you should not end up with another. -- Per Jessen, Zürich (24.8°C) Member, openSUSE Heroes
Axel Braun wrote:
Am Mittwoch, 14. September 2022, 11:47:03 CEST schrieb Per Jessen:
Axel Braun wrote:
Hello Per,
Am Dienstag, 13. September 2022, 20:55:06 CEST schrieb Per Jessen:
Both firefox and chromium store file under redirected name when downloading from download.o.o; of course when downloading from something like https://rsync.opensuse.org/ there is no redirect and so filename in .sha256 would not match.
Yup. I was also going to ask the OP why he was (or appeared to be) using a mirror directly instead of going via download.o.o
I just verified, the download gets redirected to the *Build243.2-Media.iso file. *no* direct use of Mirror.
Yes, that is to be expected, but the name of the downloaded file should _not_ contain the versioning when the request doesn't.
But it does - do we make a ticket with admin@o.o out of this?
Feel free :-) but as Andrei says, it is about the client, not the server. I don't see any infrastructure issue here. -- Per Jessen, Zürich (24.8°C) Member, openSUSE Heroes
On 2022-09-14 06:01, Andrei Borzenkov wrote:
On 13.09.2022 23:24, Carlos E. R. wrote:
On 2022-09-13 20:55, Per Jessen wrote:
Andrei Borzenkov wrote:
What I wonder is if there is some gui or some wrapper that does this checking for them.
aria2c --check-integrity=true "URL"
It does not use checksum files on server at all, it using metalink. So it does not hit this issue.
Exactly :-) It does an integrity check of its own, using the metalink file. If there is an error, it downloads again the faulty chunk, not the entire ISO, so that the end result is a guaranteed correct ISO. -- Cheers / Saludos, Carlos E. R. (from 15.3 x86_64 at Telcontar)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 El 2022-09-14 a las 12:25 +0200, Per Jessen escribió:
Andrei Borzenkov wrote:
On Wed, Sep 14, 2022 at 12:47 PM Per Jessen <per@opensuse.org> wrote:
Yes, that is to be expected, but the name of the downloaded file should _not_ contain the versioning when the request doesn't.
Why? I already tried to explain that name of downloaded file depends on client
Sorry, yes, it _does_ depend on the client, but I still say when you request one file, you should not end up with another.
We got detailed instructions to reproduce the issue in the support mail list, from Dave Howorth. And I reproduced the issue. On the download page https://get.opensuse.org/leap/15.4/#download Now observe the aarch64 offline button. Right click to copy the link: you obtain the non versioned name. But if you actually click on it, you get the versioned file instead. THAT is the problem. If you click on the dropdown list, you get this: Download -> https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-... metalink -> https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-... pick mirror -> https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-... checksum -> https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-... torrentfile -> https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-... When clicking on "checksum" you get what it says, the non versioned file. - -- Cheers, Carlos E. R. (from openSUSE 15.3 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCYyG/oBwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVtUkAnRSpeC7KL5yAqBgpdn31 10lelI4bAJ0awjgaoO/8w+yGRPreFLUb98VZRQ== =yj84 -----END PGP SIGNATURE-----
On Wed, Sep 14, 2022 at 1:25 PM Per Jessen <per@opensuse.org> wrote:
Andrei Borzenkov wrote:
On Wed, Sep 14, 2022 at 12:47 PM Per Jessen <per@opensuse.org> wrote:
Yes, that is to be expected, but the name of the downloaded file should _not_ contain the versioning when the request doesn't.
Why? I already tried to explain that name of downloaded file depends on client
Sorry, yes, it _does_ depend on the client, but I still say when you request one file, you should not end up with another.
OK, it was "should" as in "I expect it ..." vs. "should" as in "I wish it ..." :) Anyway, to eliminate this issue the server must not issue redirects. And it is not enough to stop redirects on get.o.o (or download.o.o), because you cannot control what mirror servers will be doing with symbolic links. This means there should be no symbolic links in the download area. Which I assume will not happen. The least intrusive change is to link all checksum files as well. Then users coming via browser will get (sort of) expected result and users using wget/curl probably can be considered advanced users who can cope with it. Of course it can break someone's workflow involving curl/wget. Although I am pretty sure that last time I looked into this issue at least some browsers used the original filename as well ... so the solution will work until the next browsers behavior change :)
The original reason for the link with the name Current in it was that it was pointing to the latest build during the development of it, while the earlier builds were still there. So in documentation the URL with Current in it, is/was always valid. Because now this is the latest build for 15.4 the Build does not change anymore. Keeping the name with Current means documentation referring to this build does not need to change. I have a small script that downloads the .asc and .sha256 files with Current in its name. It checks the validity of the .sha256 file and looks up the name of the .iso file in the .sha256 file and downloads that file. Finally the script checks the integrity of the .iso file. In my view nothing needs to be changed in the repo. -- fr.gr. member openSUSE Freek de Kruijf
Andrei Borzenkov wrote:
On Wed, Sep 14, 2022 at 1:25 PM Per Jessen <per@opensuse.org> wrote:
Andrei Borzenkov wrote:
On Wed, Sep 14, 2022 at 12:47 PM Per Jessen <per@opensuse.org> wrote:
Yes, that is to be expected, but the name of the downloaded file should _not_ contain the versioning when the request doesn't.
Why? I already tried to explain that name of downloaded file depends on client
Sorry, yes, it _does_ depend on the client, but I still say when you request one file, you should not end up with another.
OK, it was "should" as in "I expect it ..." vs. "should" as in "I wish it ..." :)
GRIN.
Anyway, to eliminate this issue the server must not issue redirects. And it is not enough to stop redirects on get.o.o (or download.o.o), because you cannot control what mirror servers will be doing with symbolic links. This means there should be no symbolic links in the download area. Which I assume will not happen.
Hmm, wait. The issue is the 302 redirect to the versioned file, isn't it? When a user requests : "d.o.o:/distribution/leap/15.4/iso/openSUSE-Leap-15.4-DVD-x86_64-Media.iso" he is given an http 302 redirect to: "d.o.o:/distribution/leap/15.4/iso/openSUSE-Leap-15.4-DVD-x86_64-Build243.2-Media.iso" and eventually also to the mirror with that file. Unfortunately clients don't agree on how to work with that. I wonder if this is something specific to mirrorbrain/cache ? When I use Firefox to download directly from a mirror: "http://mirror.hostsuisse.com/opensuse/distribution/leap/15.4/iso/openSUSE-Le..." there is no redirect, and I get a file called "openSUSE-Leap-15.4-DVD-x86_64-Media.iso".
The least intrusive change is to link all checksum files as well. Then users coming via browser will get (sort of) expected result
I think that's what I tried to say in one of my first responses in this thread, talking to Stephan. -- Per Jessen, Zürich (23.1°C) Member, openSUSE Heroes
Carlos E. R. wrote:
When clicking on "checksum" you get what it says, the non versioned file.
Confirm - the difference is that the nonversioned iso file is a symlink, where as the nonversioned sha256 file is not. For whatever reason, mirrorbache sends an http302 to the symlinked file, which seems(!) to be the issue. If mirrorbache would just send a redirect to the original/correct name, nobody gets to see the versioned iso, which I believe is what we want. -- Per Jessen, Zürich (22.5°C) Member, openSUSE Heroes
On 2022-09-14 17:47, Per Jessen wrote:
Carlos E. R. wrote:
When clicking on "checksum" you get what it says, the non versioned file.
Confirm - the difference is that the nonversioned iso file is a symlink, where as the nonversioned sha256 file is not. For whatever reason, mirrorbache sends an http302 to the symlinked file, which seems(!) to be the issue. If mirrorbache would just send a redirect to the original/correct name, nobody gets to see the versioned iso, which I believe is what we want.
Does that explain the different URL obtained when right clicking on the download button, or actually clicking on it? That seems to be the main problem. -- Cheers / Saludos, Carlos E. R. (from 15.3 x86_64 at Telcontar)
Carlos E. R. wrote:
On 2022-09-14 17:47, Per Jessen wrote:
Carlos E. R. wrote:
When clicking on "checksum" you get what it says, the non versioned file.
Confirm - the difference is that the nonversioned iso file is a symlink, where as the nonversioned sha256 file is not. For whatever reason, mirrorbache sends an http302 to the symlinked file, which seems(!) to be the issue. If mirrorbache would just send a redirect to the original/correct name, nobody gets to see the versioned iso, which I believe is what we want.
Does that explain the different URL obtained when right clicking on the download button, or actually clicking on it? That seems to be the main problem.
If you click on it, you get the same url as if you right clicked. The difference is that the browser hides the subsequent redirect from you and you end up getting a different filename. -- Per Jessen, Zürich (20.2°C) Member, openSUSE Heroes
On 2022-09-14 20:44, Per Jessen wrote:
Carlos E. R. wrote:
On 2022-09-14 17:47, Per Jessen wrote:
Carlos E. R. wrote:
When clicking on "checksum" you get what it says, the non versioned file.
Confirm - the difference is that the nonversioned iso file is a symlink, where as the nonversioned sha256 file is not. For whatever reason, mirrorbache sends an http302 to the symlinked file, which seems(!) to be the issue. If mirrorbache would just send a redirect to the original/correct name, nobody gets to see the versioned iso, which I believe is what we want.
Does that explain the different URL obtained when right clicking on the download button, or actually clicking on it? That seems to be the main problem.
If you click on it, you get the same url as if you right clicked. The difference is that the browser hides the subsequent redirect from you and you end up getting a different filename.
Ah. Can it be worked around "somehow"? -- Cheers / Saludos, Carlos E. R. (from 15.3 x86_64 at Telcontar)
On 14.09.2022 18:34, Per Jessen wrote:
Andrei Borzenkov wrote:
On Wed, Sep 14, 2022 at 1:25 PM Per Jessen <per@opensuse.org> wrote:
Andrei Borzenkov wrote:
On Wed, Sep 14, 2022 at 12:47 PM Per Jessen <per@opensuse.org> wrote:
Yes, that is to be expected, but the name of the downloaded file should _not_ contain the versioning when the request doesn't.
Why? I already tried to explain that name of downloaded file depends on client
Sorry, yes, it _does_ depend on the client, but I still say when you request one file, you should not end up with another.
OK, it was "should" as in "I expect it ..." vs. "should" as in "I wish it ..." :)
GRIN.
Anyway, to eliminate this issue the server must not issue redirects. And it is not enough to stop redirects on get.o.o (or download.o.o), because you cannot control what mirror servers will be doing with symbolic links. This means there should be no symbolic links in the download area. Which I assume will not happen.
Hmm, wait. The issue is the 302 redirect to the versioned file, isn't it? When a user requests :
"d.o.o:/distribution/leap/15.4/iso/openSUSE-Leap-15.4-DVD-x86_64-Media.iso"
he is given an http 302 redirect to:
"d.o.o:/distribution/leap/15.4/iso/openSUSE-Leap-15.4-DVD-x86_64-Build243.2-Media.iso"
and eventually also to the mirror with that file. Unfortunately clients don't agree on how to work with that.
I wonder if this is something specific to mirrorbrain/cache ? When I
Yes, both mirrorbrain and MirrorCache explicitly resolve symlinks and apply mirror lookup to the link target.
use Firefox to download directly from a mirror:
"http://mirror.hostsuisse.com/opensuse/distribution/leap/15.4/iso/openSUSE-Le..."
there is no redirect, and I get a file called "openSUSE-Leap-15.4-DVD-x86_64-Media.iso".
I am still unsure because the first redirection comes to the local file (not to a mirror), and I do not know whether this redirection is performed by mirrorbrain/MirrorCache or web server on download.o.o itself. But it does not really matter because the final name on mirrors will still include version.
The least intrusive change is to link all checksum files as well. Then users coming via browser will get (sort of) expected result
I think that's what I tried to say in one of my first responses in this thread, talking to Stephan.
As you just demonstrated this will break direct mirror access. The only clean fix is to avoid symlinks at all. Everything else is choosing the lest evil.
Am 14.09.22 um 17:47 schrieb Per Jessen:
Confirm - the difference is that the nonversioned iso file is a symlink, where as the nonversioned sha256 file is not. For whatever reason, mirrorbache sends an http302 to the symlinked file, which seems(!) to be the issue. If mirrorbache would just send a redirect to the original/correct name, nobody gets to see the versioned iso, which I believe is what we want. But the checksum file's content refers to the versioned iso - that's the original report is about. And that checksum file needs to be versioned as well for that reason.
Greetings, Stephan
On Thu, Sep 15, 2022 at 8:17 AM Stephan Kulow <coolo@suse.de> wrote:
Am 14.09.22 um 17:47 schrieb Per Jessen:
Confirm - the difference is that the nonversioned iso file is a symlink, where as the nonversioned sha256 file is not. For whatever reason, mirrorbache sends an http302 to the symlinked file, which seems(!) to be the issue. If mirrorbache would just send a redirect to the original/correct name, nobody gets to see the versioned iso, which I believe is what we want. But the checksum file's content refers to the versioned iso - that's the original report is about.
Actually it was exactly the opposite way. When you use download.o.o or get.o.o with your browser and request openSUSE-Leap-15.4-DVD-x86_64-Media.iso you get openSUSE-Leap-15.4-DVD-x86_64-Build243.2-Media.iso; and when you download openSUSE-Leap-15.4-DVD-x86_64-Media.iso.sha256 to verify checksum you get openSUSE-Leap-15.4-DVD-x86_64-Media.iso.sha256 with file name openSUSE-Leap-15.4-DVD-x86_64-Media.iso inside. Which does not match the ISO image that you just downloaded.
And that checksum file needs to be versioned as well for that reason.
Sure. There is also openSUSE-Leap-15.4-DVD-x86_64-Build243.2-Media.iso.sha256 which correctly matches openSUSE-Leap-15.4-DVD-x86_64-Build243.2-Media.iso. But that is not what is offered by get.o.o. I looked at Ubuntu and Fedora mirrors and none of them contain similar symlinks. Somehow they manage to offer only versioned images.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2022-09-15 at 07:16 +0200, Stephan Kulow wrote:
Am 14.09.22 um 17:47 schrieb Per Jessen:
Confirm - the difference is that the nonversioned iso file is a symlink, where as the nonversioned sha256 file is not. For whatever reason, mirrorbache sends an http302 to the symlinked file, which seems(!) to be the issue. If mirrorbache would just send a redirect to the original/correct name, nobody gets to see the versioned iso, which I believe is what we want. But the checksum file's content refers to the versioned iso - that's the original report is about. And that checksum file needs to be versioned as well for that reason.
I think that was going on months ago, but not now, I was unable to reproduce. Using wget or firefox, I get: cer@Telcontar:~/Download/Firefox_downloads> l *sha256 - -rw-r--r-- 1 cer users 630 Aug 4 2019 openSUSE-Leap-15.1-DVD-x86_64.iso.sha256 - -rw-r--r-- 1 cer users 628 Sep 3 2020 openSUSE-Leap-15.2-DVD-x86_64.iso.sha256 - -rw-r--r-- 1 cer users 105 Sep 14 13:29 openSUSE-Leap-15.4-DVD-x86_64-Media.iso.sha256 cer@Telcontar:~/Download/Firefox_downloads> cat openSUSE-Leap-15.4-DVD-x86_64-Media.iso.sha256 4683345f242397c7fd7d89a50731a120ffd60a24460e21d2634e783b3c169695 openSUSE-Leap-15.4-DVD-x86_64-Media.iso cer@Telcontar:~/Download/Firefox_downloads> wget: cer@Telcontar:/data/storage_b/Isos/Leap/15.4/greg/tmp2> l *sha256 - -rw-r--r-- 1 cer users 117 Sep 14 14:46 openSUSE-Leap-15.4-DVD-x86_64-Build243.2-Media.iso.sha256 - -rw-r--r-- 1 cer users 105 Sep 14 14:45 openSUSE-Leap-15.4-DVD-x86_64-Media.iso.sha256 cer@Telcontar:/data/storage_b/Isos/Leap/15.4/greg/tmp2> cat *sha256 4683345f242397c7fd7d89a50731a120ffd60a24460e21d2634e783b3c169695 openSUSE-Leap-15.4-DVD-x86_64-Build243.2-Media.iso 4683345f242397c7fd7d89a50731a120ffd60a24460e21d2634e783b3c169695 openSUSE-Leap-15.4-DVD-x86_64-Media.iso cer@Telcontar:/data/storage_b/Isos/Leap/15.4/greg/tmp2> Both files contain the corresponding correct line, AFAIKS. The problem is that when clicking on the download button with a web browser on <https://get.opensuse.org/leap/15.4/#download>, we get the versioned iso file, but the unversioned checksum file. - -- Cheers, Carlos E. R. (from openSUSE 15.3 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCYyL5YBwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVL2MAoIqGSI2QHoyLL6aJiO0n /IJVcoveAJ9kKiGc+/xuvkPkXgPidhmGPtcl0g== =LMJw -----END PGP SIGNATURE-----
Andrei Borzenkov wrote:
On 14.09.2022 18:34, Per Jessen wrote:
Hmm, wait. The issue is the 302 redirect to the versioned file, isn't it? When a user requests :
"d.o.o:/distribution/leap/15.4/iso/openSUSE-Leap-15.4-DVD-x86_64-Media.iso"
he is given an http 302 redirect to:
"d.o.o:/distribution/leap/15.4/iso/openSUSE-Leap-15.4-DVD-x86_64-Build243.2-Media.iso"
and eventually also to the mirror with that file. Unfortunately clients don't agree on how to work with that.
I wonder if this is something specific to mirrorbrain/cache ? When I
Yes, both mirrorbrain and MirrorCache explicitly resolve symlinks and apply mirror lookup to the link target.
I guess there is a reason for that, but I can't imagine what it is.
use Firefox to download directly from a mirror:
"http://mirror.hostsuisse.com/opensuse/distribution/leap/15.4/iso/openSUSE-Le..."
there is no redirect, and I get a file called "openSUSE-Leap-15.4-DVD-x86_64-Media.iso".
I am still unsure because the first redirection comes to the local file (not to a mirror), and I do not know whether this redirection is performed by mirrorbrain/MirrorCache or web server on download.o.o itself. But it does not really matter because the final name on mirrors will still include version.
I've already checked the apache config pontifex, I'll do it again.
The only clean fix is to avoid symlinks at all. Everything else is choosing the lest evil.
It still seems to me we just ought to avoid that initial 302 redirect - it works for all the mirrors. -- Per Jessen, Zürich (20.1°C) Member, openSUSE Heroes
Per Jessen wrote:
Andrei Borzenkov wrote:
I am still unsure because the first redirection comes to the local file (not to a mirror), and I do not know whether this redirection is performed by mirrorbrain/MirrorCache or web server on download.o.o itself. But it does not really matter because the final name on mirrors will still include version.
I've already checked the apache config pontifex, I'll do it again.
I have gone over the download.o.o apache config and I see nothing that does any redirection or rewriting based on symlinks. I can only conclude it is being done by mirrorbrain/mirrorcache. -- Per Jessen, Zürich (16.6°C) Member, openSUSE Heroes
On 15.09.2022 20:34, Per Jessen wrote:
Per Jessen wrote:
Andrei Borzenkov wrote:
I am still unsure because the first redirection comes to the local file (not to a mirror), and I do not know whether this redirection is performed by mirrorbrain/MirrorCache or web server on download.o.o itself. But it does not really matter because the final name on mirrors will still include version.
I've already checked the apache config pontifex, I'll do it again.
I have gone over the download.o.o apache config and I see nothing that does any redirection or rewriting based on symlinks. I can only conclude it is being done by mirrorbrain/mirrorcache.
For MirrorCache it apparently was added relatively recently: commit 90e6bf65e20c49a96a974b1a504a2dcea750675f Author: Andrii Nikitin <46994839+andrii-suse@users.noreply.github.com> Date: Thu Dec 9 07:09:53 2021 +0100 Special handling unversioned media symlinks (#235) * Remove outdated variable MIRRORCACHE_COUNTRY_RESCAN_TIMEOUT * Improve test 04-remote-link * Try to use redirect for unversioned media symlinks * Track symlink with hashes for unversioned media files if in the same folder * Do not calcuclate hashes for symlinks I won't claim to understand what MirrorCache tries to do it for, but it only does it for two files - -Media and -Current: +sub _detect_ln { + my ($dir, $file) = @_; + return undef unless $file && $file =~ m/.*(Media|Current)\.iso(\.sha256)?/; + Tests added in the same commit expect that sha256 files are links +echo now change the symlink and make sure redirect changes +( + cd $ap9/dt/folder1 + ln -sf file2.1-Media.iso file-Media.iso + ln -sf file2.1-Media.iso.sha256 file-Media.iso.sha256 +) +$mc/backstage/job -e folder_sync -a '["/folder1"]' +$mc/backstage/shoot +$mc/curl -I /download/folder1/file-Media.iso | grep -C 10 302 | grep /download/folder1/file2.1-Media.iso +$mc/curl -I /download/folder1/file-Media.iso.sha256 | grep -C 10 302 | grep /download/folder1/file2.1-Media.iso.sha256 +$mc/curl -L /download/folder1/file-Media.iso.sha256 | grep -q "2019dd7afaf5759c68cec4d0e7553227657f01c69da168489116a1c48e40270e "
Andrei Borzenkov wrote:
On 15.09.2022 20:34, Per Jessen wrote:
I have gone over the download.o.o apache config and I see nothing that does any redirection or rewriting based on symlinks. I can only conclude it is being done by mirrorbrain/mirrorcache.
For MirrorCache it apparently was added relatively recently:
Thanks Andrei, that's a nice find!!
I won't claim to understand what MirrorCache tries to do it for,
Me neither :-( I have opened a ticket - https://progress.opensuse.org/issues/116638 -- Per Jessen, Zürich (17.3°C) Member, openSUSE Heroes (2016-present) We're hiring - https://en.opensuse.org/openSUSE:Heroes
On 15.09.22 20:38, Andrei Borzenkov wrote:
I won't claim to understand what MirrorCache tries to do it for, but it only does it for two files - -Media and -Current:
my guess is that it tries to check that the redirect actually points to a current version. Theoretical scenario: Latest release: current -> build.123 outdated mirror: current -> build.111 now if download.o.o would just send to "mirror/current" the user would download an outdated build. Even more surprise if the iso would come from a different mirror than the checksum file, one outdated and the other not. So my guess is the idea behind this algorithm is to avoid such surprises. Now maybe one way to avoid it would be to *check* the symlink if it points to the correct file, but then still redirect to the "correct" (whatever that is, I don't really dig the whole ordeal anymore ;-)) location. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
participants (10)
-
Andreas Schwab -
Andrei Borzenkov -
Axel Braun -
Carlos E. R. -
Freek de Kruijf -
Luna Jernberg -
Per Jessen -
Stefan Brüns -
Stefan Seyfried -
Stephan Kulow