[opensuse-buildservice] OBS gateway hiccup?
I get the following kind of error when accessing OBS for "osc build" or "osc ci". Building groupware-image-13.2.kiwi for images/x86_64 Getting buildinfo from server and store to /srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image/.osc/_buildinfo-images-x86_64.xml Server returned an error: HTTP Error 502: Bad Gateway However, osc up works. -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 19.01.2015 um 16:27 schrieb Ralf Lang:
I get the following kind of error when accessing OBS for "osc build" or "osc ci".
Building groupware-image-13.2.kiwi for images/x86_64 Getting buildinfo from server and store to /srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image/.osc/_buildinfo-images-x86_64.xml
Server returned an error: HTTP Error 502: Bad Gateway
That is usually a timeout. Greetings, Stephan - -- Ma muaß weiterkämpfen, kämpfen bis zum Umfalln, a wenn die ganze Welt an Arsch offen hat, oder grad deswegn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlS+AOMACgkQwFSBhlBjoJbqvgCgt8GjInSkdy08shTHqXJElMiT +5gAmwfYuMR4nByQ7g7yIW0MBtDLcGl6 =hQ44 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Montag, 19. Januar 2015, 16:27:12 wrote Ralf Lang:
I get the following kind of error when accessing OBS for "osc build" or "osc ci".
Building groupware-image-13.2.kiwi for images/x86_64 Getting buildinfo from server and store to /srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image/.osc/_buildinfo-images-x86_64.xml Server returned an error: HTTP Error 502: Bad Gateway
it is often a timeout error. And it is quite like here the case due to the stressful calculation of kiwi images (many package deps to solve). However, it works often when you repeat it after some time, because the result becomes cached. How long does it take until you get the error? -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 20.01.2015 08:31, Adrian Schröter wrote:
On Montag, 19. Januar 2015, 16:27:12 wrote Ralf Lang:
I get the following kind of error when accessing OBS for "osc build" or "osc ci".
Building groupware-image-13.2.kiwi for images/x86_64 Getting buildinfo from server and store to /srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image/.osc/_buildinfo-images-x86_64.xml Server returned an error: HTTP Error 502: Bad Gateway
it is often a timeout error. And it is quite like here the case due to the stressful calculation of kiwi images (many package deps to solve).
However, it works often when you repeat it after some time, because the result becomes cached.
How long does it take until you get the error?
It takes roughly 10 minutes. Getting buildinfo from server and store to /srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image/.osc/_buildinfo-images-x86_64.xml ^[[HServer returned an error: HTTP Error 502: Bad Gateway Request: https://api.opensuse.org/build/isv:B1-Systems:Horde5:opensuse-appliance/imag... Headers: content-language: en transfer-encoding: chunked accept-ranges: bytes strict-transport-security: max-age=31536000 vary: accept-language,accept-charset server: Apache/2.2.12 (Linux/SUSE) connection: close date: Tue, 20 Jan 2015 08:38:23 GMT content-type: text/html; charset=iso-8859-1 real 10m0.923s user 0m0.286s sys 0m0.046s -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On 20.01.2015 09:53, Ralf Lang wrote:
On 20.01.2015 08:31, Adrian Schröter wrote:
On Montag, 19. Januar 2015, 16:27:12 wrote Ralf Lang:
I get the following kind of error when accessing OBS for "osc build" or "osc ci".
Building groupware-image-13.2.kiwi for images/x86_64 Getting buildinfo from server and store to /srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image/.osc/_buildinfo-images-x86_64.xml Server returned an error: HTTP Error 502: Bad Gateway
it is often a timeout error. And it is quite like here the case due to the stressful calculation of kiwi images (many package deps to solve).
However, it works often when you repeat it after some time, because the result becomes cached.
How long does it take until you get the error?
Still failing as of today. The explanation makes sense for "osc build" but I do not understand why a "ci" to this repo should be complicated. -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On Dienstag, 20. Januar 2015, 10:40:59 wrote Ralf Lang:
On 20.01.2015 09:53, Ralf Lang wrote:
On 20.01.2015 08:31, Adrian Schröter wrote:
On Montag, 19. Januar 2015, 16:27:12 wrote Ralf Lang:
I get the following kind of error when accessing OBS for "osc build" or "osc ci".
Building groupware-image-13.2.kiwi for images/x86_64 Getting buildinfo from server and store to /srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image/.osc/_buildinfo-images-x86_64.xml Server returned an error: HTTP Error 502: Bad Gateway
it is often a timeout error. And it is quite like here the case due to the stressful calculation of kiwi images (many package deps to solve).
However, it works often when you repeat it after some time, because the result becomes cached.
How long does it take until you get the error?
Still failing as of today. The explanation makes sense for "osc build" but I do not understand why a "ci" to this repo should be complicated.
right. However, did you tried today to submit and got a 50x error? I do not see it in our logs then. Can you please retry via osc -d ci and send me the output please? -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Still failing as of today. The explanation makes sense for "osc build" but I do not understand why a "ci" to this repo should be complicated.
right. However, did you tried today to submit and got a 50x error?
I do not see it in our logs then.
Can you please retry via
osc -d ci
and send me the output please?
I tried that but now it's hanging "forever" (more than 20 minutes) rlang@rlang-t430b1:/srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2> osc -d ci GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... Sending root.tar.bz2 Sending groupware-image-13.2.kiwi POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2?comment=Image+for+13.2&cmd=commitfilelist&user=ralflangb1 -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On Dienstag, 20. Januar 2015, 12:31:44 wrote Ralf Lang:
Still failing as of today. The explanation makes sense for "osc build" but I do not understand why a "ci" to this repo should be complicated.
right. However, did you tried today to submit and got a 50x error?
I do not see it in our logs then.
Can you please retry via
osc -d ci
and send me the output please?
I tried that but now it's hanging "forever" (more than 20 minutes)
rlang@rlang-t430b1:/srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2> osc -d ci GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... Sending root.tar.bz2 Sending groupware-image-13.2.kiwi POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2?comment=Image+for+13.2&cmd=commitfilelist&user=ralflangb1
Is the root tar ball very large, so it is a timeout during upload? -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 20.01.2015 12:42, Adrian Schröter wrote:
On Dienstag, 20. Januar 2015, 12:31:44 wrote Ralf Lang:
Still failing as of today. The explanation makes sense for "osc build" but I do not understand why a "ci" to this repo should be complicated.
right. However, did you tried today to submit and got a 50x error?
I do not see it in our logs then.
Can you please retry via
osc -d ci
and send me the output please?
I tried that but now it's hanging "forever" (more than 20 minutes)
rlang@rlang-t430b1:/srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2> osc -d ci GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... Sending root.tar.bz2 Sending groupware-image-13.2.kiwi POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2?comment=Image+for+13.2&cmd=commitfilelist&user=ralflangb1
Is the root tar ball very large, so it is a timeout during upload?
No, it's rather small. -rw-r--r-- 1 rlang users 2041 19. Jan 15:12 groupware-image-13.2.kiwi -rw-r--r-- 1 rlang users 9052 8. Jan 16:10 root.tar.bz2 -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
and send me the output please?
I tried that but now it's hanging "forever" (more than 20 minutes)
rlang@rlang-t430b1:/srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2> osc -d ci GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... Sending root.tar.bz2 Sending groupware-image-13.2.kiwi POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2?comment=Image+for+13.2&cmd=commitfilelist&user=ralflangb1
Is the root tar ball very large, so it is a timeout during upload?
No, it's rather small. -rw-r--r-- 1 rlang users 2041 19. Jan 15:12 groupware-image-13.2.kiwi -rw-r--r-- 1 rlang users 9052 8. Jan 16:10 root.tar.bz2
It's weird. I have a timeout without -d but I have "it just never finishes" with -d rlang@rlang-t430b1:/srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2> osc ci Sending root.tar.bz2 Sending groupware-image-13.2.kiwi Server returned an error: HTTP Error 502: Bad Gateway Transmitting file data Request: https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2?comment=Image+for+13.2&cmd=commitfilelist&user=ralflangb1 Headers: content-language: en transfer-encoding: chunked accept-ranges: bytes strict-transport-security: max-age=31536000 vary: accept-language,accept-charset server: Apache/2.2.12 (Linux/SUSE) connection: close date: Tue, 20 Jan 2015 18:31:09 GMT content-type: text/html; charset=iso-8859-1 rlang@rlang-t430b1:/srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2> osc ci -d osc ci: no such option: -d Try 'osc help ci' for info. rlang@rlang-t430b1:/srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2> osc -d ci GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... Sending root.tar.bz2 Sending groupware-image-13.2.kiwi POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2?comment=Image+for+13.2&cmd=commitfilelist&user=ralflangb1 -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On 2015-01-20 19:57:37 +0100, Ralf Lang wrote:
and send me the output please?
I tried that but now it's hanging "forever" (more than 20 minutes)
rlang@rlang-t430b1:/srv/obs/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2> osc -d ci GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance GET https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/gro... Sending root.tar.bz2 Sending groupware-image-13.2.kiwi POST https://api.opensuse.org/source/isv:B1-Systems:Horde5:opensuse-appliance/groupware-image-13.2?comment=Image+for+13.2&cmd=commitfilelist&user=ralflangb1
Is the root tar ball very large, so it is a timeout during upload?
No, it's rather small. -rw-r--r-- 1 rlang users 2041 19. Jan 15:12 groupware-image-13.2.kiwi -rw-r--r-- 1 rlang users 9052 8. Jan 16:10 root.tar.bz2
It's weird. I have a timeout without -d but I have "it just never finishes" with -d
That's strange, because "-d" does not influence the actual http request code - it just prints some debug output. Did you try to strace the osc process to see where it gets stuck? (Be careful when posting the strace output, because it can contain your credentials.) Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 21.01.2015 15:40, Marcus Hüwe wrote:
On 2015-01-20 19:57:37 +0100, Ralf Lang wrote:
Is the root tar ball very large, so it is a timeout during upload?
No, it's rather small. -rw-r--r-- 1 rlang users 2041 19. Jan 15:12 groupware-image-13.2.kiwi -rw-r--r-- 1 rlang users 9052 8. Jan 16:10 root.tar.bz2
It's weird. I have a timeout without -d but I have "it just never finishes" with -d
That's strange, because "-d" does not influence the actual http request code - it just prints some debug output. Did you try to strace the osc process to see where it gets stuck? (Be careful when posting the strace output, because it can contain your credentials.)
Marcus
Hi Marcus, see attached trace. It stays "forever" in a socket read. I do not know obs well enough to understand the semantics of this communication and where exactly it breaks. It seems to expect some server output which does not arrive. I have not found where my credentials are sent so I will habitually change my password tomorrow. -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Hi Marcus, see attached trace. It stays "forever" in a socket read.
I do not know obs well enough to understand the semantics of this communication and where exactly it breaks. It seems to expect some server output which does not arrive.
I have not found where my credentials are sent so I will habitually change my password tomorrow.
I think I found the problem: Just now, the upload failed when I had <!-- --!> markers inside but worked when I removed these lines. It should be better if a ci would emit some kind of error instead of just doing nothing but it will take me a while to understand obs code and provide a patch. It's also not top of my list, sadly :( -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On Freitag, 23. Januar 2015, 10:05:11 wrote Ralf Lang:
Hi Marcus, see attached trace. It stays "forever" in a socket read.
I do not know obs well enough to understand the semantics of this communication and where exactly it breaks. It seems to expect some server output which does not arrive.
I have not found where my credentials are sent so I will habitually change my password tomorrow.
I think I found the problem:
Just now, the upload failed when I had <!-- --!> markers inside but worked when I removed these lines.
you had them inside of what exactly?
It should be better if a ci would emit some kind of error instead of just doing nothing but it will take me a while to understand obs code and provide a patch. It's also not top of my list, sadly :(
-- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 23.01.2015 10:06, Adrian Schröter wrote:
On Freitag, 23. Januar 2015, 10:05:11 wrote Ralf Lang:
Hi Marcus, see attached trace. It stays "forever" in a socket read.
I do not know obs well enough to understand the semantics of this communication and where exactly it breaks. It seems to expect some server output which does not arrive.
I have not found where my credentials are sent so I will habitually change my password tomorrow.
I think I found the problem:
Just now, the upload failed when I had <!-- --!> markers inside but worked when I removed these lines.
you had them inside of what exactly?
Inside repository section of a kiwi file <repository type="rpm-md"> <source path="obs://openSUSE:13.2/standard"/> <!-- <source path="obs://isv:B1-Systems:Horde5:rolling/openSUSE_13.2"/> <source path="obs://isv:B1-Systems:Horde5:dependencies/openSUSE_13.2"/> --!> </repository> -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On Freitag, 23. Januar 2015, 10:11:32 wrote Ralf Lang:
On 23.01.2015 10:06, Adrian Schröter wrote:
On Freitag, 23. Januar 2015, 10:05:11 wrote Ralf Lang:
Hi Marcus, see attached trace. It stays "forever" in a socket read.
I do not know obs well enough to understand the semantics of this communication and where exactly it breaks. It seems to expect some server output which does not arrive.
I have not found where my credentials are sent so I will habitually change my password tomorrow.
I think I found the problem:
Just now, the upload failed when I had <!-- --!> markers inside but worked when I removed these lines.
you had them inside of what exactly?
Inside repository section of a kiwi file
<repository type="rpm-md"> <source path="obs://openSUSE:13.2/standard"/> <!-- <source path="obs://isv:B1-Systems:Horde5:rolling/openSUSE_13.2"/> <source path="obs://isv:B1-Systems:Horde5:dependencies/openSUSE_13.2"/> --!> </repository>
hm, I would take care, but I can not reproduce this here ... -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
<repository type="rpm-md"> <source path="obs://openSUSE:13.2/standard"/> <!-- <source path="obs://isv:B1-Systems:Horde5:rolling/openSUSE_13.2"/> <source path="obs://isv:B1-Systems:Horde5:dependencies/openSUSE_13.2"/> --!> </repository>
It may be coincidence that it started working just as I removed that markup. I will bring this up when it happens again. -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
participants (4)
-
Adrian Schröter -
Marcus Hüwe -
Ralf Lang -
Stephan Kulow