From hpj@urpla.net Mon Dec 23 09:20:04 2019
From: Hans-Peter Jansen <hpj@urpla.net>
To: buildservice@lists.opensuse.org
Subject: [opensuse-buildservice] broken builds due to "interconnect error: rpc
 timeout" with PMBS
Date: Mon, 23 Dec 2019 10:19:53 +0100
Message-ID: <10450735.8oIF6szpNo@xrated>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============2550600834934598533=="

--===============2550600834934598533==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hi,

I stumbled across broken builds in a local OBS due to "interconnect error: rp=
c=20
timeout" for packages, that link to PMBS.

PMBS is setup as:

<project name=3D"Packman">
  <title>Packman Build Service PMBS</title>
  <description>This instance can be used to access resources from Packman.</
description>
  <remoteurl>https://pmbs-api.links2linux.de/public</remoteurl>
</project>

OBS is fully updated from OBS:/Server:/2.10/openSUSE_15.1.

The magic spell, that comes closest to mitigate the issue *ought* to be:

    obs_admin --recheck-dod https://pmbs-api.links2linux.de

or

    obs_admin --recheck-dod https://pmbs-api.links2linux.de/public

but even after creating the missing /srv/obs/dods directory, things still=20
misbehave.

Is this a PMBS issue, a local configuration issue, or something completely=20
different?!?

Thanks in advance,

Pete


--=20
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

--===============2550600834934598533==--


From hpj@urpla.net Wed Dec 25 17:37:06 2019
From: Hans-Peter Jansen <hpj@urpla.net>
To: buildservice@lists.opensuse.org
Subject: Re: [opensuse-buildservice] broken builds due to "interconnect error:
 rpc timeout" with PMBS
Date: Wed, 25 Dec 2019 18:37:02 +0100
Message-ID: <2685642.tkDjrHWLbm@xrated>
In-Reply-To: <10450735.8oIF6szpNo@xrated>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1651058294461570027=="

--===============1651058294461570027==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Am Montag, 23. Dezember 2019, 10:19:53 CET schrieb Hans-Peter Jansen:
> Hi,
> 
> I stumbled across broken builds in a local OBS due to "interconnect error:
> rpc timeout" for packages, that link to PMBS.
> 
> PMBS is setup as:
> 
> <project name="Packman">
>   <title>Packman Build Service PMBS</title>
>   <description>This instance can be used to access resources from Packman.</
> description>
>   <remoteurl>https://pmbs-api.links2linux.de/public</remoteurl>
> </project>
> 
> OBS is fully updated from OBS:/Server:/2.10/openSUSE_15.1.
> 
> The magic spell, that comes closest to mitigate the issue *ought* to be:
> 
>     obs_admin --recheck-dod https://pmbs-api.links2linux.de
> 
> or
> 
>     obs_admin --recheck-dod https://pmbs-api.links2linux.de/public
> 
> but even after creating the missing /srv/obs/dods directory, things still
> misbehave.
> 
> Is this a PMBS issue, a local configuration issue, or something completely
> different?!?

Anything wrong with this question? 

Any help solving this issue is much appreciated.

Thanks,
Pete


-- 
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

--===============1651058294461570027==--


From ngompa13@gmail.com Wed Dec 25 17:41:30 2019
From: Neal Gompa <ngompa13@gmail.com>
To: buildservice@lists.opensuse.org
Subject: Re: [opensuse-buildservice] broken builds due to "interconnect error:
 rpc timeout" with PMBS
Date: Wed, 25 Dec 2019 12:40:51 -0500
Message-ID:
 <CAEg-Je8F-_3gv79ffcZrrFd=sTKgRdvQjopHPLU53ALuXq=z8w@mail.gmail.com>
In-Reply-To: <2685642.tkDjrHWLbm@xrated>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============5291980259964853844=="

--===============5291980259964853844==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

On Wed, Dec 25, 2019 at 12:37 PM Hans-Peter Jansen <hpj(a)urpla.net> wrote:
>
> Am Montag, 23. Dezember 2019, 10:19:53 CET schrieb Hans-Peter Jansen:
> > Hi,
> >
> > I stumbled across broken builds in a local OBS due to "interconnect error:
> > rpc timeout" for packages, that link to PMBS.
> >
> > PMBS is setup as:
> >
> > <project name=3D"Packman">
> >   <title>Packman Build Service PMBS</title>
> >   <description>This instance can be used to access resources from Packman=
.</
> > description>
> >   <remoteurl>https://pmbs-api.links2linux.de/public</remoteurl>
> > </project>
> >
> > OBS is fully updated from OBS:/Server:/2.10/openSUSE_15.1.
> >
> > The magic spell, that comes closest to mitigate the issue *ought* to be:
> >
> >     obs_admin --recheck-dod https://pmbs-api.links2linux.de
> >
> > or
> >
> >     obs_admin --recheck-dod https://pmbs-api.links2linux.de/public
> >
> > but even after creating the missing /srv/obs/dods directory, things still
> > misbehave.
> >
> > Is this a PMBS issue, a local configuration issue, or something completely
> > different?!?
>
> Anything wrong with this question?
>

This is a PMBS issue. The API endpoint for PMBS is probably misbehaving.

--=20
=E7=9C=9F=E5=AE=9F=E3=81=AF=E3=81=84=E3=81=A4=E3=82=82=E4=B8=80=E3=81=A4=EF=
=BC=81/ Always, there's only one truth!
--=20
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

--===============5291980259964853844==--


From hpj@urpla.net Thu Dec 26 18:14:03 2019
From: Hans-Peter Jansen <hpj@urpla.net>
To: buildservice@lists.opensuse.org
Subject: Re: [opensuse-buildservice] broken builds due to "interconnect error:
 rpc timeout" with PMBS
Date: Thu, 26 Dec 2019 19:13:56 +0100
Message-ID: <3291644.catU4ZWl6l@xrated>
In-Reply-To:
 <CAEg-Je8F-_3gv79ffcZrrFd=sTKgRdvQjopHPLU53ALuXq=z8w@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0986990883484371620=="

--===============0986990883484371620==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Am Mittwoch, 25. Dezember 2019, 18:40:51 CET schrieb Neal Gompa:
> On Wed, Dec 25, 2019 at 12:37 PM Hans-Peter Jansen <hpj(a)urpla.net> wrote:
> > Am Montag, 23. Dezember 2019, 10:19:53 CET schrieb Hans-Peter Jansen:
> > > Hi,
> > >=20
> > > I stumbled across broken builds in a local OBS due to "interconnect
> > > error:
> > > rpc timeout" for packages, that link to PMBS.
> > >=20
> > > PMBS is setup as:
> > >=20
> > > <project name=3D"Packman">
> > >=20
> > >   <title>Packman Build Service PMBS</title>
> > >   <description>This instance can be used to access resources from
> > >   Packman.</
> > >=20
> > > description>
> > >=20
> > >   <remoteurl>https://pmbs-api.links2linux.de/public</remoteurl>
> > >=20
> > > </project>
> > >=20
> > > OBS is fully updated from OBS:/Server:/2.10/openSUSE_15.1.
> > >=20
> > > The magic spell, that comes closest to mitigate the issue *ought* to be:
> > >     obs_admin --recheck-dod https://pmbs-api.links2linux.de
> > >=20
> > > or
> > >=20
> > >     obs_admin --recheck-dod https://pmbs-api.links2linux.de/public
> > >=20
> > > but even after creating the missing /srv/obs/dods directory, things
> > > still
> > > misbehave.
> > >=20
> > > Is this a PMBS issue, a local configuration issue, or something
> > > completely
> > > different?!?
> >=20
> > Anything wrong with this question?
>=20
> This is a PMBS issue. The API endpoint for PMBS is probably misbehaving.

Thanks Neal, for your answer, but simply fetching the public URL looks pretty=
=20
similar, compared to openSUSE.org:

# openSUSE.org

$ wget https://api.opensuse.org/public
--2019-12-26 18:47:45--  https://api.opensuse.org/public
Aufl=C3=B6sen des Hostnamens api.opensuse.org (api.opensuse.org)=E2=80=A6 195=
.135.221.133,=20
2620:113:80c0:8::12
Verbindungsaufbau zu api.opensuse.org (api.opensuse.org)|195.135.221.133|:443=
=20
=E2=80=A6 verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet =E2=80=A6 302 Found
Platz: https://api.opensuse.org/about [folgend]
--2019-12-26 18:47:45--  https://api.opensuse.org/about
Wiederverwendung der bestehenden Verbindung zu api.opensuse.org:443.
HTTP-Anforderung gesendet, auf Antwort wird gewartet =E2=80=A6 200 OK
L=C3=A4nge: nicht spezifiziert [application/xml]
Wird in =C2=BBpublic=C2=AB gespeichert.

public                                      [ <=3D>                          =
                                                 =20
]     300  --.-KB/s    in 0s     =20

2019-12-26 18:47:45 (13,3 MB/s) - =C2=BBpublic=C2=AB gespeichert [300]

$ cat public
<about>
  <title>Open Build Service API</title>
  <description>API to the Open Build Service</description>
  <revision>2.11~alpha.20191218T101855.4c27b561b8</revision>
  <last_deployment>2019-12-18 10:12:06 +0000</last_deployment>
  <commit>4c27b561b88c09a29babc95f0cdcaf72046876c2</commit>
</about>

# PMBS

$ wget https://pmbs-api.links2linux.de/public
--2019-12-26 18:48:08--  https://pmbs-api.links2linux.de/public
Aufl=C3=B6sen des Hostnamens pmbs-api.links2linux.de (pmbs-api.links2linux.de=
)=E2=80=A6=20
95.216.108.108
Verbindungsaufbau zu pmbs-api.links2linux.de (pmbs-api.links2linux.de)|
95.216.108.108|:443 =E2=80=A6 verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet =E2=80=A6 302 Found
Platz: https://pmbs-api.links2linux.de/about [folgend]
--2019-12-26 18:48:08--  https://pmbs-api.links2linux.de/about
Wiederverwendung der bestehenden Verbindung zu pmbs-api.links2linux.de:443.
HTTP-Anforderung gesendet, auf Antwort wird gewartet =E2=80=A6 200 OK
L=C3=A4nge: nicht spezifiziert [application/xml]
Wird in =C2=BBpublic.1=C2=AB gespeichert.

public.1                                    [ <=3D>                          =
                                                 =20
]     267  --.-KB/s    in 0s     =20

2019-12-26 18:48:08 (22,5 MB/s) - =C2=BBpublic.1=C2=AB gespeichert [267]

$ cat public.1=20
<about>
  <title>Open Build Service API</title>
  <description>API to the Open Build Service</description>
  <revision>2.10</revision>
  <last_deployment>2019-07-05 14:35:37 +0200</last_deployment>
  <commit>1b66a6c499ef52f61406c333d92731323d852fbc</commit>
</about>

Sure, this doesn't tell anything about specific API access to specific=20
packages.=20

/srv/obs/log/scheduler_x86_64.log shows this:

2019-12-26 18:07:17: [2762]  waiting for an event...
response from watcher for https://pmbs-api.links2linux.de/public
400 remote error: document element must be 'events', was 'html'
retrying in 60 seconds

Cheers,
Pete


--=20
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

--===============0986990883484371620==--


From stefan@botter.cc Thu Dec 26 19:05:55 2019
From: Stefan Botter <stefan@botter.cc>
To: buildservice@lists.opensuse.org
Subject: Re: [opensuse-buildservice] broken builds due to "interconnect error:
 rpc timeout" with PMBS
Date: Thu, 26 Dec 2019 20:05:47 +0100
Message-ID: <1749403.Fz2aqmyv1s@blackium>
In-Reply-To:
 <CAEg-Je8F-_3gv79ffcZrrFd=sTKgRdvQjopHPLU53ALuXq=z8w@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============4700564518755397395=="

--===============4700564518755397395==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Hi Neal,

On Wednesday, December 25, 2019 6:40:51 PM CET Neal Gompa wrote:
> This is a PMBS issue. The API endpoint for PMBS is probably
> misbehaving.

Possible, but not really probable. Do you have any pointers to look for 
'misbehaving'?

Thank you,

Stefan

--===============4700564518755397395==
Content-Type: application/pgp-signature
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="signature.asc"
MIME-Version: 1.0

LS0tLS1CRUdJTiBQR1AgU0lHTkFUVVJFLS0tLS0KCmlRRXpCQUFCQ2dBZEZpRUVtdXhwZ0Z4QUJR
bHIxN2pWYjk4WUVadkJNOVlGQWw0RkJJc0FDZ2tRYjk4WUVadkIKTTliaHRnZ0F5Vlg1WDllTzR0
RmlSZ25MdHYyYUc3cmw1KzVmeUx6M2FKeDBON092QjNlcitHT1BtaXNzSW83UwpGL0VhQUE0SEx0
a0JXWmMzYUoybUpjSDhldk5YTUV3eFRjMXVEUXRSaEdLV1BmL3FTaFhzSmFhNzFBS2dRT0tVCkx4
ekdLaDB4dTRqNGtzalV6UHNUSXpQTytCMzJlaEZsNE1NcGorS0d4U0FuZHRZMm16Tk50dzlVcTJ6
N1JheloKT0pndWZPMTd1MnJRcjI5UzBKTDN4NzJGYWhpTFcvUER3VUZnQWZsYXhveUNmeE4zZWFx
V2p4WFB2UU5GRzBPbwoxYzIrMDd5TDdGdU53d3lCQ2VuemZ3YTd4S1hhbTJqelY1K2NrU2hSN3ZX
dWJNY0VPbjBjV0Z3eU0yUFJhbzlYCjFIWWIzRnhTWGxiRG0zNDdyRkt3WmJnb1BQU3cxdz09Cj1p
WURaCi0tLS0tRU5EIFBHUCBTSUdOQVRVUkUtLS0tLQo=

--===============4700564518755397395==--


From jengelh@inai.de Thu Dec 26 20:16:06 2019
From: Jan Engelhardt <jengelh@inai.de>
To: buildservice@lists.opensuse.org
Subject: Re: [opensuse-buildservice] broken builds due to "interconnect error:
 rpc timeout" with PMBS
Date: Thu, 26 Dec 2019 21:16:04 +0100
Message-ID: <nycvar.YFH.7.76.1912262113310.29797@n3.vanv.qr>
In-Reply-To: <3291644.catU4ZWl6l@xrated>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============6687010764380694609=="

--===============6687010764380694609==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


On Thursday 2019-12-26 19:13, Hans-Peter Jansen wrote:
>
>/srv/obs/log/scheduler_x86_64.log shows this:
>
>2019-12-26 18:07:17: [2762]  waiting for an event...
>response from watcher for https://pmbs-api.links2linux.de/public
>400 remote error: document element must be 'events', was 'html'
>retrying in 60 seconds

HTML error page of some kind (i.e. of the "readable" kind that you could open
with your browser too, not some XML that needs a special parser). opensuse.org
no longer seems to like the request that PMBS made. Hooray for API stability,
hooray for... not seeing any of the HTTP headers sent or received :-/
-- 
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

--===============6687010764380694609==--


From hpj@urpla.net Fri Dec 27 13:28:05 2019
From: Hans-Peter Jansen <hpj@urpla.net>
To: buildservice@lists.opensuse.org
Subject: Re: [opensuse-buildservice] broken builds due to "interconnect error:
 rpc timeout" with PMBS
Date: Fri, 27 Dec 2019 14:28:01 +0100
Message-ID: <4638222.v36FrtDR5o@xrated>
In-Reply-To: <nycvar.YFH.7.76.1912262113310.29797@n3.vanv.qr>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============5030774086034325872=="

--===============5030774086034325872==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hi Jan,

Am Donnerstag, 26. Dezember 2019, 21:16:04 CET schrieb Jan Engelhardt:
> On Thursday 2019-12-26 19:13, Hans-Peter Jansen wrote:
> >/srv/obs/log/scheduler_x86_64.log shows this:
> >
> >2019-12-26 18:07:17: [2762]  waiting for an event...
> >response from watcher for https://pmbs-api.links2linux.de/public
> >400 remote error: document element must be 'events', was 'html'
> >retrying in 60 seconds
>=20
> HTML error page of some kind (i.e. of the "readable" kind that you could
> open with your browser too, not some XML that needs a special parser).
> opensuse.org no longer seems to like the request that PMBS made. Hooray for
> API stability, hooray for... not seeing any of the HTTP headers sent or
> received :-/

Another interesting observation: it indeed makes a difference, if I access th=
e=20
public path of OBS and PMBS with curl or wget.=20

(All done from the local obs server):

$ wget -qO- https://pmbs-api.links2linux.de/public
<about>
  <title>Open Build Service API</title>
  <description>API to the Open Build Service</description>
  <revision>2.10</revision>
  <last_deployment>2019-07-05 14:35:37 +0200</last_deployment>
  <commit>1b66a6c499ef52f61406c333d92731323d852fbc</commit>
</about>

$ curl -o- https://pmbs-api.links2linux.de/public
<html><body>You are being <a href=3D"https://pmbs.links2linux.de/
about">redirected</a>.</body></html>

but the behavior is consistent with OBS.

I can provide many more details, but still have no idea, where these issues=20
stem from. It just shows any package build as broken, that links to PMBS.

I don't think, it's API related, it smells more like some transient issue,=20
that eventually turned chronic..

Cheers,
Pete


--=20
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

--===============5030774086034325872==--


From schwab@linux-m68k.org Fri Dec 27 22:02:17 2019
From: Andreas Schwab <schwab@linux-m68k.org>
To: buildservice@lists.opensuse.org
Subject: [opensuse-buildservice] Re: broken builds due to "interconnect error:
 rpc timeout" with PMBS
Date: Fri, 27 Dec 2019 23:02:13 +0100
Message-ID: <878smxbf6i.fsf@igel.home>
In-Reply-To: <4638222.v36FrtDR5o@xrated>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0719083370242120388=="

--===============0719083370242120388==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Dez 27 2019, Hans-Peter Jansen wrote:

> $ curl -o- https://pmbs-api.links2linux.de/public

You need curl -L.

Andreas.

-- 
Andreas Schwab, schwab(a)linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
-- 
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

--===============0719083370242120388==--


From hpj@urpla.net Sat Dec 28 00:56:21 2019
From: Hans-Peter Jansen <hpj@urpla.net>
To: buildservice@lists.opensuse.org
Subject: Re: [opensuse-buildservice] Re: broken builds due to "interconnect
 error: rpc timeout" with PMBS
Date: Sat, 28 Dec 2019 01:56:15 +0100
Message-ID: <1892468.9288UUijBz@xrated>
In-Reply-To: <878smxbf6i.fsf@igel.home>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1500007320642888171=="

--===============1500007320642888171==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Am Freitag, 27. Dezember 2019, 23:02:13 CET schrieb Andreas Schwab:
> On Dez 27 2019, Hans-Peter Jansen wrote:
> > $ curl -o- https://pmbs-api.links2linux.de/public
> 
> You need curl -L.



Thanks, Andreas:

curl -Lo- https://pmbs-api.links2linux.de/public
<about>
  <title>Open Build Service API</title>
  <description>API to the Open Build Service</description>
  <revision>2.10</revision>
  <last_deployment>2019-07-05 14:35:37 +0200</last_deployment>
  <commit>1b66a6c499ef52f61406c333d92731323d852fbc</commit>
</about>

but the underlying problem is still unresolved.

Do you have an idea, where these rpc timeouts for PMBS could stem from?

Cheers,
Pete


-- 
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

--===============1500007320642888171==--