[opensuse-buildservice] Crazy hermes...
Hi... Today hermes sent me 14 emails about submit-requests and accept messages from myself at the same time 6:16pm...while those activities actually happened two days ago. I think our experts may have to look at it...actually: * Since the very beginning of this year, I can't receive any notifications on time...eg, I do something on OBS, notifications will arrive one or two days later...even weeks later...And it will always bomb you by sending lots of...I think they may stuck somewhere... * I can't receive any notification about SRs against my repository...eg: someone SRed to M17N:fonts...I can't be aware of it until I login through web UI...same thing happened to my rejected requests and accepted requests made by others...well the only type of SR among such "response needed" types I can receive is the "review request". * I always receive two emails telling me about same thing at the same time: ** One is for status change, while the other is just a copy of it. ** One tells me there is a request, while the other tells me I've already accepted it. (should they arrive at the same time?) While maybe it's time to invest in new hardware? ...delayed notification is spam-like right? or hermes has been changed to an "archiving" system? Greetings Marguerite -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Marguerite Su writes:
* I always receive two emails telling me about same thing at the same time: ** One is for status change, while the other is just a copy of it. ** One tells me there is a request, while the other tells me I've already accepted it. (should they arrive at the same time?)
I'd guess you receive one mail for each role you have wrt the request. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 11.11.2013 12:37, Marguerite Su wrote:
Hi...
Today hermes sent me 14 emails about submit-requests and accept messages from myself at the same time 6:16pm...while those activities actually happened two days ago.
Yes, I fixed the notifications this morning - it was broken since 7th as you can see on https://hermes.opensuse.org/statistics
I think our experts may have to look at it...actually:
* Since the very beginning of this year, I can't receive any notifications on time...eg, I do something on OBS, notifications will arrive one or two days later...even weeks later...And it will always bomb you by sending lots of...I think they may stuck somewhere...
It's news to me that this is the generic behaviour - today is special as clearly visible in the statistics graph.
* I can't receive any notification about SRs against my repository...eg: someone SRed to M17N:fonts...I can't be aware of it until I login through web UI...same thing happened to my rejected requests and accepted requests made by others...well the only type of SR among such "response needed" types I can receive is the "review request". * I always receive two emails telling me about same thing at the same time: ** One is for status change, while the other is just a copy of it. ** One tells me there is a request, while the other tells me I've already accepted it. (should they arrive at the same time?)
You really need to check your subscriptions.
While maybe it's time to invest in new hardware? ...delayed notification is spam-like right? or hermes has been changed to an "archiving" system?
Yes, it's time to work on the notification system. It's actually time since quite some years, but nobody did it ;( Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 11.11.2013 13:04, Stephan Kulow wrote: Hi,
While maybe it's time to invest in new hardware? ...delayed notification is spam-like right? or hermes has been changed to an "archiving" system? Yes, it's time to work on the notification system. It's actually time since quite some years, but nobody did it ;( Now you exaggerate ;-)
regards, Klaas -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mon, Nov 11, 2013 at 8:04 PM, Stephan Kulow
* Since the very beginning of this year, I can't receive any notifications on time...eg, I do something on OBS, notifications will arrive one or two days later...even weeks later...And it will always bomb you by sending lots of...I think they may stuck somewhere... It's news to me that this is the generic behaviour - today is special as clearly visible in the statistics graph.
Hi, coolo... Just trust me... http://paste.opensuse.org/82409309 Basically this can help you know the frequency and style Hermes bombs me... It either don't notify at all, or notify at the same time (sadly I don't know when) no matter when the SRs were actually created. eg: If I did something at day 1, another something at day 2, those SRs will arrive on day 3 or day 4, at the same time, since the very beginning of this year.
* I can't receive any notification about SRs against my repository...eg: someone SRed to M17N:fonts...I can't be aware of it until I login through web UI...same thing happened to my rejected requests and accepted requests made by others...well the only type of SR among such "response needed" types I can receive is the "review request". * I always receive two emails telling me about same thing at the same time: ** One is for status change, while the other is just a copy of it. ** One tells me there is a request, while the other tells me I've already accepted it. (should they arrive at the same time?)
You really need to check your subscriptions.
Well...is this normal setting? http://paste.opensuse.org/25243677 Marguerite -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 12.11.2013 00:51, Marguerite Su wrote:
On Mon, Nov 11, 2013 at 8:04 PM, Stephan Kulow
wrote: * Since the very beginning of this year, I can't receive any notifications on time...eg, I do something on OBS, notifications will arrive one or two days later...even weeks later...And it will always bomb you by sending lots of...I think they may stuck somewhere... It's news to me that this is the generic behaviour - today is special as clearly visible in the statistics graph.
Hi, coolo...
Just trust me... Oh, I'm far from disbelieving hermes being broken - just ask Klaas ;) But that very behaviour is indeed news to me.
http://paste.opensuse.org/82409309 I don't actually spot anything unusual if you ignore that all mails from 7th-11th arrived on 11st, but as I said: that's the exception.
Basically this can help you know the frequency and style Hermes bombs me...
It either don't notify at all, or notify at the same time (sadly I don't know when) no matter when the SRs were actually created.
eg: If I did something at day 1, another something at day 2, those SRs will arrive on day 3 or day 4, at the same time, since the very beginning of this year.
The mail headers of a day 3 mail would be more interesting than your above screenshot.
* I can't receive any notification about SRs against my repository...eg: someone SRed to M17N:fonts...I can't be aware of it until I login through web UI...same thing happened to my rejected requests and accepted requests made by others...well the only type of SR among such "response needed" types I can receive is the "review request". * I always receive two emails telling me about same thing at the same time: ** One is for status change, while the other is just a copy of it. ** One tells me there is a request, while the other tells me I've already accepted it. (should they arrive at the same time?)
You really need to check your subscriptions.
Well...is this normal setting?
You are subscribed twice to requests - once if they are created and once in general. And that's the way hermes works, two subscriptions mean two mails ;( Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 12.11.2013 08:54, Stephan Kulow wrote:
On 12.11.2013 00:51, Marguerite Su wrote:
On Mon, Nov 11, 2013 at 8:04 PM, Stephan Kulow
wrote: * Since the very beginning of this year, I can't receive any notifications on time...eg, I do something on OBS, notifications will arrive one or two days later...even weeks later...And it will always bomb you by sending lots of...I think they may stuck somewhere... It's news to me that this is the generic behaviour - today is special as clearly visible in the statistics graph.
Hi, coolo...
Just trust me... Oh, I'm far from disbelieving hermes being broken - just ask Klaas ;) But that very behaviour is indeed news to me.
OK, your domain resolves to an (for hermes unreachable) IPV6 address connect to gmail-smtp-in.l.google.com[2a00:1450:4001:c02::1a]:25: Network is unreachable If you resolve gmail.com directly, you get gmail-smtp-in.l.google.com[173.194.70.26]:25 This is something for our network admins to solve Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tuesday 2013-11-12 09:00, Stephan Kulow wrote:
OK, your domain resolves to an (for hermes unreachable) IPV6 address
connect to gmail-smtp-in.l.google.com[2a00:1450:4001:c02::1a]:25: Network is unreachable
Hurr. Did that happen to all IPv6-reachable targets? -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 12.11.2013 09:07, Jan Engelhardt wrote:
On Tuesday 2013-11-12 09:00, Stephan Kulow wrote:
OK, your domain resolves to an (for hermes unreachable) IPV6 address
connect to gmail-smtp-in.l.google.com[2a00:1450:4001:c02::1a]:25: Network is unreachable
Hurr. Did that happen to all IPv6-reachable targets?
I'm not in the mood to make a list of all IPv6-reachable targets. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Stephan Kulow
OK, your domain resolves to an (for hermes unreachable) IPV6 address
connect to gmail-smtp-in.l.google.com[2a00:1450:4001:c02::1a]:25: Network is unreachable
It should fall back to the IPv4 address without error. (Once the IPv6 support in Nürnberg is complete this will be moot anyway.) Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 12.11.2013 09:57, Andreas Schwab wrote:
Stephan Kulow
writes: OK, your domain resolves to an (for hermes unreachable) IPV6 address
connect to gmail-smtp-in.l.google.com[2a00:1450:4001:c02::1a]:25: Network is unreachable
It should fall back to the IPv4 address without error. Don't tell me what it should do, tell postfix about IPv6 ;(
Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Stephan Kulow
On 12.11.2013 09:57, Andreas Schwab wrote:
Stephan Kulow
writes: OK, your domain resolves to an (for hermes unreachable) IPV6 address
connect to gmail-smtp-in.l.google.com[2a00:1450:4001:c02::1a]:25: Network is unreachable
It should fall back to the IPv4 address without error. Don't tell me what it should do, tell postfix about IPv6 ;(
Have you considered filing a bug? Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Luckily this time notifications arrive "on time" (at least for me
considering what I've suffered...):
http://paste.opensuse.org/62561548
They came to me at 5:13 GMT+0800, which means GMT 9:13...
But note: these SRs were accepted 6:45...
Where's our missing 2 and a half hours...:-)
Greetings
Marguerite
On Tue, Nov 12, 2013 at 5:23 PM, Andreas Schwab
Stephan Kulow
writes: On 12.11.2013 09:57, Andreas Schwab wrote:
Stephan Kulow
writes: OK, your domain resolves to an (for hermes unreachable) IPV6 address
connect to gmail-smtp-in.l.google.com[2a00:1450:4001:c02::1a]:25: Network is unreachable
It should fall back to the IPv4 address without error. Don't tell me what it should do, tell postfix about IPv6 ;(
Have you considered filing a bug?
Andreas.
-- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 12.11.2013 10:32, Marguerite Su wrote:
Luckily this time notifications arrive "on time" (at least for me considering what I've suffered...):
http://paste.opensuse.org/62561548
They came to me at 5:13 GMT+0800, which means GMT 9:13...
But note: these SRs were accepted 6:45...
Where's our missing 2 and a half hours...:-)
This morning I found another bug that causes mail delays. It basically throttled the notification output - and if a lot of things build the notifications get backqueued. This queue was meanwhile so long that it was noticable in my inbox. Let's see if this improves now. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 13.11.2013 09:17, Stephan Kulow wrote:
On 12.11.2013 10:32, Marguerite Su wrote:
Luckily this time notifications arrive "on time" (at least for me considering what I've suffered...):
http://paste.opensuse.org/62561548
They came to me at 5:13 GMT+0800, which means GMT 9:13...
But note: these SRs were accepted 6:45...
Where's our missing 2 and a half hours...:-)
This morning I found another bug that causes mail delays. It basically throttled the notification output - and if a lot of things build the notifications get backqueued. This queue was meanwhile so long that it was noticable in my inbox. Let's see if this improves now.
OK, to me the situation looks much better this morning - how is your inbox looking like? :) Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Thu, Nov 14, 2013 at 4:07 PM, Stephan Kulow
This morning I found another bug that causes mail delays. It basically throttled the notification output - and if a lot of things build the notifications get backqueued. This queue was meanwhile so long that it was noticable in my inbox. Let's see if this improves now.
OK, to me the situation looks much better this morning - how is your inbox looking like? :)
Greetings, Stephan
Ha...I didn't submit any SR since you said you were upgrading OBS... And I'm writing a new thread about an OBS bug... Wait a minute.. Marguerite -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi, coolo,
On Thu, Nov 14, 2013 at 4:07 PM, Stephan Kulow
OK, to me the situation looks much better this morning - how is your inbox looking like? :)
Yes, instant delivery now. Thanks for your hard work! Marguerite -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Quoting Stephan Kulow
http://paste.opensuse.org/82409309 I don't actually spot anything unusual if you ignore that all mails from 7th-11th arrived on 11st, but as I said: that's the exception.
Honestly, an exception that happened multiple time over the last few weeks (I did not keep track on often it was asked on -buildservice IRC channel or how often I asked Darix about it). But in general, it's usable.. @coolo: what does 'fixing' it temporarily mean: rebooting the box? restarting the server? Black magic at each occurrence? Maybe it could at least be monitored and automatic action be triggered (I know, it does not fix the problem/cause, but at least it works around the after-effects). Dominique -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 12.11.2013 09:02, Dominique Leuenberger a.k.a. Dimstar wrote:
Quoting Stephan Kulow
: http://paste.opensuse.org/82409309 I don't actually spot anything unusual if you ignore that all mails from 7th-11th arrived on 11st, but as I said: that's the exception.
Honestly, an exception that happened multiple time over the last few weeks (I did not keep track on often it was asked on -buildservice IRC channel or how often I asked Darix about it).
But in general, it's usable..
@coolo: what does 'fixing' it temporarily mean: rebooting the box? restarting the server? Black magic at each occurrence? Maybe it could at least be monitored and automatic action be triggered (I know, it does not fix the problem/cause, but at least it works around the after-effects).
The main reason this stalled in the past is the dying background job. And the reason it's dying most of the time is an unclear memory leak. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Nov 12, 2013 at 3:54 PM, Stephan Kulow
On 12.11.2013 00:51, Marguerite Su wrote:
On Mon, Nov 11, 2013 at 8:04 PM, Stephan Kulow
wrote: * Since the very beginning of this year, I can't receive any notifications on time...eg, I do something on OBS, notifications will arrive one or two days later...even weeks later...And it will always bomb you by sending lots of...I think they may stuck somewhere... It's news to me that this is the generic behaviour - today is special as clearly visible in the statistics graph.
Hi, coolo...
Just trust me... Oh, I'm far from disbelieving hermes being broken - just ask Klaas ;) But that very behaviour is indeed news to me.
http://paste.opensuse.org/82409309 I don't actually spot anything unusual if you ignore that all mails from 7th-11th arrived on 11st, but as I said: that's the exception.
Yes, I understand it's an exception...I wasn't trolling... I just wanted to show the "style"...eg: one day you knew you didn't do anything on OBS, but hermes gave you a list of such SRs in this screenshot...
The mail headers of a day 3 mail would be more interesting than your above screenshot.
Ha...I think I was stupid...I thought it was normal so I deleted those SRs when they came... But I'm still able to test :-) eg: Today I forwarded 3 SRs to Factory: brise, fcitx-cloudpinyin, googletest. They're accepted on 2013-11-12T06:47:18, 2013-11-12T06:46:20, 2013-11-12T06:45:13 And now it's GMT 08:29:43, I still didn't get any notification about them... So I think maybe a few hours or days later, I'll be able to provide such weird screenshots...
You really need to check your subscriptions.
Well...is this normal setting?
You are subscribed twice to requests - once if they are created and once in general. And that's the way hermes works, two subscriptions mean two mails ;(
Ha...I understand...I subscribled for my two roles, one as packager (myself), one as repo maintainer (my role taken). so those 2 SRs are actually one for me and one for my role. Greetings Marguerite -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Monday 11 November 2013, Marguerite Su wrote:
Hi...
Today hermes sent me 14 emails about submit-requests and accept messages from myself at the same time 6:16pm...while those activities actually happened two days ago.
Currently I'm also missing again a lot build failure emails and at least one accepted sr mail. cu, Rudi -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am 22.11.2013 15:49, schrieb Ruediger Meier:
On Monday 11 November 2013, Marguerite Su wrote:
Hi...
Today hermes sent me 14 emails about submit-requests and accept messages from myself at the same time 6:16pm...while those activities actually happened two days ago.
Currently I'm also missing again a lot build failure emails and at least one accepted sr mail.
And you can exclude your mail filters as problem? Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Quoting Stephan Kulow
Am 22.11.2013 15:49, schrieb Ruediger Meier:
On Monday 11 November 2013, Marguerite Su wrote:
Hi...
Today hermes sent me 14 emails about submit-requests and accept messages from myself at the same time 6:16pm...while those activities actually happened two days ago.
Currently I'm also missing again a lot build failure emails and at least one accepted sr mail.
And you can exclude your mail filters as problem?
I confirm seeing the same.... so unlike Ruediger manages my mail filters too, we can likely exclude this :) Dominique -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am Freitag, 22. November 2013, 17:04:14 schrieb Dominique Leuenberger a.k.a. Dimstar:
Quoting Stephan Kulow
: Am 22.11.2013 15:49, schrieb Ruediger Meier:
On Monday 11 November 2013, Marguerite Su wrote:
Hi...
Today hermes sent me 14 emails about submit-requests and accept messages from myself at the same time 6:16pm...while those activities actually happened two days ago.
Currently I'm also missing again a lot build failure emails and at least one accepted sr mail.
And you can exclude your mail filters as problem?
I confirm seeing the same.... so unlike Ruediger manages my mail filters too, we can likely exclude this :)
Dominique
Confirming the same. The last hermes mails I got were sent Wednesday this week... -- Aeneas Jaißle » e: aj@ajaissle.de Sent using Kontact and Kolab on openSUSE -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am 22.11.2013 17:16, schrieb Aeneas Jaißle:
Am Freitag, 22. November 2013, 17:04:14 schrieb Dominique Leuenberger a.k.a. Dimstar:
Quoting Stephan Kulow
: Am 22.11.2013 15:49, schrieb Ruediger Meier:
On Monday 11 November 2013, Marguerite Su wrote:
Hi...
Today hermes sent me 14 emails about submit-requests and accept messages from myself at the same time 6:16pm...while those activities actually happened two days ago.
Currently I'm also missing again a lot build failure emails and at least one accepted sr mail.
And you can exclude your mail filters as problem?
I confirm seeing the same.... so unlike Ruediger manages my mail filters too, we can likely exclude this :)
Dominique
Confirming the same. The last hermes mails I got were sent Wednesday this week...
Ah, flatline in https://hermes.opensuse.org/statistics again - I understood the problem as only some were missing Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (8)
-
Aeneas Jaißle
-
Andreas Schwab
-
Dominique Leuenberger a.k.a. Dimstar
-
Jan Engelhardt
-
Klaas Freitag
-
Marguerite Su
-
Ruediger Meier
-
Stephan Kulow