Mailinglist Archive: opensuse-buildservice (88 mails)

< Previous Next >
[opensuse-buildservice] Re: bs_srcserver seems stuck at a specific lastnotification number
Hi all,

JFTR: this is OBS-2.8.4 still, but I think the issue is present in 2.9

On 19.06.2018 17:31, Stefan Seyfried wrote:
Hi,

I was investigating why my OBS server has a considerable amount of CPU load
all the time.
It's bs_srcserver that occupies one CPU almost entirely.
Investigating, it looks like it is processing the
/srv/obs/events/lastnotifications over and over again.

I have no real idea how this all plays together, and what I have to do to get
this into a sane state again.

(has nothing to do with schedulers, architectures etc...)

I have put debugging code into
src/api/app/models/update_notification_events.rb:update_events, in the
ActiveRecord::StatementInvalid path.

I, [2018-06-20T07:45:48.800294 #31698] INFO -- : [31698:20.91]
#<ActiveRecord::StatementInvalid: Mysql2::Error:
Incorrect string value: '\xC2\x9B:wq"...' for column 'payload' at row 1: INSERT
INTO `events` (`eventtype`, `payload`,
`created_at`, `updated_at`, `lock_version`) VALUES ('Event::Commit',
'{\"project\":\"home:username:branches:Images:S4H\",\"package\":\"sles12_sp1_x86_64_s4h\",\"comment\":\"awq\",\"user\":\"username\",\"files\":\"Modified:\\n
sles12_sp2_x86_64_s4h.kiwi\\n\\n\",\"rev\":\"22\"}', '2018-06-20 07:45:48',
'2018-06-20 07:45:48', 0)>

obs:/srv/obs/events # grep 297709 lastnotifications
297709|SRCSRV_COMMIT|1511433089|comment%3Da�%9B:wq|files%3DModified:%0A
sles12_sp2_x86_64_s4h.kiwi%0A%0A|package%3Dsles12_sp1_x86_64_s4h|project%3Dhome:username:branches:Images:S4H|rev%3D22|user%3Dusername

obs:/srv/obs/events # grep 297709 lastnotifications | hexdump -C
00000000 32 39 37 37 30 39 7c 53 52 43 53 52 56 5f 43 4f |297709|SRCSRV_CO|
00000010 4d 4d 49 54 7c 31 35 31 31 34 33 33 30 38 39 7c |MMIT|1511433089||
00000020 63 6f 6d 6d 65 6e 74 25 33 44 61 c2 25 39 42 3a |comment%3Da.%9B:|
00000030 77 71 7c 66 69 6c 65 73 25 33 44 4d 6f 64 69 66 |wq|files%3DModif|
00000040 69 65 64 3a 25 30 41 20 20 73 6c 65 73 31 32 5f |ied:%0A sles12_|
00000050 73 70 32 5f 78 38 36 5f 36 34 5f 73 34 68 2e 6b |sp2_x86_64_s4h.k|
00000060 69 77 69 25 30 41 25 30 41 7c 70 61 63 6b 61 67 |iwi%0A%0A|packag|
00000070 65 25 33 44 73 6c 65 73 31 32 5f 73 70 31 5f 78 |e%3Dsles12_sp1_x|
00000080 38 36 5f 36 34 5f 73 34 68 7c 70 72 6f 6a 65 63 |86_64_s4h|projec|
00000090 74 25 33 44 68 6f 6d 65 3a 75 73 65 72 6e 61 6d |t%3Dhome:usernam|
000000a0 65 3a 62 72 61 6e 63 68 65 73 3a 49 6d 61 67 65 |e:branches:Image|
000000b0 73 3a 53 34 48 7c 72 65 76 25 33 44 32 32 7c 75 |s:S4H|rev%3D22|u|
000000c0 73 65 72 25 33 44 75 73 65 72 6e 61 6d 65 0a |ser%3Dusername.|
000000cf

So a user managed to sneak in an invalid character 0xC2 into a comment and this
breaks OBS.

USERS! We should really forbid access for them! ;-)

I removed the \xC2%9B combo, then the code took the sync lost path:

if @last['sync'] == 'lost'
# we're doomed, but we can't help - it's not supposed to happen
BackendInfo.lastnotification_nr = Integer(@last['next'])
return
end

and now everything seems back to normal.

Maybe more input checking is needed to prevent this.
--
Stefan Seyfried

"For a successful technology, reality must take precedence over
public relations, for nature cannot be fooled." -- Richard Feynman
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-buildservice+owner@xxxxxxxxxxxx

< Previous Next >
Follow Ups
References