Mailinglist Archive: opensuse-buildservice (74 mails)

< Previous Next >
RE: [opensuse-buildservice] Cannot upload big source files to private OBS instance
Hey Henne,

Last week I was ill, so here some news about the issue.
The download did finish after very long time (17 hours). This is much too long
other files (e.g a 120 MB) source file did finish in less a minute.
The OBS server had 2.5 GB of ram free at this time and lot of space in the
filesystem.

On 25.04.19 23:13, Karsten Keil wrote:

Any hints howto debug this ?

Logs? :-) Depending on your setup, there should be an apache log
(access.log) and a rails log (production.log) in /srv/www/obs/api/log/

Before it did finish here was no sign of the file in the logs.
Now it is, but the log entry seems to be written after the upload did finish it
looks wired reordered in the ssl_request log:

[27/Apr/2019:01:11:01 +0200] 192.168.3.47 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
"GET /monitor/update_building HTTP/1.1" 182
"https://buildservice.ftgs.net/monitor"; "Mozilla/5.0 (X11; Linux x86_64;
rv:52.0) Gecko/20100101 Firefox/52.0"
[27/Apr/2019:01:11:21 +0200] 192.168.3.47 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
"GET /monitor/events?range=24&arch=x86_64 HTTP/1.1" 9535
"https://buildservice.ftgs.net/monitor"; "Mozilla/5.0 (X11; Linux x86_64;
rv:52.0) Gecko/20100101 Firefox/52.0"
[27/Apr/2019:01:11:21 +0200] 192.168.3.47 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
"GET /monitor/events?range=24&arch=x86_64 HTTP/1.1" 9535
"https://buildservice.ftgs.net/monitor"; "Mozilla/5.0 (X11; Linux x86_64;
rv:52.0) Gecko/20100101 Firefox/52.0"
[26/Apr/2019:08:36:26 +0200] 192.168.3.47 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
"PUT
/source/FTGS:common_2019.001/libqt5-qtdoc/qt-everywhere-src-5.12.3.tar.xz?rev=repository
HTTP/1.1" 112 "-" "osc/0.161.1"
[26/Apr/2019:08:36:26 +0200] 192.168.3.47 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
"PUT
/source/FTGS:common_2019.001/libqt5-qtdoc/qt-everywhere-src-5.12.3.tar.xz?rev=repository
HTTP/1.1" 112 "-" "osc/0.161.1"
[27/Apr/2019:01:12:41 +0200] 192.168.3.47 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
"GET /monitor/update_building HTTP/1.1" 182
"https://buildservice.ftgs.net/monitor"; "Mozilla/5.0 (X11; Linux x86_64;
rv:52.0) Gecko/20100101 Firefox/52.0"
[27/Apr/2019:01:12:41 +0200] 192.168.3.47 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
"GET /monitor/update_building HTTP/1.1" 182
"https://buildservice.ftgs.net/monitor"; "Mozilla/5.0 (X11; Linux x86_64;
rv:52.0) Gecko/20100101 Firefox/52.0"
[27/Apr/2019:01:12:41 +0200] 192.168.3.47 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
"GET /monitor/events?range=24&arch=x86_64 HTTP/1.1" 9535
"https://buildservice.ftgs.net/monitor"; "Mozilla/5.0 (X11; Linux x86_64;
rv:52.0) Gecko/20100101 Firefox/52.0"
[27/Apr/2019:01:12:41 +0200] 192.168.3.47 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
"GET /monitor/events?range=24&arch=x86_64 HTTP/1.1" 9535
"https://buildservice.ftgs.net/monitor"; "Mozilla/5.0 (X11; Linux x86_64;
rv:52.0) Gecko/20100101 Firefox/52.0"
[27/Apr/2019:01:12:00 +0200] 192.168.3.47 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
"PUT
/source/FTGS:common_2019.001/libqt5-qtdoc/qt-everywhere-src-5.12.3.tar.xz?rev=repository
HTTP/1.1" 92 "-" "osc/0.161.1"
[27/Apr/2019:01:12:00 +0200] 192.168.3.47 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
"PUT
/source/FTGS:common_2019.001/libqt5-qtdoc/qt-everywhere-src-5.12.3.tar.xz?rev=repository
HTTP/1.1" 92 "-" "osc/0.161.1"

My last attempt was that I did start an new try with osc to commit the file
Friday (26.4.) morning, so the [26/Apr/2019:08:36:26 +0200] entry would
correspond to this.
This time I did not kill the upload attempt after few houres, so it indeed
seems to finish in the night after ~17 hours.
In the production log is a corresponding entry for [27/Apr/2019:01:12:41
+0200].

I did start osc with debug of http and it did stall in the PUT request, a
tcpdump shows data packets with big size for about one minute,
then only few (6-8)small packets (1k) did flow between the client and the the
server. Very strange.

I do not have so much ideas about the reason, I would expect an upload time of
not more as 3-4 minutes if a 150 MB file finished in less than 1 minute.
Or is here some wired cgroup/systemd magic hitting us ?

Is here a place where the file is saved temporary during the API put
operation?

Yes, /srv/www/obs/api/tmp

No sign of the file in this place at the time where the upload did stall for
long time. I looked with ls -atrl and even lsof did not find any open file
which looks as it could be related.

Do I need to increase some limits ?

Maybe, depends on what is happening and your server configuration. By
default, on the appliance, there are no limits. Do you have XForward
configured?
use_xforward: true
in options.yml

Karsten

The information in this e-mail is confidential. The contents may not be
disclosed or used by anyone other than the addressee. Access to this e-mail by
anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and
delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of
this e-mail as it has been sent over public networks. If you have any concerns
over the content of this message or its Accuracy or Integrity, please contact
Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus
scanning software but you should take whatever measures you deem to be
appropriate to ensure that this message and any attachments are virus free.
< Previous Next >
Follow Ups