[Bug 656281] New: some source files are damaged after checkout
https://bugzilla.novell.com/show_bug.cgi?id=656281 https://bugzilla.novell.com/show_bug.cgi?id=656281#c0 Summary: some source files are damaged after checkout Classification: Internal Novell Products Product: openSUSE Build Service Version: 2.0 Platform: x86-64 OS/Version: Linux Status: NEW Severity: Critical Priority: P5 - None Component: osc AssignedTo: adrian@novell.com ReportedBy: lpechacek@novell.com QAContact: adrian@novell.com CC: mcihar@novell.com, ptesarik@novell.com, mvancura@novell.com, dsterba@novell.com, mhocko@novell.com Found By: L3 Blocker: --- Some source files have zero size after checkout from internal build service. The occurrence of this error is random. You can reproduce with: for i in $(seq 1 25) do echo $i------------------ osc -HA https://api.suse.de co -r1 PTF:2282 kernel-source echo $? mv -v PTF:2282 PTF:2282-$i done and then find zero-sized files with: find PTF\:2282* -exec test -f {} -a \! -s {} \; -print The osc -H debug log shows the following in the failure case GET https://api.suse.de/source/PTF:2282/kernel-source/linux-2.6.32.tar.bz2?rev=d... send: 'GET https://api.suse.de/source/PTF:2282/kernel-source/linux-2.6.32.tar.bz2?rev=d... HTTP/1.1\r\nAccept-Encoding: identity\r\ nHost: api.suse.de\r\nCookie: IPCZQX018ef15359=000038016720c83f0e19f601; ZNPCQ002-apide=V001ec3960db\r\nConnection: close\r\nUser-agent: osc/0.125.5\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: X-Opensuse-Apiversion: 2.1.0 header: Content-Transfer-Encoding: binary header: X-Runtime: 4424 header: Content-Disposition: attachment header: Cache-Control: private header: Date: Mon, 29 Nov 2010 14:23:49 GMT header: Server: lighttpd/1.5.0 header: Content-Type: application/octet-stream header: Connection: close header: Content-Length: header: Via: 1.1 ICS_SERVER (iChain 2.3.419) A PTF:2282/kernel-source/linux-2.6.32.tar.bz2 Please note the missing numeric value in the Content-Length field. Osc exit code is zero in the failure case, making the breakage mostly silent. Frequency of the above breakage is 1-5 times per test run. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656281 https://bugzilla.novell.com/show_bug.cgi?id=656281#c1 --- Comment #1 from Adrian Schröter <adrian@novell.com> 2010-11-30 15:42:30 UTC --- I was able to reproduce it this morning, but not this afternoon anymore :/ I added a validation in osc at least, in case Content-Length size is not matching streamed data. It will abort in this situation. This may solve this problem already, but I suppose the real problem is either api or iChain. However, I can't debug it for now, because I can't reproduce atm. In best case current osc git master is already solving your problem .... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656281 https://bugzilla.novell.com/show_bug.cgi?id=656281#c2 --- Comment #2 from Libor Pecháček <lpechacek@novell.com> 2010-11-30 17:11:15 CET --- I also was not able to reproduce the issue early morning today, but after 10am CET osc started to download zero-sized files. Seems to depend on the load. Just for record, this script allows to reproduce the issue without osc. Replace "user:pass" with your credentials. L=0; CL=0; N=0; while test $L -eq $CL; do L=$(curl -D bs.trace -u 'user:pass' https://api.suse.de/source/PTF:2282/kernel-source/postun.sh?rev=df97383ac5ca6710d7a881a9c42c7f23|wc -c); CL=$(fgrep Content-Length: bs.trace | cut -f2 -d\ |tr -d \\n\\r); N=$[$N+1]; echo $N - $L/$CL; done The number of iterations before it fails varies, and can be up to 1500. Checking the Content-Length in osc is a good idea, thanks Adrian for implementing it. BTW, it might be even better to retry upon receiving the invalid header as osc users would not have to care about these intermittent errors. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656281 https://bugzilla.novell.com/show_bug.cgi?id=656281#c3 --- Comment #3 from Michal Čihař <mcihar@novell.com> 2010-12-01 10:27:55 CET --- Created an attachment (id=402864) --> (http://bugzilla.novell.com/attachment.cgi?id=402864) Patch implementing retries Attaching patch which does implement retries until the request has valid Content-Length or (configurable) number of retries is reached. Adrian does is it acceptable to push this to osc? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656281 https://bugzilla.novell.com/show_bug.cgi?id=656281#c4 Michal Čihař <mcihar@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #402864|0 |1 is obsolete| | --- Comment #4 from Michal Čihař <mcihar@novell.com> 2010-12-01 10:30:30 CET --- Created an attachment (id=402865) --> (http://bugzilla.novell.com/attachment.cgi?id=402865) Patch implementing retries, v2 Sorry, I've uploaded older version of the patch which was not correctly handling number of retries, this one works as expected. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656281 https://bugzilla.novell.com/show_bug.cgi?id=656281#c6 Marcus Hüwe <suse-tux@gmx.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |suse-tux@gmx.de --- Comment #6 from Marcus Hüwe <suse-tux@gmx.de> 2010-12-02 20:05:43 UTC --- (In reply to comment #3)
Adrian does is it acceptable to push this to osc? Your patch won't work if osc receives chunked data (in this case no "Content-Length" header is present) so we also need to take the "Transfer-Encoding" header into consideration. But I'm not quite sure if it's a good idea to workaround the "problem" instead of fixing it.
-- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656281 https://bugzilla.novell.com/show_bug.cgi?id=656281#c7 --- Comment #7 from Michal Čihař <mcihar@novell.com> 2010-12-02 21:47:51 CET --- It should work when Content-Length is not present (get returns None in such case). Indeed it would be much better to fix it on server side. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656281 https://bugzilla.novell.com/show_bug.cgi?id=656281#c8 --- Comment #8 from Adrian Schröter <adrian@novell.com> 2010-12-06 09:35:12 UTC --- it would be indeed to fix it on server side, but it would be way too much time consuming atm with the iChain proxy and we will most likely anyway need to work on obsoleting it in the next few month. So I do not want to waste time here, but take this as an opportunity to make osc more robust against server errors for now. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656281 https://bugzilla.novell.com/show_bug.cgi?id=656281#c9 --- Comment #9 from Michal Čihař <mcihar@novell.com> 2010-12-06 16:06:54 CET --- Okay, retries patch pushed to osc repo. Anyway it should be investigated whether iChain is really guilty in this case. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656281 https://bugzilla.novell.com/show_bug.cgi?id=656281#c10 --- Comment #10 from Adrian Schröter <adrian@novell.com> 2010-12-06 15:09:01 UTC --- I did not say that iChain is guilty (but it might). But alone to debug it without having access to it is very time consuming. So we will obsolete it. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=656281 https://bugzilla.novell.com/show_bug.cgi?id=656281#c11 Adrian Schröter <adrian@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |FIXED --- Comment #11 from Adrian Schröter <adrian@suse.com> 2012-04-02 11:55:03 UTC --- should be solved after switching to apache and using our own login proxy. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com