[Bug 291347] New: Unable to upload 250MB source tarball via webclient
https://bugzilla.novell.com/show_bug.cgi?id=291347 Summary: Unable to upload 250MB source tarball via webclient Product: openSUSE.org Version: unspecified Platform: Other OS/Version: Other Status: NEW Severity: Normal Priority: P5 - None Component: BuildService AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: lmichnovic@novell.com QAContact: adrian@novell.com Found By: --- I'm unable to upload 250 MB large tarball to my project(lmich) package openarena. The server error response was: OpenSUSE Webclient Error: Error Details: Errorcode: 500 Message: failed to allocate memory Exception from Webclient: NoMemoryError: failed to allocate memory Stack Trace: (eval):3:in `read' (eval):3:in `read' /srv/www/opensuse/webclient/current/public/../config/../app/models/package.rb:19:in `save_file' /srv/www/opensuse/webclient/current/public/../config/../app/controllers/package_controller.rb:344:in `save_file' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:1095:in `perform_action_without_filters' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:632:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:449:in `call' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:619:in `perform_action_without_benchmark' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue' /usr/lib64/ruby/1.8/benchmark.rb:293:in `measure' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/rescue.rb:83:in `perform_action' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `process_without_filters' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:624:in `process_without_session_management_support' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session_management.rb:114:in `process' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:330:in `process' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/dispatcher.rb:41:in `dispatch' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:168:in `process_request' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:143:in `process_each_request!' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:109:in `with_signal_handler' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:142:in `process_each_request!' /usr/lib64/ruby/1.8/fcgi.rb:612:in `each_cgi' /usr/lib64/ruby/1.8/fcgi.rb:609:in `each_cgi' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:141:in `process_each_request!' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:55:in `process!' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:25:in `process!' /srv/www/opensuse/webclient/current/public/dispatch.fcgi:24 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=291347#c1 --- Comment #1 from Ladislav Michnovič <lmichnovic@novell.com> 2007-07-12 02:39:33 MST --- It doesn't work even using osc : osc ci Sending oa070.tar.bz2 Transmitting file data .Traceback (most recent call last): File "/usr/bin/osc", line 10, in <module> sys.exit( osc.main() ) File "/usr/lib/python2.5/site-packages/osc/cmdln.py", line 256, in main return self.cmd(args) File "/usr/lib/python2.5/site-packages/osc/cmdln.py", line 279, in cmd retval = self.onecmd(argv) File "/usr/lib/python2.5/site-packages/osc/cmdln.py", line 395, in onecmd return self._dispatch_cmd(handler, argv) File "/usr/lib/python2.5/site-packages/osc/cmdln.py", line 1070, in _dispatch_ cmd return handler(argv[0], opts, *args) File "/usr/lib/python2.5/site-packages/osc/commandline.py", line 758, in do_co mmit p.put_source_file(filename) File "/usr/lib/python2.5/site-packages/osc/core.py", line 318, in put_source_f ile http_PUT(u, file = os.path.join(self.dir, n)) File "/usr/lib/python2.5/site-packages/osc/core.py", line 761, in http_PUT def http_PUT(*args, **kwargs): return http_request('PUT', *args, **kwargs ) File "/usr/lib/python2.5/site-packages/osc/core.py", line 750, in http_request fd = urllib2.urlopen(req, data=data) File "/usr/lib/python2.5/urllib2.py", line 121, in urlopen return _opener.open(url, data) File "/usr/lib/python2.5/urllib2.py", line 374, in open response = self._open(req, data) File "/usr/lib/python2.5/urllib2.py", line 392, in _open '_open', req) File "/usr/lib/python2.5/urllib2.py", line 353, in _call_chain result = func(*args) File "/usr/lib/python2.5/urllib2.py", line 1108, in https_open return self.do_open(httplib.HTTPSConnection, req) File "/usr/lib/python2.5/urllib2.py", line 1075, in do_open raise URLError(err) urllib2.URLError: <urlopen error (110, 'Connection timed out')> -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=291347 Matej Horvath <mhorvath@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|bnc-team- |opensuse-bugs@opensuse.org |screening@forge.provo.novell| |.com | Severity|Normal |Critical -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=291347#c2 --- Comment #2 from Peter Poeml <poeml@novell.com> 2007-07-13 05:19:17 MST --- IMO, such a large amount of data should not be held in memory, but rather temporarily saved to disk. 250 MB per process in memory sounds far too much and seems like an easy way to cause denial of service. Also, a quota is desirable to restrict size of uploads. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=291347 Hylton Conacher <hylton@conacher.co.za> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|opensuse-bugs@opensuse.org |bnc-team-screening@forge.provo.novell.com -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=291347 Andreas Jaeger <aj@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|bnc-team- |adrian@novell.com |screening@forge.provo.novell| |.com | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=291347 User adrian@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=291347#c3 Adrian Schröter <adrian@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|adrian@novell.com |abauer@novell.com --- Comment #3 from Adrian Schröter <adrian@novell.com> 2008-01-02 03:29:03 MST --- Andreas, does uploaded files not get stored to the FS ? (php has an option to configure this, dunno about rails). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=291347 User abauer@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=291347#c4 Andreas Bauer <abauer@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |lmichnovic@novell.com --- Comment #4 from Andreas Bauer <abauer@novell.com> 2008-01-02 07:24:41 MST --- No, they're indeed stored in memory and yes, that's not a good idea. I had to patch rails core to enable correct handling of PUT requests and did this in the most simple way. There wasn't put more work into this because we planned to use a feature of lighty 1.5 to forward file uploads to the backend without having to send data through the rails processes. Anyway, I increased the timeouts of lighty which solved a few similar issues. Could you try uploading the 250MB again? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=291347 User lmichnovic@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=291347#c5 Ladislav Michnovič <lmichnovic@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|lmichnovic@novell.com | --- Comment #5 from Ladislav Michnovič <lmichnovic@novell.com> 2008-01-03 05:38:40 MST --- I've tried and it is still not working. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=291347 User lmichnovic@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=291347#c6 --- Comment #6 from Ladislav Michnovič <lmichnovic@novell.com> 2008-01-03 05:39:12 MST --- OpenSUSE Webclient Error: Error Details: Errorcode: unknown Message: execution expired Exception from Webclient: Timeout::Error: execution expired Stack Trace: /usr/lib64/ruby/1.8/timeout.rb:54:in `rbuf_fill' /usr/lib64/ruby/1.8/timeout.rb:56:in `timeout' /usr/lib64/ruby/1.8/timeout.rb:76:in `timeout' /usr/lib64/ruby/1.8/net/protocol.rb:132:in `rbuf_fill' /usr/lib64/ruby/1.8/net/protocol.rb:116:in `readuntil' /usr/lib64/ruby/1.8/net/protocol.rb:126:in `readline' /usr/lib64/ruby/1.8/net/http.rb:2017:in `read_status_line' /usr/lib64/ruby/1.8/net/http.rb:2006:in `read_new' /usr/lib64/ruby/1.8/net/http.rb:1047:in `request' /usr/lib64/ruby/1.8/net/http.rb:854:in `put' /srv/www/opensuse/common/current/lib/activexml/transport.rb:498:in `http_do' /srv/www/opensuse/common/current/lib/activexml/transport.rb:417:in `direct_http' /srv/www/opensuse/webclient/current/public/../config/../lib/frontend_compatrb:51:in `put_file' /srv/www/opensuse/webclient/current/public/../config/../app/models/package.rb:149:in `save_file' /srv/www/opensuse/webclient/current/public/../config/../app/controllers/package_controller.rb:345:in `save_file' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:1095:in `send' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:1095:in `perform_action_without_filters' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:632:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:449:in `call' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:619:in `perform_action_without_benchmark' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue' /usr/lib64/ruby/1.8/benchmark.rb:293:in `measure' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/rescue.rb:83:in `perform_action' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `send' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `process_without_filters' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:624:in `process_without_session_management_support' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session_management.rb:114:in `process' /usr/lib64/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:330:in `process' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/dispatcher.rb:41:in `dispatch' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:168:in `process_request' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:143:in `process_each_request!' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:109:in `with_signal_handler' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:142:in `process_each_request!' /usr/lib64/ruby/vendor_ruby/1.8/fcgi.rb:612:in `each_cgi' /usr/lib64/ruby/vendor_ruby/1.8/fcgi.rb:609:in `each' /usr/lib64/ruby/vendor_ruby/1.8/fcgi.rb:609:in `each_cgi' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:141:in `process_each_request!' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:55:in `process!' /usr/lib64/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:25:in `process!' /srv/www/opensuse/webclient/current/public/dispatch.fcgi:24 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=291347 User lmichnovic@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=291347#c7 --- Comment #7 from Ladislav Michnovič <lmichnovic@novell.com> 2008-01-03 06:15:04 MST --- But the file actually got uploaded. MD5SUM is correct. (Downloaded with osc) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=291347 User abauer@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=291347#c8 --- Comment #8 from Andreas Bauer <abauer@novell.com> 2008-01-03 12:51:50 MST --- Makes sense. webclient did timeout while waiting for API/backend to finish their upload. I'm not sure how to solve this. The browser somehow would have to stream the data through webclient and API to the backend. I know google does something similar when processing uploads in gmail but I have no idea how. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply@novell.com