Branch: refs/heads/master Home: https://github.com/openSUSE/osc Commit: c932f95d46c5438fff2bfab365dc9b0df6c6c90e https://github.com/openSUSE/osc/commit/c932f95d46c5438fff2bfab365dc9b0df6c6c... Author: Marcus Huewe <suse-tux@gmx.de> Date: 2021-04-10 (Sat, 10 Apr 2021) Changed paths: M osc/conf.py M osc/core.py Log Message: ----------- Support an arbitrary sized file in core.http_request The old code only supports a file whose size is less then or equal to INT_MAX (due to a reasonable(!) limit in M2Crypto). The actual issue is in core.http_request which mmap(...)s the file, wraps it into a memoryview/buffer and then passes the memoryview/buffer to urlopen. Eventually, the whole memoryview/buffer is read into memory (see m2_PyObject_GetBufferInt). If the file is too large (> INT_MAX), m2_PyObject_GetBufferInt raises a ValueError (which is perfectly fine!). Reading a whole file into memory is completely insane. In order to avoid this, we now simply pass a file-like object to urlopen (more precisely, the file-like object is associated with the Request instance that is passed to urlopen). The advantange is that the file-like object is processed in chunks of 8192 bytes (see http.client.HTTPConnection) (that is, only 8192 bytes are read into memory (instead of the whole file)). There are two pitfalls when passing a file-like object to urlopen: * By default, a chunked Transfer-Encoding is applied. It seems that some servers (like api.o.o) do not like this (PUTing a file with a chunked Transfer-Encoding to api.o.o results in status 400). In order to avoid a chunked Transfer-Encoding, we explicitly set a Content-Length header (we also do this in the non-file case (just for the sake of completeness)). * If the request fails with status 401, it is retried with an appropriate Authorization header. When retrying the request, the file's offset has to be repositioned to the beginning of the file (otherwise, a 0-length body is sent which most likely does not match the Content-Length header). Note: core.http_request's "data" and "file" parameters are now mutually exclusive because specifying both makes no sense (only one of them is considered) and it simplifies the implementation a bit. Fixes: #202 ("osc user authentification seems to be broken with last commit") Fixes: #304 ("osc ci - cannot handle more than 2 GB file uploads") Commit: dd9130a292cfe095fbac816bf4400fa985c1c0e1 https://github.com/openSUSE/osc/commit/dd9130a292cfe095fbac816bf4400fa985c1c... Author: Marcus Huewe <suse-tux@gmx.de> Date: 2021-04-16 (Fri, 16 Apr 2021) Changed paths: M osc/conf.py M osc/core.py Log Message: ----------- Merge branch 'http_request_no_filesize_limit' of https://github.com/marcus-h/osc Support an arbitrary sized file in core.http_request. Compare: https://github.com/openSUSE/osc/compare/fc5470a152a5...dd9130a292cf