On 2017-02-17 16:00, jan matejek wrote:
2. dropping .pyc and compiling on installation That's certainly possible (Debian does something of the sort) and has even some interesting advantages: it allows you to transparently support many different python versions simultaneously with just one package install. Of course, it would be a lot of work. Doing this manually in every package is not realistic, we'd need some sort of automation that does this for a package in a single macro, or maybe a "python-central" tool that you run against the filelist in %post, or something... You would also have to %ghost all the .pyc files, otherwise all you get is a big ugly mess of not-owned files. The singlespec macro set could help with this, possibly even creating the %post/%preun scriptlets and %ghost entries automatically in packages.
yes, indeed.
This requires further discussion on the tradeoffs, probably at least a rudimentary install time benchmark, but I'm not opposed to including something like this.
I did some quick benchmarking today, that showed ~10 MB of source can be compiled in 1-2 seconds (as long as you do not compile them one-by-one, which creates some 10x overhead for loading python). For the typical 200MB .py files that would make 20-40 seconds extra minus the 10% that .pyc decompression would have eaten minus the saved transfer time.
4. reproducible builds For now, where it matters, let's touch generated .py files with a set time (probably mtime of tarball?). I suspect this is not an issue in most packages, so automation doesn't really help? But I'd be happy to learn more, e.g. see packages that don't build reproducibly and check out why that happens.
you can search in http://rb.zq1.de/compare.factory/reproducible.json for 'unreproducible' and http://rb.zq1.de/compare.factory/ also has some dozen build-compare diffs for those cases where more than just a file timestamp differed.
OTOH, doesn't rpm store mtime? Will a build with a regenerated file count as same as the previous build, if contents of the file are unchanged but metadata are?
it can be made constant with some recent patches linked in https://github.com/rpm-software-management/rpm/pull/144 That already allowed to build reproducibly 70-76% of Factory. Those patches are also in https://build.opensuse.org/package/show/home:bmwiedemann:reproducible/rpm
5. dropping .py That's a big NO. First of all, it's simply not worth it. On my work machine, which has a higher-than-usual amount of pythonic packages, the grand total size of all "*.py" files is 208 MB. Out of a 7.6GB /usr. Salt itself, which I installed for the purpose of this experiment, is 17 MB of that.
Second, users would kill us and I personally would be one of them. As I noted, .pyc is a *cache* for the primary source, which is the .py file. For python (and other languages that run from source), the automatic presence of source code and possibility of instant modification (for local patching, debugging, etc.) is a big advantage. Not installing .py files by default breaks all sorts of conventions, user expectations, goes against the spirit of open source, and is downright power-user-hostile, all in the name of saving space that's negligible on a typical system. We don't want to be That Distro (at least as far as I'm concerned)
strongly agree there. It would be same loss we got with systemd vs sysvinit scripts (e.g. the ones that handled filesystem mounts and crypto containers which is now handled by compiled C code)