Hello, On Oct 9 11:17 Greg Freemyer wrote (excerpt):

On October 9, 2014 6:09:39 AM EDT, Todd Rme <toddrme2178@gmail.com> wrote:

...

On Thu, Oct 9, 2014 at 12:05 AM, Greg Freemyer <greg.freemyer@gmail.com> wrote:

...

All,

I'm trying to package the Google Cups Cloud Proxy Printer

https://build.opensuse.org/package/show/home:gregfreemyer:branches:home:sbra... ...

...

The failed builds report:

[ 147s] error: Installed (but unpackaged) file(s) found: [ 147s] /usr/share/cloudprint-cups/auth.pyc [ 147s] /usr/share/cloudprint-cups/ccputils.pyc ... You could try manually creating the .pyc files in the %install section so they will always be there. You should be able to use "python -m compileall %{buildroot}%{_datadir}/cloudprint-cups/" to generate the .pyc files.

Thank you, that seems to have fixed it.

I assume .pyc files are not architecture independent so that a package that contains .pyc files cannot be "noarch". When I am right, "BuildArch: noarch" must be removed from OBS home:gregfreemyer:branches:home:sbrabec/CUPS-Cloud-Print/CUPS-Cloud-Print.spec I am not at all a Python expert. Nevertheless I think it is not the right solution to compile Python files during build time in the build system environment. I think Python is meant to be compiled as needed on each end-user's system in his particular environment. Therefore I think RPMs with Python software should not contain compiled Python stuff but only the sources. FYI: In general regarding .pyc (and perhaps .pyo) files: I have a RPM package (hplip) that contains many .py files. When python created the .pyc files those files do not belong to the RPM which means plain removing the RPM would leave its .pyc files on the system. Therefore I have a RPM preun script that removes the .pyc files. I need a simple generic method because I do not have the time to maintain a long list of individual files via RPM ghost, see Printing/hplip/hplip.spec If this is not the right way how to make a RPM with Python software, I appreciate a submitrequest from a Python expert that actually does it in the right way. Note that it must also "just work" for SLE11, see "osc results Printing hplip". Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH -- Maxfeldstrasse 5 -- 90409 Nuernberg -- Germany HRB 16746 (AG Nuernberg) GF: Jeff Hawn, Jennifer Guild, Felix Imendoerffer -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dne 10.10.2014 v 11:36 Johannes Meixner napsal(a):

Hello,

On Oct 10 09:34 Johannes Meixner wrote (excerpt):

...

I assume .pyc files are not architecture independent so that a package that contains .pyc files cannot be "noarch".

(...)

It seems .pyc files compliled on a 64-bit machine are readable on a 32-bit machine but large integer numbers may get changed which might result different behaviour of the program (e.g. different output format or even different results).

Short version: For all intents and purposes, the bytecode is platform-independent. Don't worry about it. Long version: On a 32bit machine, you won't detect any difference between a 32bit and a 64bit bytecode. A number that would be int and TYPE_INT64 on a 64bit will end up being long (TYPE_LONG) on 32bit, regardless of how it is saved. Furthermore, ints and longs work together transparently. If you run into an issue with int/long, your code is platform-dependent in source form as well, because ints are shorter on 32bit. In practice, when programming Python, you simply don't care about the distinction. If you tried hard enough, you could detect a difference if you ran on a 64bit machine from a 32bit bytecode, but I'm reasonably confident that the only way to get this is to be actively trying to detect specifically this condition. In any case, AFAICT we compile noarchs on 64bits, so the point is moot. IOW, you are technically right about the platform dependence, but in practice, we can (and do) safely ignore the difference and enjoy the benefits of noarch. As for "compiled as needed on each end-user's system" -> please note that if the user doesn't have write permissions to the install locations, they have to recompile the sources every time. This is the main reason to ship compiled bytecode. Your package is apparently run as root, but that's an exceptional situation. I would actually recommend that you include .pycs and .pyos for hplip in the rpm, and simplify your %preun. Rest assured that the "usual method" works just fine in SLE11 as well ;) regards m. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUPqWNAAoJEIskb84DCy7LO+IP/iJbIBATH990PYFk9Vd5/tS7 krDcU2xPfS0lG4D4gn79Ql/XkyBkMqh7F5DmqeD2uRGiiXaU24akcOYxAUT16whm TbOQz7QN9abRzQlbJ25bTHOX/BXqeniJ2aiZrxXh1gOK5cn1ExpM3CL9+JLRutzP OFZIvTBpPchTp+9uDIiuv4JcD4EJKTrkgrON+K4xfIw8CZl1RqUvx/Ah+JPdwhqv k5A+QA0EcybqSwr703DBRlWggGm21joerAwqZmIkFBrJ2c3K3ylI6/z8zqn1EP7F qx6+juApEYTC2UWUhxFLUPLmddSDTJUOEf3sUoqfIiF8dWy2P2+nW+Wwnk+UqDx2 m7fJ5fMibADShEgyYuXQmb/NVee78F0IWL86boPjgqfaFq1IhGbsVErNr4/us/r4 eQ05JNGIeu15l5Sn4LmjDzrGhK6/MRvmvEVmkxe9dw0n/2zkJgOmTK31Kd0fL64Y 1XTG6zmirqyG0j1e0ZySAydrgclw+lnnoQ1UljBZRFN6XEQSgGt7ETEPRWM4vZDy DhHiHNbdFwjmXmmnd0irrw1IriZJnREm4029ouipzg/rV+vBKLhb2HcsEwVWoei9 skXiiyHVYKhjIo3ywXbSBztKabxMku9YWbhfc/BoOX7ggzjl6MW3p0Lz3sa7S26P h8+bNeuO61MDQSUFjbFf =eum/ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dne 15.10.2014 v 20:50 Jan Engelhardt napsal(a):

On Wednesday 2014-10-15 18:49, Jan Matějek wrote:

...

Short version: For all intents and purposes, the bytecode is platform-independent. Don't worry about it.

What about endianess?

it's explicitly stored and read as little-endian everywhere and in Python 3 the TYPE_INT64 was removed, so results on 32bit and 64bit are the same m. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUP+DrAAoJEIskb84DCy7Lxn8P/3dO1j0DUO+nORXKepuafAes zdBt1jhwgGfH4DverG/Y8tez6CuOvns5n1e8U1bB59XyQZtuAG75wKBoWxic3Mxl wQ3B5y14+8ul7WZo349U8MdKEI/9FHMvuaW6T2K9qu+Qh3x/0oq90++KpDFLp7Ib xgLZc3MjglHmFMC5NiMjUtOLZeRGqyk5o3Bn5TnIyz6RnqBpGztIj/P/kqS7u07y tLkuuRQcNLyzbAxLR8KlV/U4e1nvAF37Yu7LGXg1OxWHeXWD9jIWt96bWvSfs2md j1weLVuYr1aT6/6ZC7OdalruiOW6FP7W0DFqArbOrY0fVYcQjth13LAPnmfk1qvT pcJp5P6+n9iCApMClE4p6t7mDfBEt1R88DMdLYuLYJKivYYppk3NFiqV8oQsqjDQ hYZo1WbR2SCzYd6Ti+OiRAls2NLZWC4xHX0RUSXUjjRGXZF0p/satkJnZf/kqzqS sXLMIHL3T+Gw+26Un+hAuBZ5Y1IB9VFBBYfkE9RlU7IlTif2xdKz/Zqm+YrhOs35 5Gh6mMbpSxNxRuXyTXgRZSg1eDj8Y+NzgfeW7vDEwb0AMAC2NQhBY6M8OdAQ7HZC Y0up93F/+XR7cFGGepbsKgQsrebukpDVTJ4TOOuTFVwBSlxbopoLRetHdA2hItSA 9jyWjeOkR0nDWI4l91hI =4d0w -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, Dne 16.10.2014 v 12:11 Johannes Meixner napsal(a):

A question regarding "include .pycs and .pyos":

Do you recommend that a package schould provide both .pyc files and .pyo files for each .py file i.e. that during build both "%py_compile" and "%py_compile -O" should be run?

as long as it's not too much trouble, then yes

For me it looks overcomplicated to make and provide both and then link them via "%fdupes" if they are the same.

fair point

Why not only make and provide the optimized files in any case?

That's a long story. You can run the python interpreter either in "normal" or "optimized" mode; the latter is enabled by "python -O". In normal mode, python only looks for .pyc; in optimized mode, it only looks for .pyo. No fallback to the other one. (IIRC, there are some technical and semantical issues, and the fallback feature is not important enough to justify the tradeoff) So if we want to support users of optimized mode, we need to provide both kinds of files. At this moment, the optimizations in the "optimized" bytecode are extremely limited, so in the majority of cases, the resulting bytecode is the same. The biggest potential difference is that .pyo does not include certain kinds of strings; that also means that there is a potential difference of behavior, and that's the main reason why the optimized mode is kept separate. Personally, I believe that .pyo is kept around because the upstream is hoping that someone will start implementing some worthwhile optimizations ;) There is some work ongoing, but it's very low-priority. In practice, maybe this is one more thing we don't have to care about. I have never in my life met anyone who uses the optimized mode, so i believe it's a minority usecase; people looking for more performance are switching to PyPy or Cython implementations. My rule of thumb is this: if it's not too much trouble, provide both .pyc and .pyo and merge them with fdupes. If it causes problems, forget about .pyo.

A subsequent question:

When a package ships .pyc and/or .pyo files, I wonder if the .py files are still needed by a normal users.

Does Python software work correctly only with .pyc and/or .pyo files?

yes

If yes, shoudn't then the .py files moved into a *-python-source sub-package that is not installed by default?

We also do not install *.c source files but only the binaries.

That's honestly something that never crossed my mind. It's somewhat different than with C sources though. On an installed system, C sources are for reference only -- if you want to make changes, you have to recompile and install the result. For Python, source code is the primary material and the bytecode is just a "cached result". Changes to sources are reflected immediately on the next run; the interpreter checks mtimes, and if the source doesn't match the bytecode, runs from source (and updates the bytecode if it can). Personally, i'd vote to keep the sources for this reason alone :) It more closely matches the spirit of open source, after all. And there's also a number of smaller reasons: don't fix what ain't broken, every other distro does it like this, *-python-source would complicate packaging, etc. But it is something worth thinking about. regards m.

Strictly spaking, if it works only with .pyc and/or .pyo files, there is no need to provide the Python source files in the RPM. Only the SRPM provides the sources.

Exception: Python scripts that are meant to be adapted by the end-user.

Kind Regards Johannes Meixner

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUP9/PAAoJEIskb84DCy7LTTwQAJpOnVuHtkbnj2zMAb7q4EOU rz7hKKrrv3zXgk15nGd2CiiSizaPzzqfADHFGYWajpWNP23j4fHGePdtGoFtsTWn mCJef1Nh9JYQAUboHtubtwa0DTOnqxqzEDnC4KPtfBFfPn79Y2jlt/oAYBCzUrNS PLJtCCQV76wZMCfvsY7KGe3G3UVT59NhvVXXIWMoi26CfmVQWGucjLhkZ+ychR0K sINuNn6Q799irov4SZzQnwW301swaiEGb55qy+fYNT8N1kPez0oi9jfTK8sqbUya 5KWoE3VD33+ejLxVkgONjfecFHnEaHvyZgfpvRAwVaNtmgoyv+WQ84DL5P9wx0Su edsj5GPNJrhW+bjhHujXEpdJ67NqnJmeBEU2G7DO0+AKWz0JO/x4YwhSkE3S8cJc mylbnMNDVOWJInIdUpUxOuoHlHq/dESM0LiokD3sSkBEVWnTmaAOQvmGtg8zRBuv 62/YX659KJ89ZqkaGzIQssOz7UVQwmWKg6gRRX0TrvnYv3hAcuEdkPhZf0gnu5tA y2tXfZ01zGWFWZo8APSXzMN4hfFn6IKih/eUw0A2FaxqT4r8hTrNhz26sxJmoVfL b0gTheEX1RBX+K3IhKV0A1km/ZDxH5ABtA9FyH1NkiHjlOLI895SJZj3/vJ8aUQD 3l1vIpcjzuMmmgUQj2Wx =oWTH -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Johannes Meixner <jsmeix@suse.de> writes:

if test -f $pyo && diff -q $pyc $pyo 1>/dev/null

You want to use cmp -s here. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 10/10/2014 01:34 AM, Johannes Meixner wrote:

I assume .pyc files are not architecture independent so that a package that contains .pyc files cannot be "noarch".

When I am right, "BuildArch: noarch" must be removed from OBS home:gregfreemyer:branches:home:sbrabec/CUPS-Cloud-Print/CUPS-Cloud-Print.spec

The .pyc files are bytecode that is platform independent, like Java .class files. The .pyo files are the same but optimized. Most Python packages will install the .pyc or .pyo files with the source and they can (and should) still be noarch. Now, if there is some compiled library or executable other than the Python, then the package will have to be arch dependent.

FYI: In general regarding .pyc (and perhaps .pyo) files: I have a RPM package (hplip) that contains many .py files. When python created the .pyc files those files do not belong to the RPM which means plain removing the RPM would leave its .pyc files on the system. Therefore I have a RPM preun script that removes the .pyc files. I need a simple generic method because I do not have the time to maintain a long list of individual files via RPM ghost, see Printing/hplip/hplip.spec If this is not the right way how to make a RPM with Python software, I appreciate a submitrequest from a Python expert that actually does it in the right way. Note that it must also "just work" for SLE11, see "osc results Printing hplip".

This does not seem right. Most Python packages build with setuptools (née distribute) which will create pyc/pyo files itself and thus they will get installed with the package without one even realizing. It sounds like hplip uses some other build method, so it should probably include the command line someone mentioned previously which will manually create all the Python bytecode during package build. As far as how such a new package would interact with an already-installed package in which the bytecode isn't part of the RPM, I don't know, so I'm not sure if anything further would be needed to "fix" it. -- Jason Craig -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org