Hello,

On Oct 9 11:17 Greg Freemyer wrote (excerpt):

On October 9, 2014 6:09:39 AM EDT, Todd Rme toddrme2178@gmail.com wrote:

...

On Thu, Oct 9, 2014 at 12:05 AM, Greg Freemyer greg.freemyer@gmail.com wrote:

...

All,

I'm trying to package the Google Cups Cloud Proxy Printer

https://build.opensuse.org/package/show/home:gregfreemyer:branches:home:sbra...

...

...

...

The failed builds report:

[ 147s] error: Installed (but unpackaged) file(s) found: [ 147s] /usr/share/cloudprint-cups/auth.pyc [ 147s] /usr/share/cloudprint-cups/ccputils.pyc

...

...

You could try manually creating the .pyc files in the %install section so they will always be there. You should be able to use "python -m compileall %{buildroot}%{_datadir}/cloudprint-cups/" to generate the .pyc files.

Thank you, that seems to have fixed it.

I assume .pyc files are not architecture independent so that a package that contains .pyc files cannot be "noarch".

When I am right, "BuildArch: noarch" must be removed from OBS home:gregfreemyer:branches:home:sbrabec/CUPS-Cloud-Print/CUPS-Cloud-Print.spec

I am not at all a Python expert. Nevertheless I think it is not the right solution to compile Python files during build time in the build system environment. I think Python is meant to be compiled as needed on each end-user's system in his particular environment. Therefore I think RPMs with Python software should not contain compiled Python stuff but only the sources.

FYI: In general regarding .pyc (and perhaps .pyo) files: I have a RPM package (hplip) that contains many .py files. When python created the .pyc files those files do not belong to the RPM which means plain removing the RPM would leave its .pyc files on the system. Therefore I have a RPM preun script that removes the .pyc files. I need a simple generic method because I do not have the time to maintain a long list of individual files via RPM ghost, see Printing/hplip/hplip.spec If this is not the right way how to make a RPM with Python software, I appreciate a submitrequest from a Python expert that actually does it in the right way. Note that it must also "just work" for SLE11, see "osc results Printing hplip".

Kind Regards Johannes Meixner

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Dne 10.10.2014 v 11:36 Johannes Meixner napsal(a):

Hello,

On Oct 10 09:34 Johannes Meixner wrote (excerpt):

...

I assume .pyc files are not architecture independent so that a package that contains .pyc files cannot be "noarch".

(...)

It seems .pyc files compliled on a 64-bit machine are readable on a 32-bit machine but large integer numbers may get changed which might result different behaviour of the program (e.g. different output format or even different results).

Short version: For all intents and purposes, the bytecode is platform-independent. Don't worry about it.

Long version:

On a 32bit machine, you won't detect any difference between a 32bit and a 64bit bytecode. A number that would be int and TYPE_INT64 on a 64bit will end up being long (TYPE_LONG) on 32bit, regardless of how it is saved. Furthermore, ints and longs work together transparently. If you run into an issue with int/long, your code is platform-dependent in source form as well, because ints are shorter on 32bit. In practice, when programming Python, you simply don't care about the distinction.

If you tried hard enough, you could detect a difference if you ran on a 64bit machine from a 32bit bytecode, but I'm reasonably confident that the only way to get this is to be actively trying to detect specifically this condition. In any case, AFAICT we compile noarchs on 64bits, so the point is moot.

IOW, you are technically right about the platform dependence, but in practice, we can (and do) safely ignore the difference and enjoy the benefits of noarch.

As for "compiled as needed on each end-user's system" -> please note that if the user doesn't have write permissions to the install locations, they have to recompile the sources every time. This is the main reason to ship compiled bytecode. Your package is apparently run as root, but that's an exceptional situation. I would actually recommend that you include .pycs and .pyos for hplip in the rpm, and simplify your %preun.

Rest assured that the "usual method" works just fine in SLE11 as well ;)

regards m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Dne 15.10.2014 v 20:50 Jan Engelhardt napsal(a):

On Wednesday 2014-10-15 18:49, Jan Matějek wrote:

...

Short version: For all intents and purposes, the bytecode is platform-independent. Don't worry about it.

What about endianess?

it's explicitly stored and read as little-endian everywhere

and in Python 3 the TYPE_INT64 was removed, so results on 32bit and 64bit are the same

m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

hi,

Dne 16.10.2014 v 12:11 Johannes Meixner napsal(a):

A question regarding "include .pycs and .pyos":

Do you recommend that a package schould provide both .pyc files and .pyo files for each .py file i.e. that during build both "%py_compile" and "%py_compile -O" should be run?

as long as it's not too much trouble, then yes

For me it looks overcomplicated to make and provide both and then link them via "%fdupes" if they are the same.

fair point

Why not only make and provide the optimized files in any case?

That's a long story. You can run the python interpreter either in "normal" or "optimized" mode; the latter is enabled by "python -O". In normal mode, python only looks for .pyc; in optimized mode, it only looks for .pyo. No fallback to the other one. (IIRC, there are some technical and semantical issues, and the fallback feature is not important enough to justify the tradeoff) So if we want to support users of optimized mode, we need to provide both kinds of files.

At this moment, the optimizations in the "optimized" bytecode are extremely limited, so in the majority of cases, the resulting bytecode is the same. The biggest potential difference is that .pyo does not include certain kinds of strings; that also means that there is a potential difference of behavior, and that's the main reason why the optimized mode is kept separate. Personally, I believe that .pyo is kept around because the upstream is hoping that someone will start implementing some worthwhile optimizations ;) There is some work ongoing, but it's very low-priority.

In practice, maybe this is one more thing we don't have to care about. I have never in my life met anyone who uses the optimized mode, so i believe it's a minority usecase; people looking for more performance are switching to PyPy or Cython implementations.

My rule of thumb is this: if it's not too much trouble, provide both .pyc and .pyo and merge them with fdupes. If it causes problems, forget about .pyo.

A subsequent question:

When a package ships .pyc and/or .pyo files, I wonder if the .py files are still needed by a normal users.

Does Python software work correctly only with .pyc and/or .pyo files?

yes

If yes, shoudn't then the .py files moved into a *-python-source sub-package that is not installed by default?

We also do not install *.c source files but only the binaries.

That's honestly something that never crossed my mind.

It's somewhat different than with C sources though. On an installed system, C sources are for reference only -- if you want to make changes, you have to recompile and install the result. For Python, source code is the primary material and the bytecode is just a "cached result". Changes to sources are reflected immediately on the next run; the interpreter checks mtimes, and if the source doesn't match the bytecode, runs from source (and updates the bytecode if it can).

Personally, i'd vote to keep the sources for this reason alone :) It more closely matches the spirit of open source, after all. And there's also a number of smaller reasons: don't fix what ain't broken, every other distro does it like this, *-python-source would complicate packaging, etc.

But it is something worth thinking about.

regards m.

Strictly spaking, if it works only with .pyc and/or .pyo files, there is no need to provide the Python source files in the RPM. Only the SRPM provides the sources.

Exception: Python scripts that are meant to be adapted by the end-user.

Kind Regards Johannes Meixner

Johannes Meixner jsmeix@suse.de writes:

if test -f $pyo && diff -q $pyc $pyo 1>/dev/null

You want to use cmp -s here.

Andreas.

On 10/10/2014 01:34 AM, Johannes Meixner wrote:

I assume .pyc files are not architecture independent so that a package that contains .pyc files cannot be "noarch".

When I am right, "BuildArch: noarch" must be removed from OBS home:gregfreemyer:branches:home:sbrabec/CUPS-Cloud-Print/CUPS-Cloud-Print.spec

The .pyc files are bytecode that is platform independent, like Java .class files. The .pyo files are the same but optimized. Most Python packages will install the .pyc or .pyo files with the source and they can (and should) still be noarch. Now, if there is some compiled library or executable other than the Python, then the package will have to be arch dependent.

FYI: In general regarding .pyc (and perhaps .pyo) files: I have a RPM package (hplip) that contains many .py files. When python created the .pyc files those files do not belong to the RPM which means plain removing the RPM would leave its .pyc files on the system. Therefore I have a RPM preun script that removes the .pyc files. I need a simple generic method because I do not have the time to maintain a long list of individual files via RPM ghost, see Printing/hplip/hplip.spec If this is not the right way how to make a RPM with Python software, I appreciate a submitrequest from a Python expert that actually does it in the right way. Note that it must also "just work" for SLE11, see "osc results Printing hplip".

This does not seem right. Most Python packages build with setuptools (née distribute) which will create pyc/pyo files itself and thus they will get installed with the package without one even realizing. It sounds like hplip uses some other build method, so it should probably include the command line someone mentioned previously which will manually create all the Python bytecode during package build.

As far as how such a new package would interact with an already-installed package in which the bytecode isn't part of the RPM, I don't know, so I'm not sure if anything further would be needed to "fix" it.

-- Jason Craig