Greg Freemyer writes:

The failed builds report:

[ 75s] Processing files: CUPS-Cloud-Print-0~20140814-33.1.noarch [ 76s] 2014-10-08 22:04:57|ERROR|Unable to write to log file /var/log/cups/cloudprint_log [ 76s] Traceback (most recent call last): [ 76s] File "/home/abuild/rpmbuild/BUILDROOT/CUPS-Cloud-Print-0~20140814-33.1.x86_64/usr/lib/cups/driver/cupscloudprint", line 93, in <module> [ 76s] requestors, storage = Auth.SetupAuth(False) [ 76s] File "/home/abuild/rpmbuild/BUILDROOT/CUPS-Cloud-Print-0~20140814-33.1.x86_64/usr/share/cloudprint-cups/auth.py", line 157, in SetupAuth [ 76s] permissions) [ 76s] File "/home/abuild/rpmbuild/BUILDROOT/CUPS-Cloud-Print-0~20140814-33.1.x86_64/usr/share/cloudprint-cups/oauth2client/multistore_file.py", line 81, in get_credential_storage [ 76s] filename, _MultiStore(filename, warn_on_readonly)) [ 76s] File "/home/abuild/rpmbuild/BUILDROOT/CUPS-Cloud-Print-0~20140814-33.1.x86_64/usr/share/cloudprint-cups/oauth2client/multistore_file.py", line 102, in __init__ [ 76s] self._create_file_if_needed() [ 76s] File "/home/abuild/rpmbuild/BUILDROOT/CUPS-Cloud-Print-0~20140814-33.1.x86_64/usr/share/cloudprint-cups/oauth2client/multistore_file.py", line 181, in _create_file_if_needed [ 76s] open(self._file.filename(), 'a+b').close() [ 76s] IOError: [Errno 13] Permission denied: '/etc/cloudprint.conf' [ 76s] Something is running python on the newly installed files, which will generate the *.pyc file on-the-fly. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On October 9, 2014 6:09:39 AM EDT, Todd Rme wrote:

On Thu, Oct 9, 2014 at 12:05 AM, Greg Freemyer wrote:

...

All,

I'm trying to package the Google Cups Cloud Proxy Printer

https://build.opensuse.org/package/show/home:gregfreemyer:branches:home:sbra...

...

It is building on my 64-bit laptop via "osc build" so in theory it should be fine for factory builds.

On OBS, both factory 32 and 64-bit builds are failing. On the other hand, they are both working for 13.1 builds.

The failed builds report:

[ 147s] error: Installed (but unpackaged) file(s) found: [ 147s] /usr/share/cloudprint-cups/auth.pyc [ 147s] /usr/share/cloudprint-cups/ccputils.pyc [ 147s] /usr/share/cloudprint-cups/cloudprintrequestor.pyc [ 147s] /usr/share/cloudprint-cups/oauth2client/__init__.pyc [ 147s] /usr/share/cloudprint-cups/oauth2client/anyjson.pyc [ 147s] /usr/share/cloudprint-cups/oauth2client/client.pyc [ 147s] /usr/share/cloudprint-cups/oauth2client/clientsecrets.pyc [ 147s] /usr/share/cloudprint-cups/oauth2client/crypt.pyc [ 147s] /usr/share/cloudprint-cups/oauth2client/locked_file.pyc [ 147s]

/usr/share/cloudprint-cups/oauth2client/multistore_file.pyc

...

[ 147s] /usr/share/cloudprint-cups/printer.pyc [ 147s] /usr/share/cloudprint-cups/printermanager.pyc

But in my %files section I have:

%dir %{_datadir}/cloudprint-cups %{_datadir}/cloudprint-cups/.coveragerc %{_datadir}/cloudprint-cups/*

I previously just had:

%{_datadir}/cloudprint-cups

But I had the same complaint, so I made it more explicit. Maybe I need to list every file?

Part of the strangeness is the exact release / arch that is failing seems to change each time I do a commit. It feels almost like it is a random issue that I am assigning meaning to.

Any ideas?

Thanks Greg

You could try manually creating the .pyc files in the %install section so they will always be there. You should be able to use "python -m compileall %{buildroot}%{_datadir}/cloudprint-cups/" to generate the .pyc files.

Thank you, that seems to have fixed it. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Hello, On Oct 10 09:34 Johannes Meixner wrote (excerpt):

I assume .pyc files are not architecture independent so that a package that contains .pyc files cannot be "noarch".

When I am right, "BuildArch: noarch" must be removed from OBS home:gregfreemyer:branches:home:sbrabec/CUPS-Cloud-Print/CUPS-Cloud-Print.spec

What I found via Google was http://daeken.com/python-marshal-format excerpt: ------------------------------------------------------------------------- The marshal format is used in .pyc files ... Note: I wrote all of this for the 2.x line. I don't know how much has changed in 3.x. ... i (TYPE_INT) -- Represents a int on a 32-bit machine. Stored as an int32. I (TYPE_INT64) -- Represents a int on a 64-bit machine. Stored as an int64. When read on a 32-bit machine, this may automatically become a long (if it's above 2**31). ... l (TYPE_LONG) -- Represents a long. ------------------------------------------------------------------------- It seems .pyc files compliled on a 64-bit machine are readable on a 32-bit machine but large integer numbers may get changed which might result different behaviour of the program (e.g. different output format or even different results). Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH -- Maxfeldstrasse 5 -- 90409 Nuernberg -- Germany HRB 16746 (AG Nuernberg) GF: Jeff Hawn, Jennifer Guild, Felix Imendoerffer -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 10/10/2014 01:34 AM, Johannes Meixner wrote:

I assume .pyc files are not architecture independent so that a package that contains .pyc files cannot be "noarch".

When I am right, "BuildArch: noarch" must be removed from OBS home:gregfreemyer:branches:home:sbrabec/CUPS-Cloud-Print/CUPS-Cloud-Print.spec

The .pyc files are bytecode that is platform independent, like Java .class files. The .pyo files are the same but optimized. Most Python packages will install the .pyc or .pyo files with the source and they can (and should) still be noarch. Now, if there is some compiled library or executable other than the Python, then the package will have to be arch dependent.

FYI: In general regarding .pyc (and perhaps .pyo) files: I have a RPM package (hplip) that contains many .py files. When python created the .pyc files those files do not belong to the RPM which means plain removing the RPM would leave its .pyc files on the system. Therefore I have a RPM preun script that removes the .pyc files. I need a simple generic method because I do not have the time to maintain a long list of individual files via RPM ghost, see Printing/hplip/hplip.spec If this is not the right way how to make a RPM with Python software, I appreciate a submitrequest from a Python expert that actually does it in the right way. Note that it must also "just work" for SLE11, see "osc results Printing hplip".

This does not seem right. Most Python packages build with setuptools (née distribute) which will create pyc/pyo files itself and thus they will get installed with the package without one even realizing. It sounds like hplip uses some other build method, so it should probably include the command line someone mentioned previously which will manually create all the Python bytecode during package build. As far as how such a new package would interact with an already-installed package in which the bytecode isn't part of the RPM, I don't know, so I'm not sure if anything further would be needed to "fix" it. -- Jason Craig -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Wednesday 2014-10-15 18:49, Jan Matějek wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Dne 10.10.2014 v 11:36 Johannes Meixner napsal(a):

...

Hello,

On Oct 10 09:34 Johannes Meixner wrote (excerpt):

...

I assume .pyc files are not architecture independent so that a package that contains .pyc files cannot be "noarch".

Short version: For all intents and purposes, the bytecode is platform-independent. Don't worry about it.

What about endianess? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, Dne 16.10.2014 v 12:11 Johannes Meixner napsal(a):

A question regarding "include .pycs and .pyos":

Do you recommend that a package schould provide both .pyc files and .pyo files for each .py file i.e. that during build both "%py_compile" and "%py_compile -O" should be run?

as long as it's not too much trouble, then yes

For me it looks overcomplicated to make and provide both and then link them via "%fdupes" if they are the same.

fair point

Why not only make and provide the optimized files in any case?

That's a long story. You can run the python interpreter either in "normal" or "optimized" mode; the latter is enabled by "python -O". In normal mode, python only looks for .pyc; in optimized mode, it only looks for .pyo. No fallback to the other one. (IIRC, there are some technical and semantical issues, and the fallback feature is not important enough to justify the tradeoff) So if we want to support users of optimized mode, we need to provide both kinds of files. At this moment, the optimizations in the "optimized" bytecode are extremely limited, so in the majority of cases, the resulting bytecode is the same. The biggest potential difference is that .pyo does not include certain kinds of strings; that also means that there is a potential difference of behavior, and that's the main reason why the optimized mode is kept separate. Personally, I believe that .pyo is kept around because the upstream is hoping that someone will start implementing some worthwhile optimizations ;) There is some work ongoing, but it's very low-priority. In practice, maybe this is one more thing we don't have to care about. I have never in my life met anyone who uses the optimized mode, so i believe it's a minority usecase; people looking for more performance are switching to PyPy or Cython implementations. My rule of thumb is this: if it's not too much trouble, provide both .pyc and .pyo and merge them with fdupes. If it causes problems, forget about .pyo.

A subsequent question:

When a package ships .pyc and/or .pyo files, I wonder if the .py files are still needed by a normal users.

Does Python software work correctly only with .pyc and/or .pyo files?

yes

If yes, shoudn't then the .py files moved into a *-python-source sub-package that is not installed by default?

We also do not install *.c source files but only the binaries.

That's honestly something that never crossed my mind. It's somewhat different than with C sources though. On an installed system, C sources are for reference only -- if you want to make changes, you have to recompile and install the result. For Python, source code is the primary material and the bytecode is just a "cached result". Changes to sources are reflected immediately on the next run; the interpreter checks mtimes, and if the source doesn't match the bytecode, runs from source (and updates the bytecode if it can). Personally, i'd vote to keep the sources for this reason alone :) It more closely matches the spirit of open source, after all. And there's also a number of smaller reasons: don't fix what ain't broken, every other distro does it like this, *-python-source would complicate packaging, etc. But it is something worth thinking about. regards m.

Strictly spaking, if it works only with .pyc and/or .pyo files, there is no need to provide the Python source files in the RPM. Only the SRPM provides the sources.

Exception: Python scripts that are meant to be adapted by the end-user.

Kind Regards Johannes Meixner

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUP9/PAAoJEIskb84DCy7LTTwQAJpOnVuHtkbnj2zMAb7q4EOU rz7hKKrrv3zXgk15nGd2CiiSizaPzzqfADHFGYWajpWNP23j4fHGePdtGoFtsTWn mCJef1Nh9JYQAUboHtubtwa0DTOnqxqzEDnC4KPtfBFfPn79Y2jlt/oAYBCzUrNS PLJtCCQV76wZMCfvsY7KGe3G3UVT59NhvVXXIWMoi26CfmVQWGucjLhkZ+ychR0K sINuNn6Q799irov4SZzQnwW301swaiEGb55qy+fYNT8N1kPez0oi9jfTK8sqbUya 5KWoE3VD33+ejLxVkgONjfecFHnEaHvyZgfpvRAwVaNtmgoyv+WQ84DL5P9wx0Su edsj5GPNJrhW+bjhHujXEpdJ67NqnJmeBEU2G7DO0+AKWz0JO/x4YwhSkE3S8cJc mylbnMNDVOWJInIdUpUxOuoHlHq/dESM0LiokD3sSkBEVWnTmaAOQvmGtg8zRBuv 62/YX659KJ89ZqkaGzIQssOz7UVQwmWKg6gRRX0TrvnYv3hAcuEdkPhZf0gnu5tA y2tXfZ01zGWFWZo8APSXzMN4hfFn6IKih/eUw0A2FaxqT4r8hTrNhz26sxJmoVfL b0gTheEX1RBX+K3IhKV0A1km/ZDxH5ABtA9FyH1NkiHjlOLI895SJZj3/vJ8aUQD 3l1vIpcjzuMmmgUQj2Wx =oWTH -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Hello Jan, first and foremost again very many thanks for your comprehensive explanations that really help me to understand what I am doing (or at least make me feel more confident in doing things hopefully right ;-) Only FYI: In package hplip in OBS project "Printing" I have now in hplip.spec (excerpt excluding some of my comments therein): -------------------------------------------------------------------- %install make install DESTDIR=%{buildroot} ... %py_compile %{buildroot}%{_datadir}/hplip ... %py_compile -O %{buildroot}%{_datadir}/hplip # Hardlink .pyc and .pyo when they have same content. # Do not run "fdupes buildroot/_datadir/hplip" because # fdupes will link any files with same content there # which can have unexpected side-effects, compare # https://bugzilla.opensuse.org/show_bug.cgi?id=784670 for pyc in $( find %{buildroot}%{_datadir}/hplip -name '*.pyc' ) do pyo="${pyc%.pyc}.pyo" if test -f $pyo && diff -q $pyc $pyo 1>/dev/null then echo hardlinking $pyc and $pyo because both have same content ln -f $pyc $pyo fi done -------------------------------------------------------------------- Newest fdupes doesn't help me because I build also for SLE11 (and I do not really trust even the newest fdupes ;-)

...

... shoudn't then the .py files moved into a *-python-source sub-package that is not installed by default?

We also do not install *.c source files but only the binaries. ... ... there's also a number of smaller reasons: don't fix what ain't broken, every other distro does it like this, *-python-source would complicate packaging, etc.

But it is something worth thinking about.

I fully agree that openSUSE should not needlessly introduce complications that no other distro does which could only make our users complain that "openSUSE broke it". But perhaps it is worth discussing it with other distros. Note that I never suggested to not make .py files easily installable (not like .c sources that are in no binary RPM). I only suggested not to install .py files by default (but still provide them in a binary RPM sub-package in the same repository where also the other binary RPMs of that software are). FWIW: All .py files in /usr/share/hplip/ have a disk usage of 2.9M. All .pyc files in /usr/share/hplip/ have a disk usage of 2.3M and also all .pyo files in /usr/share/hplip/ have a disk usage of 2.3M which is not really surprising because almost all are hardlinked. All files in /usr/share/hplip/ have a disk usage of 16M. Perhaps it is nitpicking to not install .py files by default. Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH -- Maxfeldstrasse 5 -- 90409 Nuernberg -- Germany HRB 16746 (AG Nuernberg) GF: Jeff Hawn, Jennifer Guild, Felix Imendoerffer -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org