[opensuse-packaging] fixes for rpmlint ? python singlespec pyc, opt-1.pyc

newer
[opensuse-packaging] package is...

Bruno Friedmann

5 Apr 2017 5 Apr '17

14:23

For any packages build with the new singlespec way there those warnings [ 21s] RPMLINT report: [ 21s] =============== [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.pyc /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [ 21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings. Should RPMLINT be fixed to become smarter, and just ignore those, or something still need to be fixed in the python part ? -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch Bareos Partner, openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Show replies by date

Brüns, Stefan

5 Apr 5 Apr

15:35

On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote:

...

For any packages build with the new singlespec way there those warnings

[ 21s] RPMLINT report: [ 21s] =============== [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.pyc /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [ 21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings.

Should RPMLINT be fixed to become smarter, and just ignore those, or something still need to be fixed in the python part ?

RPMLINT is right on this, we should not ship duplicate files. The files are duplicate because the opt-1 optimization level has no effect on the bytecode files. At least run %fdupes -s on the target directory, this will remove (symlink) the duplicates. The next question is, should we even ship both no-optimization and opt-1 bytecode files? The bytecode itself improves startup time significantly, but the opt-1 is dubious: - it removes any assert(...) statements. Although may improve speed if there are assertions in a hot path, I doubt this is a good idea in general - opt-1 bytecode files will be ignored until we use "python3 -O1" or set PYTONOPTIMIZE=1 - if there are no assertions, both bytecode files will be identical To test it yourself, put the following three files in the same directory, and execute the tests with: a) python3 ./opttest.py b) python3 -O ./opttest.py c) ./opttest.py d) PYTHONOPTIMIZE=1 ./opttest.py --- # optmodule_noassert.py def test_no_assert(): print("no assertion") --- # optmodule.py def test_assert_good(): assert(1) print("assert good") def test_assert_fail(): assert(0) print("assert fail") --- #! /usr/bin/python3 # opttest.py import optmodule import optmodule_noassert print("opttest") optmodule_noassert.test_no_assert() optmodule.test_assert_good() optmodule.test_assert_fail() --- Command a) and c) will throw an assertion and generate __pycache__/ optmodule{_noassert}.cpython-36.pyc. Command b) and d) will throw no assertion and generate __pycache__/ optmodule{_noassert}.cpython-36.opt-1.pyc. Both optmodule_noassert bytecode files are binary identical. Kind regards, Stefan -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

jan matejek

16:17

New subject: [opensuse-packaging] fdupes -s [was: Re: fixes for rpmlint ? python singlespec pyc, opt-1.pyc]

on a sidenote: On 5.4.2017 17:35, Brüns, Stefan wrote:

...

At least run %fdupes -s on the target directory, this will remove (symlink) the duplicates.

why do you recommend "-s" and symlinks? The default is to create hardlinks, but apparently some people consistently use the "-s" flag. Is there a good reason for this? ISTM hardlinks are better from performance standpoint and have no downsides otherwise, unless you're fdupe'ing across mountpoints. thanks m.

Bruno Friedmann

16:44

New subject: [opensuse-packaging] rpmlint broken for hardlink ,was fdupes, fixes for rpmlint :-)

On mercredi, 5 avril 2017 18.17:52 h CEST jan matejek wrote:

...

on a sidenote:

On 5.4.2017 17:35, Brüns, Stefan wrote:

...
At least run %fdupes -s on the target directory, this will remove (symlink) the duplicates.

why do you recommend "-s" and symlinks?

The default is to create hardlinks, but apparently some people consistently use the "-s" flag. Is there a good reason for this? ISTM hardlinks are better from performance standpoint and have no downsides otherwise, unless you're fdupe'ing across mountpoints.

thanks m.

by the way -s doesn't make rpmlint happy I got a symlink but this the complain :-( So if I expand rpm --eval %fdupes on my TW I'm getting this _target=""; _symlinks=0; fdupes -q -p -n -H -o name -r %1 | while read _file; do if test -z "$_target" ; then _target="$_file"; else if test -z "$_file" ; then _target=""; continue ; fi ; if test "$_symlinks" = 1; then ln -sf "${_target#/home/bruno/rpmbuild/BUILDROOT/%{name}-%{version}-% {release}.x86_64}" "$_file"; else ln -f "$_target" "$_file"; fi ; fi ; done fdupes -q -p -n -H -o name -r /some/path/with/files give the duplicates correctly /tmp/mappyfile/__init__.pyc /tmp/mappyfile/__init__.pyo /tmp/mappyfile/ordereddict.pyc /tmp/mappyfile/ordereddict.pyo /tmp/mappyfile/tokens.pyc /tmp/mappyfile/tokens.pyo /tmp/mappyfile/utils.pyc /tmp/mappyfile/utils.pyo And the result extracted is right ls -ld __init__.py* -rw-r--r-- 1 root root 147 Apr 5 18:26 __init__.py -rw-r--r-- 2 root root 307 Apr 5 18:26 __init__.pyc -rw-r--r-- 2 root root 307 Apr 5 18:26 __init__.pyo We got the 2 links for the duplicate confirmed by stat stat __init__.py* File: __init__.py Size: 147 Blocks: 8 IO Block: 4096 regular file Device: 27h/39d Inode: 1030758 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2017-04-05 18:26:52.287792333 +0200 Modify: 2017-04-05 18:26:52.287792333 +0200 Change: 2017-04-05 18:26:52.291792345 +0200 Birth: - File: __init__.pyc Size: 307 Blocks: 8 IO Block: 4096 regular file Device: 27h/39d Inode: 1030772 Links: 2 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2017-04-05 18:28:13.052033536 +0200 Modify: 2017-04-05 18:26:52.291792345 +0200 Change: 2017-04-05 18:26:52.291792345 +0200 Birth: - File: __init__.pyo Size: 307 Blocks: 8 IO Block: 4096 regular file Device: 27h/39d Inode: 1030772 Links: 2 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2017-04-05 18:28:13.052033536 +0200 Modify: 2017-04-05 18:26:52.291792345 +0200 Change: 2017-04-05 18:26:52.291792345 +0200 Birth: - So rpmlint is broken :-) -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch Bareos Partner, openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Bruno Friedmann

6 Apr 6 Apr

15:25

On mercredi, 5 avril 2017 17.35:06 h CEST Brüns, Stefan wrote:

...

On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote:

...
For any packages build with the new singlespec way there those warnings

[ 21s] RPMLINT report: [ 21s] =============== [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.pyc /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [

21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings.

Should RPMLINT be fixed to become smarter, and just ignore those, or something still need to be fixed in the python part ?

RPMLINT is right on this, we should not ship duplicate files. The files are duplicate because the opt-1 optimization level has no effect on the bytecode files.

At least run %fdupes -s on the target directory, this will remove (symlink) the duplicates.

The next question is, should we even ship both no-optimization and opt-1 bytecode files? The bytecode itself improves startup time significantly, but the opt-1 is dubious:

- it removes any assert(...) statements. Although may improve speed if there are assertions in a hot path, I doubt this is a good idea in general

- opt-1 bytecode files will be ignored until we use "python3 -O1" or set PYTONOPTIMIZE=1

- if there are no assertions, both bytecode files will be identical

To test it yourself, put the following three files in the same directory, and execute the tests with: a) python3 ./opttest.py b) python3 -O ./opttest.py c) ./opttest.py d) PYTHONOPTIMIZE=1 ./opttest.py

--- # optmodule_noassert.py def test_no_assert(): print("no assertion")

--- # optmodule.py def test_assert_good(): assert(1) print("assert good")

def test_assert_fail(): assert(0) print("assert fail") --- #! /usr/bin/python3 # opttest.py import optmodule import optmodule_noassert

print("opttest")

optmodule_noassert.test_no_assert() optmodule.test_assert_good() optmodule.test_assert_fail() ---

Command a) and c) will throw an assertion and generate __pycache__/ optmodule{_noassert}.cpython-36.pyc.

Command b) and d) will throw no assertion and generate __pycache__/ optmodule{_noassert}.cpython-36.opt-1.pyc.

Both optmodule_noassert bytecode files are binary identical.

Kind regards,

Stefan

The fault is due to rpmlint, mostly there's a patch in it but it has 3.5 hardcoded :-) Reported https://bugzilla.opensuse.org/show_bug.cgi?id=1032748 -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch Bareos Partner, openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Todd Rme

7 Apr 7 Apr

14:25

On Wed, Apr 5, 2017 at 11:35 AM, Brüns, Stefan wrote:

...

On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote:

...
For any packages build with the new singlespec way there those warnings

[ 21s] RPMLINT report: [ 21s] =============== [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.pyc /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [ 21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings.

Should RPMLINT be fixed to become smarter, and just ignore those, or something still need to be fixed in the python part ?

RPMLINT is right on this, we should not ship duplicate files. The files are duplicate because the opt-1 optimization level has no effect on the bytecode files.

At least run %fdupes -s on the target directory, this will remove (symlink) the duplicates.

The next question is, should we even ship both no-optimization and opt-1 bytecode files? The bytecode itself improves startup time significantly, but the opt-1 is dubious:

- it removes any assert(...) statements. Although may improve speed if there are assertions in a hot path, I doubt this is a good idea in general

- opt-1 bytecode files will be ignored until we use "python3 -O1" or set PYTONOPTIMIZE=1

- if there are no assertions, both bytecode files will be identical

To test it yourself, put the following three files in the same directory, and execute the tests with: a) python3 ./opttest.py b) python3 -O ./opttest.py c) ./opttest.py d) PYTHONOPTIMIZE=1 ./opttest.py

--- # optmodule_noassert.py def test_no_assert(): print("no assertion")

--- # optmodule.py def test_assert_good(): assert(1) print("assert good")

def test_assert_fail(): assert(0) print("assert fail") --- #! /usr/bin/python3 # opttest.py import optmodule import optmodule_noassert

print("opttest")

optmodule_noassert.test_no_assert() optmodule.test_assert_good() optmodule.test_assert_fail() ---

Command a) and c) will throw an assertion and generate __pycache__/ optmodule{_noassert}.cpython-36.pyc.

Command b) and d) will throw no assertion and generate __pycache__/ optmodule{_noassert}.cpython-36.opt-1.pyc.

Both optmodule_noassert bytecode files are binary identical.

Kind regards,

Stefan

Can the python singlespec macros be updated to automatically hardlink or symlink these files? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Bruno Friedmann

17:42

On vendredi, 7 avril 2017 16.25:43 h CEST Todd Rme wrote:

...

On Wed, Apr 5, 2017 at 11:35 AM, Brüns, Stefan

wrote:

...
On Mittwoch, 5. April 2017 16:23:07 CEST Bruno Friedmann wrote:

...
For any packages build with the new singlespec way there those warnings

[ 21s] RPMLINT report: [ 21s] =============== [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.pyc /usr/lib/python3.6/site- packages/mappyfile/__pycache__/tokens.cpython-36.opt-1.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/__init__.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/__init__.cpython-36.pyc [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/not-zip-safe /usr/lib/python3.6/site- packages/mappyfile-0.2.0-py3.6.egg-info/dependency_links.txt [ 21s] python3-mappyfile.noarch: W: files-duplicate /usr/lib/python3.6/site- packages/mappyfile/__pycache__/ordereddict.cpython-36.opt-1.pyc /usr/lib/ python3.6/site-packages/mappyfile/__pycache__/ordereddict.cpython-36.pyc [

21s] 3 packages and 0 specfiles checked; 0 errors, 4 warnings.

Should RPMLINT be fixed to become smarter, and just ignore those, or something still need to be fixed in the python part ?

RPMLINT is right on this, we should not ship duplicate files. The files are duplicate because the opt-1 optimization level has no effect on the bytecode files.

At least run %fdupes -s on the target directory, this will remove (symlink) the duplicates.

The next question is, should we even ship both no-optimization and opt-1 bytecode files? The bytecode itself improves startup time significantly, but the opt-1 is dubious:

- it removes any assert(...) statements. Although may improve speed if there are assertions in a hot path, I doubt this is a good idea in general

- opt-1 bytecode files will be ignored until we use "python3 -O1" or set PYTONOPTIMIZE=1

- if there are no assertions, both bytecode files will be identical

To test it yourself, put the following three files in the same directory, and execute the tests with: a) python3 ./opttest.py b) python3 -O ./opttest.py c) ./opttest.py d) PYTHONOPTIMIZE=1 ./opttest.py

--- # optmodule_noassert.py

def test_no_assert(): print("no assertion")

--- # optmodule.py

def test_assert_good(): assert(1) print("assert good")

def test_assert_fail(): assert(0) print("assert fail")

--- #! /usr/bin/python3 # opttest.py import optmodule import optmodule_noassert

print("opttest")

optmodule_noassert.test_no_assert() optmodule.test_assert_good() optmodule.test_assert_fail() ---

Command a) and c) will throw an assertion and generate __pycache__/ optmodule{_noassert}.cpython-36.pyc.

Command b) and d) will throw no assertion and generate __pycache__/ optmodule{_noassert}.cpython-36.opt-1.pyc.

Both optmodule_noassert bytecode files are binary identical.

Kind regards,

Stefan

Can the python singlespec macros be updated to automatically hardlink or symlink these files?

I've found where we need to be really carefull with singlespecs .... The fdupes line HAS to be written like this %python_expand %fdupes %{buildroot}%{$python_sitelib} Then it works -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch Bareos Partner, openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

2577

Age (days ago)

2579

Last active (days ago)

List overview

Download

6 comments

4 participants

participants (4)

Bruno Friedmann
Brüns, Stefan
jan matejek
Todd Rme