March 2019 - openSUSE Science - openSUSE Mailing Lists

[opensuse-science] Remove outdated Build Targets on network:cluster for OBS
by Egbert Eich 31 Mar '19

31 Mar '19

Hi folks! to save some build cycles I would like to remove (or at least disable) some outdated build targets for the project network:cluster: Anog these are: SEL_11_SP3 SLE_12_SP1 openSUSE_13.1 openSUSE_13.2 openSUSE_Leap_42.1 openSUSE_Leap_42.2 If they are just disabled, they may be enabled individually per package. Unless I meet strong oppostition I will start disabling them on Tuesday. I've now added SUSE:SLE-15-SP1:GA/standard as a build target. Cheers, Egbert. -- Egbert Eich (Res. & Dev.) SUSE LINUX GmbH SUSE Labs - Project Manager HPC Tel: +49 911-740 53 0 http://www.suse.com ----------------------------------------------------------------- SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5, D90409 Nürnberg, Germany-- To unsubscribe, e-mail: opensuse-science+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-science+owner(a)opensuse.org

1 0

[opensuse-science] Unable to use update-alternatives for libopenblas_openmp
by Chris Coutinho 16 Mar '19

16 Mar '19

Hi all, I think the update-alternatives configuration set up for openblas incorrectly/incompletely overrides an alternative for libblas.so.3. Namely, the libopenblas_pthreads provides an alternative for itself (libopenblas_pthreads.so.0), but then places a link to the more generic openblas library libopenblas.so.0. This overrides the alternatives set up for libblas, and gets picked up by e.g. cmake and its FindBLAS module. In short, here you can see the options available for libblas.so.3, and my override for the openmp version: $ update-alternatives --query libblas.so.3 Name: libblas.so.3 Link: /usr/lib64/libblas.so.3 Status: manual Best: /usr/lib64/blas/libblas.so.3 Value: /usr/lib64/libopenblas_openmp.so.0 Alternative: /usr/lib64/blas/libblas.so.3 Priority: 50 Alternative: /usr/lib64/libopenblas_openmp.so.0 Priority: 20 Alternative: /usr/lib64/libopenblas_pthreads.so.0 Priority: 20 But this value is overriden by libopenblas.so.0 $ ls -l /usr/lib64/*blas* | grep alternative lrwxrwxrwx 1 root root 30 Sep 11 2018 /usr/lib64/libblas.so.3 -> /etc/alternatives/libblas.so.3 lrwxrwxrwx 1 root root 31 Sep 11 2018 /usr/lib64/libcblas.so.3 -> /etc/alternatives/libcblas.so.3 lrwxrwxrwx 1 root root 43 Sep 11 2018 /usr/lib64/libopenblas.so.0 -> /etc/alternatives/libopenblas_pthreads.so.0 CMake will favor libopenblas, which is not necessarily a problem in and of itself, but I am trying to test an openmp application and cmake gets confused. I think this is an issue in how (open)blas is packaged. Ideally, either libblas should point to any available blas implementation (including the various libopenblas options), or blas should point to a generic libopenblas.so.0, which is itself an alternative that points to the available openblas libraries. Right now it's kind of doing both, and they end up conflicting. It would be simple enough to just remove the libopenblas_pthreads alternative and have libblas be in charge of all of them. I would be happy to submit a change if I knew where to look.

2 1

[opensuse-science] Python and MPI
by Stefan Brüns 08 Mar '19

08 Mar '19

Hi everyone, Currently, Python bindings to MPI-using libraries is one of the larger omissions we currently have - these are either not build for the MPI flavors at all, or the serial and MPI flavors are conflicting, as all install into the same prefix. I have had a look around how this can be solved, and think one of the better recipes can be found in https://fedoraproject.org/wiki/Packaging:MPI https://pagure.io/packaging-committee/issue/563 https://src.fedoraproject.org/rpms/openmpi/tree/master This works in the following way: Each MPI flavor (feel free to add e.g. compilers for HPC in your mind) sets 2 environment variables, one for Python2 and one for Python3 (this *could* be extended to more Python variants). Each variable contains the prefix for the MPI specific python prefix. This prefix then *extends* the regular python search path, e.g. for Python3: --- import sys, os; s = os.getenv('MPI_PYTHON3_SITEARCH'); s and (s in sys.path or sys.path.append(s)) --- This stanza is placed in a *.pth file [1] in the regular python prefix and gets loaded automatically. In case MPI_PYTHON*_SITEARCH is not set, this is a no-op. I would like to adopt this scheme for openSUSE/SLE. If this is accepted, there are still a few open points (I will use Python3 and openMPI hereafter, without loss of generality): 1. Prefix to use 2. Additional rpm macros 3. Setting the env variables - modules vs mpi-selector 1. - Regular Python prefix is /usr, with packages living in e.g. /usr/lib64/ python3.7/site-packages. openMPI uses a prefix of /usr/lib64/mpi/gcc/openmpi/ Proposal: MPI specific packages go into /usr/lib64/mpi/gcc/openmpi/lib64/mpi/ gcc/openmpi/ This would only require the typical call to %{pyton_install} to be ammended by a '--prefix /usr/lib64/mpi/gcc/openmpi/' Note - this is different to Fedora which uses %{python_sitearch}/%{flavor} 2. - We need some additional macros here. setup.py adds a suitable path to the prefix, but this is not available currently, we need something like % {python_mpi_sitearch} e.g. for the %files sections. 3. - Our HPC mpi modules would need suitable definitions of the env variables, same for the mpivars.sh scripts. Feedback welcome. Kind regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen home: +49 241 53809034 mobile: +49 151 50412019

2 1

Re: [opensuse-science] New package: DAKOTA
by Egbert Eich 01 Mar '19

01 Mar '19

Chris Coutinho writes: > Hi Egbert, > > I've fixed all the errors that were mentioned in the log file and > cleaned up some of the rpmlnt output. Now I've submitted the package > for review. Yup. Thanks! I've seen your submission. Will wait until the build is done and take a look. One thing that we may want to address is the inclusion of the libboost. It's a bit ugly to have to define a version. This is bound to break eventually. > Thanks again for your feedback. Sure. Cheers, Egbert. > > Chris. > > On Thu, 28 Feb 2019 at 15:35, Egbert Eich <eich(a)suse.com> wrote: > > > > > > Hi Chris, > > > > Chris Coutinho writes: > > > Hi Egbert, > > > > > > Thanks for you reply and feedback. I made some adjustments to the spec > > > file and removed all the spec files - now I'm stuck at one rpmlint > > > problem keeping me above a badness value of 1000, namely the > > > 'arch-dependent-file-in-usr-share' error. > > > > > > See my project build log for the error I'm talking about. > > > > I'm seeing this: > > [ 1436s] dakota-examples.x86_64: E: arch-dependent-file-in-usr-share (Badness: 590) /usr/share/dakota/examples/hopspack/1-var-bnds-only/var_bnds_only > > [ 1436s] dakota-examples.x86_64: E: arch-dependent-file-in-usr-share (Badness: 590) /usr/share/dakota/examples/hopspack/2-linear-constraints/linear_constraints > > [ 1436s] dakota-examples.x86_64: E: arch-dependent-file-in-usr-share (Badness: 590) /usr/share/dakota/examples/hopspack/3-degen-linear-constraints/degen_linear_constraints > > [...] > > > > I'd assume these files are binaries (ie have the execution flag set). > > These should not be installed in /usr/share. Instead you may want to > > consider to either move the entire structure /usr/share/dakota(/examples) > > to /usr/lib/dakota or at least do this with a subset of the files > > - like the example binaries. > > If there is no option to tell cmake to do this, you can move them > > 'manually' in the spec file afterwards. > > > > [ 1436s] dakota.x86_64: E: devel-file-in-non-devel-package (Badness: 50) /usr/lib/librol.so > > [ 1436s] dakota.x86_64: E: devel-file-in-non-devel-package (Badness: 50) /usr/lib/libteuchoscomm.so > > [ 1436s] dakota.x86_64: E: devel-file-in-non-devel-package (Badness: 50) /usr/lib/libteuchoscore.so > > [ 1436s] dakota.x86_64: E: devel-file-in-non-devel-package (Badness: 50) /usr/lib/libteuchosnumerics.so > > [...] > > > > are there also versioned so files of the same name? ie. some > > /usr/lib/librol.so.x.y.z? If so, the unversioned .so is just > > a link and should go to the devel package. > > Another issue might be that shared libraries on x86_64 should > > go to /usr/lib64/ not /usr/lib/. This is happening despite the > > %cmake macro adding these config parameters: > > -DLIB_INSTALL_DIR:PATH=/usr/lib64 -DCMAKE_INSTALL_LIBDIR:PATH=/usr/lib64 > > I guess you could either fix how the project uses cmake (and submit the > > patch upstream) or move things manually. > > > > [ 1436s] dakota-examples.x86_64: E: env-script-interpreter (Badness: 9) /usr/share/dakota/examples/parallelism/Case3-EvaluationTiling/OpenMPI/text_book_di_dynamic.py /usr/bin/env python > > [ 1436s] dakota-examples.x86_64: E: env-script-interpreter (Badness: 9) /usr/share/dakota/examples/script_interfaces/Python/rosenbrock_bb.py /usr/bin/env python > > [...] > > > > These things should be easy to fix. For HPC-style packages we have an RPM > > macro for it. > > > > > The binaries rpmlint is complaining about are related to the examples. > > > > Ok. > > > > > In principle, the binaries are specific to the directories they're in > > > - I don't think moving them to /usr/lib/ is really applicable. Any > > > tips? > > > > Why not? The LSB considers /usr/share as shareable among machines > > - even ones with different archiectures. /usr/lib on the other hand > > is considered arch-specific, thus it may contain arch-specific > > binaries as well. > > I haven't looked at the content of these directories, but if they > > contain data etc. that needs to live next to the binaries, you > > would have to put this into /usr/lib as well: there is nothing that > > would prevent you to put arch-independent stuff in there as well. > > > > Cheers, > > Egbert. > > > > > > > > Regards, > > > Chris > > > > > > On Thu, 28 Feb 2019 at 08:34, Egbert Eich <eich(a)suse.com> wrote: > > > > > > > > > > > > Hi Chris, > > > > > > > > Chris Coutinho writes: > > > > > 2. Currently it builds using a simple patch (known by the developers) > > > > > and ignoring all rpmlint warnings/errors. I'm assuming included packages > > > > > shouldn't just ignore all errors. What's the policy for dealing with > > > > > that? > > > > > > > > I've just looked at your package and the rpmlitrc file you've added. > > > > This file will definitely not go past the beam counters and quite a > > > > few things that ought to be fixed. > > > > > > > > These are Nonos and most are easy to fix: > > > > addFilter("dakota.* files-duplicate") > > > > addFilter("dakota.* library-without-ldconfig-postin") > > > > addFilter("dakota.* library-without-ldconfig-postun") > > > > addFilter("dakota.* no-changelogname-tag") > > > > addFilter("dakota.* non-executable-in-bin") > > > > addFilter("dakota.* non-executable-script") > > > > addFilter("dakota.* script-without-shebang") > > > > addFilter("dakota.* zero-length") > > > > addFilter("dakota.* arch-dependent-file-in-usr-share") > > > > addFilter("dakota.* devel-file-in-non-devel-package") > > > > addFilter("dakota.* env-script-interpreter") > > > > addFilter("dakota.* files-duplicated-waste") > > > > > > > > These depend, some might be false positives: > > > > addFilter("dakota.* position-independent-executable-suggested") > > > > addFilter("dakota.* shared-lib-without-dependency-information") > > > > addFilter("dakota.* shlib-policy-missing-suffix") > > > > > > > > I don't remember a single HPC/scientiffic library which doesn't > > > > trigger this - this I just ignore it: > > > > addFilter("dakota.* shared-lib-calls-exit") > > > > > > > > You may want to run: > > > > osc service run format_spec_file > > > > to sanitize your spec file. > > > > > > > > It looks like you've used the spec file from the upstream project. > > > > Unfortunately, spec files from upstream projects are not always > > > > up to the standards required by distributions. > > > > > > > > Cheers, > > > > Egbert. > > > > -- > > > > To unsubscribe, e-mail: opensuse-science+unsubscribe(a)opensuse.org > > > > To contact the owner, e-mail: opensuse-science+owner(a)opensuse.org > > > > > > > > -- > > Egbert Eich (Res. & Dev.) SUSE LINUX GmbH > > SUSE Labs - Project Manager HPC > > Tel: +49 911-740 53 0 http://www.suse.com > > ----------------------------------------------------------------- > > SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) > > Maxfeldstraße 5, D90409 Nürnberg, Germany-- > > To unsubscribe, e-mail: opensuse-science+unsubscribe(a)opensuse.org > > To contact the owner, e-mail: opensuse-science+owner(a)opensuse.org > > -- Egbert Eich (Res. & Dev.) SUSE LINUX GmbH SUSE Labs - Project Manager HPC Tel: +49 911-740 53 0 http://www.suse.com ----------------------------------------------------------------- SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5, D90409 Nürnberg, Germany-- To unsubscribe, e-mail: opensuse-science+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-science+owner(a)opensuse.org

1 0

Re: [opensuse-science] New package: DAKOTA
by Egbert Eich 01 Mar '19

01 Mar '19

Hi Chris, Chris Coutinho writes: > Hi Egbert, > > Thanks for you reply and feedback. I made some adjustments to the spec > file and removed all the spec files - now I'm stuck at one rpmlint > problem keeping me above a badness value of 1000, namely the > 'arch-dependent-file-in-usr-share' error. > > See my project build log for the error I'm talking about. I'm seeing this: [ 1436s] dakota-examples.x86_64: E: arch-dependent-file-in-usr-share (Badness: 590) /usr/share/dakota/examples/hopspack/1-var-bnds-only/var_bnds_only [ 1436s] dakota-examples.x86_64: E: arch-dependent-file-in-usr-share (Badness: 590) /usr/share/dakota/examples/hopspack/2-linear-constraints/linear_constraints [ 1436s] dakota-examples.x86_64: E: arch-dependent-file-in-usr-share (Badness: 590) /usr/share/dakota/examples/hopspack/3-degen-linear-constraints/degen_linear_constraints [...] I'd assume these files are binaries (ie have the execution flag set). These should not be installed in /usr/share. Instead you may want to consider to either move the entire structure /usr/share/dakota(/examples) to /usr/lib/dakota or at least do this with a subset of the files - like the example binaries. If there is no option to tell cmake to do this, you can move them 'manually' in the spec file afterwards. [ 1436s] dakota.x86_64: E: devel-file-in-non-devel-package (Badness: 50) /usr/lib/librol.so [ 1436s] dakota.x86_64: E: devel-file-in-non-devel-package (Badness: 50) /usr/lib/libteuchoscomm.so [ 1436s] dakota.x86_64: E: devel-file-in-non-devel-package (Badness: 50) /usr/lib/libteuchoscore.so [ 1436s] dakota.x86_64: E: devel-file-in-non-devel-package (Badness: 50) /usr/lib/libteuchosnumerics.so [...] are there also versioned so files of the same name? ie. some /usr/lib/librol.so.x.y.z? If so, the unversioned .so is just a link and should go to the devel package. Another issue might be that shared libraries on x86_64 should go to /usr/lib64/ not /usr/lib/. This is happening despite the %cmake macro adding these config parameters: -DLIB_INSTALL_DIR:PATH=/usr/lib64 -DCMAKE_INSTALL_LIBDIR:PATH=/usr/lib64 I guess you could either fix how the project uses cmake (and submit the patch upstream) or move things manually. [ 1436s] dakota-examples.x86_64: E: env-script-interpreter (Badness: 9) /usr/share/dakota/examples/parallelism/Case3-EvaluationTiling/OpenMPI/text_book_di_dynamic.py /usr/bin/env python [ 1436s] dakota-examples.x86_64: E: env-script-interpreter (Badness: 9) /usr/share/dakota/examples/script_interfaces/Python/rosenbrock_bb.py /usr/bin/env python [...] These things should be easy to fix. For HPC-style packages we have an RPM macro for it. > The binaries rpmlint is complaining about are related to the examples. Ok. > In principle, the binaries are specific to the directories they're in > - I don't think moving them to /usr/lib/ is really applicable. Any > tips? Why not? The LSB considers /usr/share as shareable among machines - even ones with different archiectures. /usr/lib on the other hand is considered arch-specific, thus it may contain arch-specific binaries as well. I haven't looked at the content of these directories, but if they contain data etc. that needs to live next to the binaries, you would have to put this into /usr/lib as well: there is nothing that would prevent you to put arch-independent stuff in there as well. Cheers, Egbert. > > Regards, > Chris > > On Thu, 28 Feb 2019 at 08:34, Egbert Eich <eich(a)suse.com> wrote: > > > > > > Hi Chris, > > > > Chris Coutinho writes: > > > 2. Currently it builds using a simple patch (known by the developers) > > > and ignoring all rpmlint warnings/errors. I'm assuming included packages > > > shouldn't just ignore all errors. What's the policy for dealing with > > > that? > > > > I've just looked at your package and the rpmlitrc file you've added. > > This file will definitely not go past the beam counters and quite a > > few things that ought to be fixed. > > > > These are Nonos and most are easy to fix: > > addFilter("dakota.* files-duplicate") > > addFilter("dakota.* library-without-ldconfig-postin") > > addFilter("dakota.* library-without-ldconfig-postun") > > addFilter("dakota.* no-changelogname-tag") > > addFilter("dakota.* non-executable-in-bin") > > addFilter("dakota.* non-executable-script") > > addFilter("dakota.* script-without-shebang") > > addFilter("dakota.* zero-length") > > addFilter("dakota.* arch-dependent-file-in-usr-share") > > addFilter("dakota.* devel-file-in-non-devel-package") > > addFilter("dakota.* env-script-interpreter") > > addFilter("dakota.* files-duplicated-waste") > > > > These depend, some might be false positives: > > addFilter("dakota.* position-independent-executable-suggested") > > addFilter("dakota.* shared-lib-without-dependency-information") > > addFilter("dakota.* shlib-policy-missing-suffix") > > > > I don't remember a single HPC/scientiffic library which doesn't > > trigger this - this I just ignore it: > > addFilter("dakota.* shared-lib-calls-exit") > > > > You may want to run: > > osc service run format_spec_file > > to sanitize your spec file. > > > > It looks like you've used the spec file from the upstream project. > > Unfortunately, spec files from upstream projects are not always > > up to the standards required by distributions. > > > > Cheers, > > Egbert. > > -- > > To unsubscribe, e-mail: opensuse-science+unsubscribe(a)opensuse.org > > To contact the owner, e-mail: opensuse-science+owner(a)opensuse.org > > -- Egbert Eich (Res. & Dev.) SUSE LINUX GmbH SUSE Labs - Project Manager HPC Tel: +49 911-740 53 0 http://www.suse.com ----------------------------------------------------------------- SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5, D90409 Nürnberg, Germany-- To unsubscribe, e-mail: opensuse-science+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-science+owner(a)opensuse.org

2 1