January 2018 - openSUSE Science - openSUSE Mailing Lists

Re: [opensuse-science] gnu-hpc packages
by Egbert Eich 31 Mar '18

31 Mar '18

Hi Peter, I did overlook your message as at the time you wrote it, I was extremely busy to get a release out. I need to apologize for the delay and that I'm only answering now. Peter Hill writes: > Hello, > > I just noticed the gnu-hpc packages in Tumbleweed recently, and I was > wondering if there is an explanation of how they work somewhere? I had > a look for some information or discussion, but I've not been able to > find anything so far. Right, I should probably have blogged about this - and I may still do. > > I gather they are linked somehow to environment modules, but it's not > clear to me how exactly. For instance, "petsc-devel" installs a module > for PETSc, but "hdf5-devel" does not install one for HDF5. Are there > certain packages we're supposed to install to get this functionality? This could have been a bug that has been fixed meanwhile. Let me get a very brief overview: The theory is that 1. the runtime library package installs a module. 2. the devel package pulls in the runtime library package and any other package that may be needed for building. 3. There are packages which have the version numbers in the package name. Other packages don't - these are the 'master packages' which will pull in the latest version of a package thru a dependency. This allows to have mutliple version of the same base code. 4. Environment modules are used to select toolchain flavor/version, MPI flavor/version and packages and versions. By default, the latest version of a package is loaded, however, you can specify a version explicitly if you need an older version. The modules enforce some dependencies between libraries, ie you will have to select a toolchain for most of the libraries and you need to select the MPI version for MPI dependent libraries. Without this, these libraries will not become available. 5. For environment modules we use Lmod: https://lmod.readthedocs.io/en/latest/ There is the 'module' command available to load/unload, list, switch the modules. To get a help, call 'module help', modules are loaded with 'module load <module_name>[/<version>]', to get a list of accessible modules, do 'module avail', to get a list of all modules, call'module spider'. > Also, are they supposed to play nice with pkg-config? petsc-devel does > install PETSc.pc, but loading the module doesn't update > $PKG_CONFIG_PATH to be able to find it. This is a bug as well, lemme check if it is still there ... The module file of the package I'm looking at right now has: prepend-path PKG_CONFIG_PATH /usr/lib/hpc/gnu7.2/openmpi3/petsc/3.7.6/lib64/pkgconfig Thus it should work. If you find any more issues, please let me know - or create a ticket on bugzilla.opensuse.org. Please add hpc-maintainer-bugs(a)forge.provo.novell.com as the assignee. Plan is to provide these packages with openSUSE Leap 42.3. This should give you a more stable (and enterprise grade!) environment than openSUSE Tumbleweed. Cheers, Egbert. -- To unsubscribe, e-mail: opensuse-science+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-science+owner(a)opensuse.org

2 1

[opensuse-science] Re: HDF5 v.1.8 in Project science OBS
by Egbert Eich 19 Jan '18

19 Jan '18

Adrian Schröter writes: > On Freitag, 19. Januar 2018, 15:16:04 CET wrote Egbert Eich: > > > > Yeah, I didn't want to point to you directly ;) > > ah, the shy Egbert ... Hehe ;) > > You mean the use-for-build flags? I was under the *assumption* that > > hdf5 has them set. It would explain why things break for the HPC > > flavors of the netcdf build: the hdf5.1.8 package doesn't generate > > these flavors as the package has not been converted. > > no, they were set to use only the 1.8 version. > > This was (also?) be done since 1.10 has a different on disk format. > While it should still be able to read 1.8 it will break the workflow > for people who share the their files (as FreeCAD users). FreeCAD doesn't look like the classic HPC application while HDF5 is specifically targetting the problems of HPC users and environments - storing very large files on cluster file systems. These problems arise because this file format library is used for use cases it was never intended for :/ > > > I really should have written down the reason for the 5-1.8 package > > > in the package description, but I didn't :/ > > > > > > So, I will reconfigure it to use the 5-1.10 version now to see > > > what was the reason behind (most likely opencascade & FreeCAD). > > > > We do have hdf5 1.10.1 in Factory only, so anyone needing to build > > against hdf5 and is available in Factory would do so already. > > k, will keep an eye on the project. 5-1.8 is currently disabled and > not used, just not dropped yet ... As I said, we can keep it, but the package should be renamed (and packages that depend on it should explicitely use this package name to make sure they get the 'stable' version). Cheers, Egbert. -- To unsubscribe, e-mail: opensuse-science+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-science+owner(a)opensuse.org

1 0

[opensuse-science] Re: HDF5 v.1.8 in Project science OBS
by Egbert Eich 19 Jan '18

19 Jan '18

Adrian Schröter writes: > On Freitag, 19. Januar 2018, 15:26:02 CET wrote Adrian Schröter: > > On Freitag, 19. Januar 2018, 15:16:04 CET wrote Egbert Eich: > ... > > med-tools are failing now, because they explicit warn about 1.10 due > to on disk incompabilities: > > [ 206s] This HDF5 version 1.10.1 must not be used with med-fichier3.2.1. > [ 206s] The HDF5 library version used by med-fichier3.y.z MUST NOT be > 1.8 and have to be at least 5-1.8.11. > [ 206s] DO NOT TRY TO COMPILE med-fichier3.2.1 version with an HDF5 library which would generate an hdf5 file not compliant with HDF5-1.8.z library. > [ 206s] This would BREAK med-fichier compatibility between files with the same revision number ! > > > I can of course patch this but it feels a bit bad if they warn that way... > > Will try to find out more about it and leave the build broken atm, > so the published repository should not build anything what breaks the > files a user is storing to disk... > Let's include the version number into the package name of hdf5.1.8 then. This is what we do for HPC packages all the time. If hdf5 1.8 is considered to be some sort of reference base, we could flag it as such and build packages that require a stable reference base on it. This still doesn't give you files which you can exchange with your friends who are using tools from other distros with different 'reference bases' ... In theory, we could use environment modules to make other packages available only for the hdf5 version they have been built for. Our packaging macros for HPC are currently not set up for this. It could be done but would make things even more ugly. I at least hope these packages do not use the MPI version of hdf5. Generally, in an HPC environment, these issues don't arise as people stick to the version they have started their project with. This is why for a distro (like Leap) we keep the old versions around. Cheers, Egbert. -- To unsubscribe, e-mail: opensuse-science+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-science+owner(a)opensuse.org

1 0

[opensuse-science] Re: HDF5 v.1.8 in Project science OBS
by Egbert Eich 19 Jan '18

19 Jan '18

Hi Adrian! Adrian Schröter writes: > On Freitag, 19. Januar 2018, 14:41:29 CET wrote Egbert Eich: > > > > Hi! > > > > About 7 month ago, hdv5 v 1.8 was copypac-ed from science:unstable > > to science: > > https://build.opensuse.org/package/revisions/science/hdf5.1.8 > > > > > > The package (directory-)name was set to hdf5.1.8 to not > > conflict with the already existing package 'hdf5, which > > was updated to 1.10 roughly 1.5 years ago already. > > right, was me, I believe ;) Yeah, I didn't want to point to you directly ;) > > Unfortunately, the spec file and package names do not > > carry the ".1.8". > > Unfortunately, due to not carrying thru the naming scheme > > to the package name inside the spec-file, both packages > > generate the same files - with different version numbers. > > This breaks the build of packages depending on the hdf5- > > package which is in Factory. > > This is true, but bother versions are available for download. > You can only build against one of them though. For netcdf $ osc -A https://api.opensuse.org results -v science netcdf:gnu-hpc \ -a x86_64 -r openSUSE_Tumbleweed returns: openSUSE_Tumbleweed x86_64 unresolvable: nothing provides hdf5-devel-data = 1.10.1 needed by hdf5_1_10_1-gnu-hpc-devel You are right, on download.opensuse.org, both versions are available: $ wget -q -O - http://download.opensuse.org/repositories/science/openSUSE_Tumbleweed/x86_6… \ | grep hdf5-devel-data [..] <a href="hdf5-devel-data-1.8.17-2.13.x86_64.rpm">hdf5-devel-data-1.8.17-2.13.x86_64.rpm</a> [..] <a href="hdf5-devel-data-1.10.1-72.1.x86_64.rpm">hdf5-devel-data-1.10.1-72.1.x86_64.rpm</a> [..] > > The package has no maintainer, so there is noone to ask to > > fix this. > > > > If anyone here still needs version 1.8 of hdf5, please speak > > up and fix this package *ASAP*! > > > > If not, I will assume there is noone who cares any more and > > I will request deletion. > > > > For now, I've disabled the build. > > Does not change much, only the 5.1.8 is used for building against > atm via the flags. You mean the use-for-build flags? I was under the *assumption* that hdf5 has them set. It would explain why things break for the HPC flavors of the netcdf build: the hdf5.1.8 package doesn't generate these flavors as the package has not been converted. > I really should have written down the reason for the 5-1.8 package > in the package description, but I didn't :/ > > So, I will reconfigure it to use the 5-1.10 version now to see > what was the reason behind (most likely opencascade & FreeCAD). We do have hdf5 1.10.1 in Factory only, so anyone needing to build against hdf5 and is available in Factory would do so already. Thanks for your quick reply! Cheers, Egbert. -- To unsubscribe, e-mail: opensuse-science+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-science+owner(a)opensuse.org

2 2

[opensuse-science] HDF5 v.1.8 in Project science OBS
by Egbert Eich 19 Jan '18

19 Jan '18

Hi! About 7 month ago, hdv5 v 1.8 was copypac-ed from science:unstable to science: https://build.opensuse.org/package/revisions/science/hdf5.1.8 The package (directory-)name was set to hdf5.1.8 to not conflict with the already existing package 'hdf5, which was updated to 1.10 roughly 1.5 years ago already. Unfortunately, the spec file and package names do not carry the ".1.8". Unfortunately, due to not carrying thru the naming scheme to the package name inside the spec-file, both packages generate the same files - with different version numbers. This breaks the build of packages depending on the hdf5- package which is in Factory. The package has no maintainer, so there is noone to ask to fix this. If anyone here still needs version 1.8 of hdf5, please speak up and fix this package *ASAP*! If not, I will assume there is noone who cares any more and I will request deletion. For now, I've disabled the build. Cheers, Egbert. -- Egbert Eich (Res. & Dev.) SUSE LINUX GmbH SUSE Labs - Project Manager HPC Tel: +49 911-740 53 0 http://www.suse.com ----------------------------------------------------------------- SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5, D90409 Nürnberg, Germany-- To unsubscribe, e-mail: opensuse-science+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-science+owner(a)opensuse.org

2 1