[opensuse-python] Python and MPI
Hi everyone, Currently, Python bindings to MPI-using libraries is one of the larger omissions we currently have - these are either not build for the MPI flavors at all, or the serial and MPI flavors are conflicting, as all install into the same prefix. I have had a look around how this can be solved, and think one of the better recipes can be found in https://fedoraproject.org/wiki/Packaging:MPI https://pagure.io/packaging-committee/issue/563 https://src.fedoraproject.org/rpms/openmpi/tree/master This works in the following way: Each MPI flavor (feel free to add e.g. compilers for HPC in your mind) sets 2 environment variables, one for Python2 and one for Python3 (this *could* be extended to more Python variants). Each variable contains the prefix for the MPI specific python prefix. This prefix then *extends* the regular python search path, e.g. for Python3: --- import sys, os; s = os.getenv('MPI_PYTHON3_SITEARCH'); s and (s in sys.path or sys.path.append(s)) --- This stanza is placed in a *.pth file [1] in the regular python prefix and gets loaded automatically. In case MPI_PYTHON*_SITEARCH is not set, this is a no-op. I would like to adopt this scheme for openSUSE/SLE. If this is accepted, there are still a few open points (I will use Python3 and openMPI hereafter, without loss of generality): 1. Prefix to use 2. Additional rpm macros 3. Setting the env variables - modules vs mpi-selector 1. - Regular Python prefix is /usr, with packages living in e.g. /usr/lib64/ python3.7/site-packages. openMPI uses a prefix of /usr/lib64/mpi/gcc/openmpi/ Proposal: MPI specific packages go into /usr/lib64/mpi/gcc/openmpi/lib64/mpi/ gcc/openmpi/ This would only require the typical call to %{pyton_install} to be ammended by a '--prefix /usr/lib64/mpi/gcc/openmpi/' Note - this is different to Fedora which uses %{python_sitearch}/%{flavor} 2. - We need some additional macros here. setup.py adds a suitable path to the prefix, but this is not available currently, we need something like % {python_mpi_sitearch} e.g. for the %files sections. 3. - Our HPC mpi modules would need suitable definitions of the env variables, same for the mpivars.sh scripts. Feedback welcome. Kind regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen home: +49 241 53809034 mobile: +49 151 50412019
On Mittwoch, 6. März 2019 01:37:30 CET Stefan Brüns wrote:
I have created a _multibuild for mpi4py, which uses these conventions: https://build.opensuse.org/package/show/ home:StefanBruens:branches:devel:languages:python:numeric/python-mpi4py The package redefines the %{python_sitearch} macro to accept a prefix parameter, i.e. %{python_sitearch %{mpi_prefix}}, where %{mpi_prefix} is defined to e.g. /usr/lib64/mpi/gcc/openmpi/. This macro should of course be part of the regular python macro package. The prefix should be optional, or the macro be renamed. I have reused the name as otherwise it would not be substituted correctly by the lua code. Kind regards, Stefan -- To unsubscribe, e-mail: opensuse-python+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-python+owner@opensuse.org
On Mittwoch, 6. März 2019 01:37:30 CET Stefan Brüns wrote:
I have created a _multibuild for mpi4py, which uses these conventions: https://build.opensuse.org/package/show/ home:StefanBruens:branches:devel:languages:python:numeric/python-mpi4py The package redefines the %{python_sitearch} macro to accept a prefix parameter, i.e. %{python_sitearch %{mpi_prefix}}, where %{mpi_prefix} is defined to e.g. /usr/lib64/mpi/gcc/openmpi/. This macro should of course be part of the regular python macro package. The prefix should be optional, or the macro be renamed. I have reused the name as otherwise it would not be substituted correctly by the lua code. Kind regards, Stefan -- To unsubscribe, e-mail: opensuse-python+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-python+owner@opensuse.org
participants (2)
-
Brüns, Stefan
-
Stefan Brüns