[opensuse] Python: OBS vs PIP
I am trying to get tensorflow and related packages installed on openSUSE Leap 15.1 and/or Tumbleweed. We have a package from a consulting company who have developed with tensorflow 1.14, but OBS has only 1.15. Seems this is not compatible. I can install tensorflow via zypper or pip3. It seems they install the same version, which looks to be the version in OBS. My question is: what is the relationship between packages built in OBS and that are also available via pip3? I looked around in the openSUSE docs and there is lots on how best to build a package on OBS (via py2pack that looks very useful!). But that is from the developer's view. What about from the package user's view? Most Python packages have a setup functionality that get the needed bits via pip. Is that effectively just getting thighs from OBS? The reason I used tensorflow as an example is that for many systems (Ubuntu as an example) older versions of packages seem to be available via pip. Is the reason that I do not see older versions of tensorflow for openSUSE in pip because these older versions are no longer available in OBS? -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hi On 9/8/20 9:35 PM, Roger Oberholtzer wrote:
I am trying to get tensorflow and related packages installed on openSUSE Leap 15.1 and/or Tumbleweed.
We have a package from a consulting company who have developed with tensorflow 1.14, but OBS has only 1.15. Seems this is not compatible.
I can install tensorflow via zypper or pip3. It seems they install the same version, which looks to be the version in OBS.
My question is: what is the relationship between packages built in OBS and that are also available via pip3? I looked around in the openSUSE docs and there is lots on how best to build a package on OBS (via py2pack that looks very useful!). But that is from the developer's view. What about from the package user's view? Most Python packages have a setup functionality that get the needed bits via pip. Is that effectively just getting thighs from OBS?
The difference is they both have slightly different aims and goals, the goal of the distro packages is probably most clear, the aim of the python packages in the distro is to provide a stable platform for all the python apps in the distro (and that users may also choose to use). The important thing here is that we know that each python lib works well with each other lib, python packages in the distro often get there source code from pip as a starting point but they get all there dependencies from other distro packages not pip. This also means particularly in Leap we might use a slightly older version then in pip because we know it works with everything else, it also tends to mean that we only ship one version of each package. From that the aim of the packages in obs is mostly to keep the python packages in tumbleweed up to date and working in the way I described above this also means that when we create the next major version of SLE and Leap we have a fair idea that the python stack will already work pretty well together.
The reason I used tensorflow as an example is that for many systems (Ubuntu as an example) older versions of packages seem to be available via pip. Is the reason that I do not see older versions of tensorflow for openSUSE in pip because these older versions are no longer available in OBS?
I don't think so, the content in pip is uploaded by the tensorflow developers so for some reason they have chosen to drop support for that version. Tensorflow is an interesting example because I believe it calls into a bunch of C libs and it could be that they can't build those libs or are choosing not to build those libs for older versions on openSUSE. You will really need to ask the tensorflow devs why something isn't available on pip, you may still be able to build it from source. Generally when developing a python app you have two choices if you don't care too much about the library versions you need just using the distro python libs will probably save you some headaches, (Some SUSE customers use SLE with LTSS for this because they know they will have 5 years of python libs not really changing). If you really care about certain versions then you really need to manage everything yourself by using a python virtualenv and installing everything from pip which is probably the more common way for businesses etc to do it but it then means you need to keep on top of security fixes etc. What really isn't recommended is using a mix of system and pip packages, that will probably break at some point. Cheers -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Wed, Sep 9, 2020 at 4:40 AM Simon Lees <sflees@suse.de> wrote:
The reason I used tensorflow as an example is that for many systems (Ubuntu as an example) older versions of packages seem to be available via pip. Is the reason that I do not see older versions of tensorflow for openSUSE in pip because these older versions are no longer available in OBS?
I don't think so, the content in pip is uploaded by the tensorflow developers so for some reason they have chosen to drop support for that version. Tensorflow is an interesting example because I believe it calls into a bunch of C libs and it could be that they can't build those libs or are choosing not to build those libs for older versions on openSUSE. You will really need to ask the tensorflow devs why something isn't available on pip, you may still be able to build it from source.
Indeed. Packages with binary components make things more complicated. I will probably add tensorflow 1.14 to OBS (I already build things there). Isee that someone else has 1.14 in OBS. But not the Pyhton part, It sounds like there are some tensorflow developers who are building for openSUSE and putting things in PIP. I wonder if they are the same ones that build the package in OBS.
If you really care about certain versions then you really need to manage everything yourself by using a python virtualenv and installing everything from pip which is probably the more common way for businesses etc to do it but it then means you need to keep on top of security fixes etc. What really isn't recommended is using a mix of system and pip packages, that will probably break at some point.
The Python code I need to use was provided by consultants who have said that they need the specific versions of the various Python packages as they have defined in their setup procedure. Their claim is that some things they do are not supported in newer versions. I can't say if that is true or just laziness on their part to update their code to stay current with the packages they use. I suspect the latter. They run on Ubuntu and have been pressing us to run this in Azure. That's not going to happen. Our ultimate goal is to use only the C libraries for all this. but we have to work through what they have done with the deep neural network (looking at painted road markings). The only things I have not been able to satisfy are tensorflow and tensorboard. Of course, when those are available, other missing dependencies might appear. I know what I'm working on the next day or so... -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 9/9/20 5:18 PM, Roger Oberholtzer wrote:
On Wed, Sep 9, 2020 at 4:40 AM Simon Lees <sflees@suse.de> wrote:
The reason I used tensorflow as an example is that for many systems (Ubuntu as an example) older versions of packages seem to be available via pip. Is the reason that I do not see older versions of tensorflow for openSUSE in pip because these older versions are no longer available in OBS?
I don't think so, the content in pip is uploaded by the tensorflow developers so for some reason they have chosen to drop support for that version. Tensorflow is an interesting example because I believe it calls into a bunch of C libs and it could be that they can't build those libs or are choosing not to build those libs for older versions on openSUSE. You will really need to ask the tensorflow devs why something isn't available on pip, you may still be able to build it from source.
Indeed. Packages with binary components make things more complicated. I will probably add tensorflow 1.14 to OBS (I already build things there). Isee that someone else has 1.14 in OBS. But not the Pyhton part,
It sounds like there are some tensorflow developers who are building for openSUSE and putting things in PIP. I wonder if they are the same ones that build the package in OBS.
If you really care about certain versions then you really need to manage everything yourself by using a python virtualenv and installing everything from pip which is probably the more common way for businesses etc to do it but it then means you need to keep on top of security fixes etc. What really isn't recommended is using a mix of system and pip packages, that will probably break at some point.
The Python code I need to use was provided by consultants who have said that they need the specific versions of the various Python packages as they have defined in their setup procedure. Their claim is that some things they do are not supported in newer versions. I can't say if that is true or just laziness on their part to update their code to stay current with the packages they use. I suspect the latter. They run on Ubuntu and have been pressing us to run this in Azure. That's not going to happen. Our ultimate goal is to use only the C libraries for all this. but we have to work through what they have done with the deep neural network (looking at painted road markings).
The only things I have not been able to satisfy are tensorflow and tensorboard. Of course, when those are available, other missing dependencies might appear.
I know what I'm working on the next day or so...
Some SUSE people have been working on getting these packaged, you might have some luck with tensorflow by taking the 1.15 package [1] and dropping the version back down. I started work on tensorboard at one point before getting moved onto another project [2] if you manage to get further then I did i'd be interested to here about it as would some others. 1. https://build.opensuse.org/package/show/science:machinelearning/tensorflow 2. https://build.opensuse.org/package/show/home:simotek:ml/tensorboard -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Wed, Sep 9, 2020 at 12:10 PM Simon Lees <sflees@suse.de> wrote:
Some SUSE people have been working on getting these packaged, you might have some luck with tensorflow by taking the 1.15 package [1] and dropping the version back down.
I was looking at that. But there are so many patches. it's a rather complicated package. I may still give it a try. I also tried to build on Tumbleweed using bezel. It got to a point where it was complaining about compiling some cpp file. I suspect that this might be related to the version of gcc that is used. Not sure.
I started work on tensorboard at one point before getting moved onto another project [2] if you manage to get further then I did i'd be interested to here about it as would some others.
I will do so. I might also break down and do this in a docker image. And the developers suggested Anaconda.But I don't see how that will solve coming up with tensorflow 1.14. It's not the Python stuff that is the problem in doing the build. It's the whole environment.
1. https://build.opensuse.org/package/show/science:machinelearning/tensorflow
2. https://build.opensuse.org/package/show/home:simotek:ml/tensorboard
I will look at these to see if they help. Thanks! I just wish the exercise was not getting an old version of tensorflow to build. Seems like the time should be spent moving the package to a more recent tensorflow... -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 9/10/20 3:59 PM, Roger Oberholtzer wrote:
On Wed, Sep 9, 2020 at 12:10 PM Simon Lees <sflees@suse.de> wrote:
Some SUSE people have been working on getting these packaged, you might have some luck with tensorflow by taking the 1.15 package [1] and dropping the version back down.
I was looking at that. But there are so many patches. it's a rather complicated package. I may still give it a try.
Yeah the biggest issue with these packages is upstream uses bazel to download and build a bunch of dependencies. We can't do that on obs so most of the patches are making it build with versions on the system rather then re downloading them.
I also tried to build on Tumbleweed using bezel. It got to a point where it was complaining about compiling some cpp file. I suspect that this might be related to the version of gcc that is used. Not sure.
That is a possibility tumbleweed's gcc is pretty new, can you paste the error somewhere it might be possible to get around it by disabling a compiler warning / error that didn't used to be on by default. Cheers -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
participants (2)
-
Roger Oberholtzer -
Simon Lees