On Mon, Nov 18, 2019 at 6:58 PM William Brown wrote:

Hi,

Recently I've been trying to package some utilities for SUSE and Fedora via OBS that are written in Rust. I'm running into a bit of a problem though.

Rust as a language has a very aggressive release cycle compared to what we expect on a platform like SUSE - 6 weeks (https://github.com/rust-lang/rust/blob/master/RELEASES.md). In each cycle Rust releases new features, and despite the core language being "stable" and safe (which is great!) new features tend to be used very rapidly by library authors. For example the release from Version 1.34.0 (2019-04-11) contains convert::TryFrom, which is now in use by a large number of libraries.

Rust also has an (unfortunate) requirement that you are essentially forced to use cargo which is a build and dependency management tool. Cargo is extremely opinionated and inflexible which makes it difficult to use. However as it's also a dependency management tool, this has encouraged an npm-style ecosystem of dependencies to spring up on https://crates.io/ - and it's effectively the only way to use libraries in Rust. But additionally, it's promoted a system where a library may have a large number of small dependencies too.

Due to the fast release cycle, developers aggressively using new features, and the npm style micro dependency system we have a recipe for problems - if you are not using the latest stable compiler, it's extremely likely that your libraries, or their dependents may not build on your project. Which is exactly the issue I have run into where rust on fedora 30 and opensuse leap 15.1 are simply too old to support the features that have been used in the last 6 months by library developers.

Which leads me to the question of "what to do".

* I don't believe it's feasible to ask Rust to "slow down". It's just not going to happen, and they will keep adding features that people will "want" to use. * We can't expect people *not* to update their dependencies in projects as that would prevent security updates being included. So we have to accept projects that will use "latest and greatest" complier features somewhere in their dependency graph. * Which leaves distros (like us) speeding up our rust compiler cycle somehow.

For opensuse I can probably do something like adding devel:languages:rust as a respository to my project. Would this strategy be viable on SLE? What about for projects which we want to expose packages for fedora or other? Are we able to have toolchains move faster than our base system?

I'm really looking for ideas on how we can sustainably use rust projects within the governance and social constraints that exist in the space. Ideas?

Unfortunately, SUSE rust compiler packaging spec isn't the same as the RH/Fedora/Mageia one, so essentially it takes double the work to keep things up to date everywhere. If we integrated the SUSE support bits into the RH/Fedora/Mageia rust packaging spec, that could make things easier. It's not exactly the highest thing on my list to look into right now, though... That said, Fedora and Mageia update the Rust compiler within a week of a new stable compiler release. As of right now, Fedora 29 and higher all have Rust 1.39.0. CentOS/RHEL 7 will receive Rust 1.39.0 soon too[1]. Crate packaging and macros are completely in sync among Fedora, Mageia, and openSUSE. These are provided by the rust-packaging and rust-srpm-macros packages. This code is pretty much the same across all three distributions. I find the process of shipping updates to SUSE distributions thoroughly annoying and unappealing, so I don't personally do anything to target anything other than Tumbleweed. The process is too slow because it's often bottlenecked on either legal-auto or the all-too-tiny number of reviewers for SRs and MRs. This is also true for Tumbleweed, but it seems to be somewhat faster there. I'd be more interested in helping ship updates to Leap if we could make this process less awful. [1]: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-037907db2c -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 19 Nov 2019, at 10:43, William Brown wrote:

...

On 19 Nov 2019, at 10:39, Neal Gompa wrote:

On Mon, Nov 18, 2019 at 6:58 PM William Brown wrote:

...

Hi,

Recently I've been trying to package some utilities for SUSE and Fedora via OBS that are written in Rust. I'm running into a bit of a problem though.

Rust as a language has a very aggressive release cycle compared to what we expect on a platform like SUSE - 6 weeks (https://github.com/rust-lang/rust/blob/master/RELEASES.md). In each cycle Rust releases new features, and despite the core language being "stable" and safe (which is great!) new features tend to be used very rapidly by library authors. For example the release from Version 1.34.0 (2019-04-11) contains convert::TryFrom, which is now in use by a large number of libraries.

Rust also has an (unfortunate) requirement that you are essentially forced to use cargo which is a build and dependency management tool. Cargo is extremely opinionated and inflexible which makes it difficult to use. However as it's also a dependency management tool, this has encouraged an npm-style ecosystem of dependencies to spring up on https://crates.io/ - and it's effectively the only way to use libraries in Rust. But additionally, it's promoted a system where a library may have a large number of small dependencies too.

Due to the fast release cycle, developers aggressively using new features, and the npm style micro dependency system we have a recipe for problems - if you are not using the latest stable compiler, it's extremely likely that your libraries, or their dependents may not build on your project. Which is exactly the issue I have run into where rust on fedora 30 and opensuse leap 15.1 are simply too old to support the features that have been used in the last 6 months by library developers.

Which leads me to the question of "what to do".

* I don't believe it's feasible to ask Rust to "slow down". It's just not going to happen, and they will keep adding features that people will "want" to use. * We can't expect people *not* to update their dependencies in projects as that would prevent security updates being included. So we have to accept projects that will use "latest and greatest" complier features somewhere in their dependency graph. * Which leaves distros (like us) speeding up our rust compiler cycle somehow.

For opensuse I can probably do something like adding devel:languages:rust as a respository to my project. Would this strategy be viable on SLE? What about for projects which we want to expose packages for fedora or other? Are we able to have toolchains move faster than our base system?

I'm really looking for ideas on how we can sustainably use rust projects within the governance and social constraints that exist in the space. Ideas?

Unfortunately, SUSE rust compiler packaging spec isn't the same as the RH/Fedora/Mageia one, so essentially it takes double the work to keep things up to date everywhere. If we integrated the SUSE support bits into the RH/Fedora/Mageia rust packaging spec, that could make things easier. It's not exactly the highest thing on my list to look into right now, though...

That said, Fedora and Mageia update the Rust compiler within a week of a new stable compiler release. As of right now, Fedora 29 and higher all have Rust 1.39.0. CentOS/RHEL 7 will receive Rust 1.39.0 soon too[1].

Hmmmm, I'm seeing that on Fedora 30 on OBS that this is failing due to a lack of try_from:

https://build.opensuse.org/package/live_build_log/home:firstyear:kanidm/kani...

So this indicates that our Fedora 30 in obs may not be getting these updates?

Okay, so when you do: Project -> Repositories -> Add from a distribution -> select fedora 30 That only adds fedora 30 standard: https://build.opensuse.org/repositories/Fedora:30 But not update. This is why all the f30 builds I have are failing. This seems like a bug or error to me.

...

Crate packaging and macros are completely in sync among Fedora, Mageia, and openSUSE. These are provided by the rust-packaging and rust-srpm-macros packages. This code is pretty much the same across all three distributions.

I find the process of shipping updates to SUSE distributions thoroughly annoying and unappealing, so I don't personally do anything to target anything other than Tumbleweed. The process is too slow because it's often bottlenecked on either legal-auto or the all-too-tiny number of reviewers for SRs and MRs. This is also true for Tumbleweed, but it seems to be somewhat faster there. I'd be more interested in helping ship updates to Leap if we could make this process less awful.

I think this process has to become less awful, else we'll be missing out on plenty of rust tooling that is quickly coming available for platforms.

...

[1]: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-037907db2c

-- 真実はいつも一つ！/ Always, there's only one truth!

— Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server SUSE Labs

-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

— Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 19 Nov 2019, at 11:04, Neal Gompa wrote:

On Mon, Nov 18, 2019 at 7:32 PM William Brown wrote:

...

Okay, so when you do:

Project -> Repositories -> Add from a distribution -> select fedora 30

That only adds fedora 30 standard:

https://build.opensuse.org/repositories/Fedora:30

But not update. This is why all the f30 builds I have are failing. This seems like a bug or error to me.

It is done this way to minimize automatic rebuilds by default, as new updated packages in the repositories will trigger rebuilds. If you want updates, just change "standard" to "update" and call it gravy. Most don't require this, but your Rust case probably will.

For what it's worth, openSUSE Leap targets are similarly set up.

This is really counter intuitive to anyone who is reasonably inexperienced at packaging (IE me), which is probably another barrier to getting people to do tasks like this. It could be improved both as a default (include updates because that's the expected behaviour) or via UI (offer updates as an option in the distribution menu, or make it easier to alter standard -> update in the respository list).

-- 真実はいつも一つ！/ Always, there's only one truth!

— Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 19 Nov 2019, at 11:19, Simon Lees wrote:

On 11/19/19 11:07 AM, William Brown wrote:

...

...

On 19 Nov 2019, at 11:04, Neal Gompa wrote:

On Mon, Nov 18, 2019 at 7:32 PM William Brown wrote:

...

Okay, so when you do:

Project -> Repositories -> Add from a distribution -> select fedora 30

That only adds fedora 30 standard:

https://build.opensuse.org/repositories/Fedora:30

But not update. This is why all the f30 builds I have are failing. This seems like a bug or error to me.

It is done this way to minimize automatic rebuilds by default, as new updated packages in the repositories will trigger rebuilds. If you want updates, just change "standard" to "update" and call it gravy. Most don't require this, but your Rust case probably will.

For what it's worth, openSUSE Leap targets are similarly set up.

This is really counter intuitive to anyone who is reasonably inexperienced at packaging (IE me), which is probably another barrier to getting people to do tasks like this. It could be improved both as a default (include updates because that's the expected behaviour) or via UI (offer updates as an option in the distribution menu, or make it easier to alter standard -> update in the respository list).

One reason that it is done like this by default is because core SLE / Leap packages shouldn't be break API/ABI for updates, so in the vast majority of cases building against standard is fine and saves huge amounts of resources on obs. You are only really hitting this issue because you are working with one of the few echosystems that as you say doesn't follow our traditional model.

Ecosystems are changing, and less and less are following the traditional model - I think we can't keep treating them as exceptions, because this will become the norm for many languages. So how can we improve this? How can we make rust a first class language for our platforms so that we can keep including relevant and modern tools in our platforms?

--

Simon Lees (Simotek) http://simotek.net

Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B

— Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 19 Nov 2019, at 12:27, Simon Lees wrote:

On 11/19/19 11:22 AM, William Brown wrote:

...

...

On 19 Nov 2019, at 11:19, Simon Lees wrote:

On 11/19/19 11:07 AM, William Brown wrote:

...

...

On 19 Nov 2019, at 11:04, Neal Gompa wrote:

On Mon, Nov 18, 2019 at 7:32 PM William Brown wrote:

...

Okay, so when you do:

Project -> Repositories -> Add from a distribution -> select fedora 30

That only adds fedora 30 standard:

https://build.opensuse.org/repositories/Fedora:30

But not update. This is why all the f30 builds I have are failing. This seems like a bug or error to me.

It is done this way to minimize automatic rebuilds by default, as new updated packages in the repositories will trigger rebuilds. If you want updates, just change "standard" to "update" and call it gravy. Most don't require this, but your Rust case probably will.

For what it's worth, openSUSE Leap targets are similarly set up.

This is really counter intuitive to anyone who is reasonably inexperienced at packaging (IE me), which is probably another barrier to getting people to do tasks like this. It could be improved both as a default (include updates because that's the expected behaviour) or via UI (offer updates as an option in the distribution menu, or make it easier to alter standard -> update in the respository list).

One reason that it is done like this by default is because core SLE / Leap packages shouldn't be break API/ABI for updates, so in the vast majority of cases building against standard is fine and saves huge amounts of resources on obs. You are only really hitting this issue because you are working with one of the few echosystems that as you say doesn't follow our traditional model.

Ecosystems are changing, and less and less are following the traditional model - I think we can't keep treating them as exceptions, because this will become the norm for many languages.

So how can we improve this? How can we make rust a first class language for our platforms so that we can keep including relevant and modern tools in our platforms?

This is not just a tooling problem but also a process problem, the Haskell team have managed to script there entire process, there scripts can automatically push updates too there 2500 packages. This then revealed a different set of issues where our manual package review team and legal team didn't have enough manpower to regularly deal with the extra 2500 requests and I think for now the end solution was to cut back and just have the most useful packages in factory and the rest in a devel repo.

So it has been shown that atleast with one platform once the core is there and working, handling the frequent updates of various packages can be largely automated at least for tumbleweed as long as they are capable of doing offline builds.

For SUSE there is definitely a discussion that should be had around whether we should promise binary compatibility for the whole stack but that's a discussion for SUSE's product managers etc.

Who needs to be contacted to begin this discussion and get actions happening - I'm in a situation where I will be making rust a requirement for 389-ds soon, so this needs to have solutions.

--

Simon Lees (Simotek) http://simotek.net

Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B

— Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Mon, Nov 18, 2019 at 9:07 PM William Brown wrote:

...

On 19 Nov 2019, at 12:27, Simon Lees wrote:

On 11/19/19 11:22 AM, William Brown wrote:

...

...

On 19 Nov 2019, at 11:19, Simon Lees wrote:

On 11/19/19 11:07 AM, William Brown wrote:

...

...

On 19 Nov 2019, at 11:04, Neal Gompa wrote:

On Mon, Nov 18, 2019 at 7:32 PM William Brown wrote: > > Okay, so when you do: > > Project -> Repositories -> Add from a distribution -> select fedora 30 > > That only adds fedora 30 standard: > > https://build.opensuse.org/repositories/Fedora:30 > > But not update. This is why all the f30 builds I have are failing. This seems like a bug or error to me. >

It is done this way to minimize automatic rebuilds by default, as new updated packages in the repositories will trigger rebuilds. If you want updates, just change "standard" to "update" and call it gravy. Most don't require this, but your Rust case probably will.

For what it's worth, openSUSE Leap targets are similarly set up.

This is really counter intuitive to anyone who is reasonably inexperienced at packaging (IE me), which is probably another barrier to getting people to do tasks like this. It could be improved both as a default (include updates because that's the expected behaviour) or via UI (offer updates as an option in the distribution menu, or make it easier to alter standard -> update in the respository list).

One reason that it is done like this by default is because core SLE / Leap packages shouldn't be break API/ABI for updates, so in the vast majority of cases building against standard is fine and saves huge amounts of resources on obs. You are only really hitting this issue because you are working with one of the few echosystems that as you say doesn't follow our traditional model.

Ecosystems are changing, and less and less are following the traditional model - I think we can't keep treating them as exceptions, because this will become the norm for many languages.

So how can we improve this? How can we make rust a first class language for our platforms so that we can keep including relevant and modern tools in our platforms?

This is not just a tooling problem but also a process problem, the Haskell team have managed to script there entire process, there scripts can automatically push updates too there 2500 packages. This then revealed a different set of issues where our manual package review team and legal team didn't have enough manpower to regularly deal with the extra 2500 requests and I think for now the end solution was to cut back and just have the most useful packages in factory and the rest in a devel repo.

So it has been shown that atleast with one platform once the core is there and working, handling the frequent updates of various packages can be largely automated at least for tumbleweed as long as they are capable of doing offline builds.

For SUSE there is definitely a discussion that should be had around whether we should promise binary compatibility for the whole stack but that's a discussion for SUSE's product managers etc.

Who needs to be contacted to begin this discussion and get actions happening - I'm in a situation where I will be making rust a requirement for 389-ds soon, so this needs to have solutions.

Rust in openSUSE is mostly community, driven within the Fedora Rust SIG (Yes, Fedora Rust runs the openSUSE Rust stack). The Rust compiler package does get semi-regular contributions from two SUSE employees, but the bulk of the Rust stuff is through the Fedora Rust SIG. -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 11/19/19 12:56 PM, Neal Gompa wrote:

On Mon, Nov 18, 2019 at 8:58 PM Simon Lees wrote:

...

On 11/19/19 11:22 AM, William Brown wrote:

...

...

On 19 Nov 2019, at 11:19, Simon Lees wrote:

On 11/19/19 11:07 AM, William Brown wrote:

...

...

On 19 Nov 2019, at 11:04, Neal Gompa wrote:

On Mon, Nov 18, 2019 at 7:32 PM William Brown wrote: > > Okay, so when you do: > > Project -> Repositories -> Add from a distribution -> select fedora 30 > > That only adds fedora 30 standard: > > https://build.opensuse.org/repositories/Fedora:30 > > But not update. This is why all the f30 builds I have are failing. This seems like a bug or error to me. >

It is done this way to minimize automatic rebuilds by default, as new updated packages in the repositories will trigger rebuilds. If you want updates, just change "standard" to "update" and call it gravy. Most don't require this, but your Rust case probably will.

For what it's worth, openSUSE Leap targets are similarly set up.

This is really counter intuitive to anyone who is reasonably inexperienced at packaging (IE me), which is probably another barrier to getting people to do tasks like this. It could be improved both as a default (include updates because that's the expected behaviour) or via UI (offer updates as an option in the distribution menu, or make it easier to alter standard -> update in the respository list).

One reason that it is done like this by default is because core SLE / Leap packages shouldn't be break API/ABI for updates, so in the vast majority of cases building against standard is fine and saves huge amounts of resources on obs. You are only really hitting this issue because you are working with one of the few echosystems that as you say doesn't follow our traditional model.

Ecosystems are changing, and less and less are following the traditional model - I think we can't keep treating them as exceptions, because this will become the norm for many languages.

So how can we improve this? How can we make rust a first class language for our platforms so that we can keep including relevant and modern tools in our platforms?

This is not just a tooling problem but also a process problem, the Haskell team have managed to script there entire process, there scripts can automatically push updates too there 2500 packages. This then revealed a different set of issues where our manual package review team and legal team didn't have enough manpower to regularly deal with the extra 2500 requests and I think for now the end solution was to cut back and just have the most useful packages in factory and the rest in a devel repo.

For Rust crates, we can easily have similar automation. We're still building up the set of crates, but because of the whole manpower problem with reviews, it's going to be a rough time.

...

So it has been shown that atleast with one platform once the core is there and working, handling the frequent updates of various packages can be largely automated at least for tumbleweed as long as they are capable of doing offline builds.

For SUSE there is definitely a discussion that should be had around whether we should promise binary compatibility for the whole stack but that's a discussion for SUSE's product managers etc.

The bottleneck is almost entirely in the review process. We can do large updates easily enough. The review process does not handle large stack updates well.

Generally Legal reviews should only need to happen the first time which might make for a reasonable initial delay, after that it should just be review team reviews. If SUSE starts to require significant parts of the Rust stack and this starts to overwhelm the current review team it would be reasonable to expect SUSE to add additional resources to the review team so in the long term this isn't something that isn't unsolveable. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B

On Tuesday, November 19, 2019 12:58:35 AM CET William Brown wrote: That is funny, because I set myself to help (inside my modest skills) on both projects (Rust and Kanidm) before this issue arise. As usual I am too late. Anyways.

For opensuse I can probably do something like adding devel:languages:rust as a respository to my project. Would this strategy be viable on SLE? What about for projects which we want to expose packages for fedora or other? Are we able to have toolchains move faster than our base system?

I have a naive TODO: 1) Add in _service, script and / or document how to update rustc, and help the rustc maintaner in the update task more pro-actively 2) Add _service, script and document the upgrade of the crates, helping the work that the Fedora guys are doing in openSUSE. I think that is time for openSUSE to help here. 2.5) Research how the Haskel team is doing their magic, as IMHO is a good example to follow. 3) Using some data-driven mechanism, decide the crates / version will live in TW and the ones that will live only in the devel project. I am not sure how to tackle the multiversioning here. I hope to automatize all the steps, and do the submit to TW incrementally, as I am worried about the legal review. The compiler (and crates) in Leap and SLE cannot follow this peace, but we can help upstream to crystallize the idea of 'reference / stable compiler version'. Maybe we can skip 3 / 4 versions form TW to one in Leap. For SLE we can use the backports. The managers can interlock later the version for SLE, based on the requirements on the software that they are shipping. For Kanidm (and any project that uses Rust) this means that the devel version can compile on TW every day, but for Leap and SLE the version of the project and the compiler cannot change, and the fixes needs to be backported. This is aligned with how every product works: release and backport the fixes. This can get more complicated with the crates. One usual solution in OBS is to create a subproject for an specific release of the project (lets say, for example, Kanidm 1.0), and link the old version of the crate from devel:languages:rust:crates to the new project.

I'm really looking for ideas on how we can sustainably use rust projects within the governance and social constraints that exist in the space. Ideas?

Neal Gompa wrote:

That said, Fedora and Mageia update the Rust compiler within a week of a new stable compiler release. As of right now, Fedora 29 and higher all have Rust 1.39.0. CentOS/RHEL 7 will receive Rust 1.39.0 soon too[1].

The devel project was updated almost the same day in OBS. We (openSUSE) submitted the new version, created the upstream bug report about the issue that makes impossible the bootstrap of the compiler, and backported it to our packages. I am not going to check Fedora nor Mageia, but if they need to compile 1.39 with 1.38 (and later with 1.39) they need to use this patch. openSUSE do not have it yet on TW because we recompile all the dependencies, and this 1.39 is having issues with FF[1]. I am not sure if other distributions ship the FF compiled with 1.3X and the rustc from 1.39 independently. We cannot do that: we need to fix the issues as we find them. Simon Lees wrote:

One reason that it is done like this by default is because core SLE / Leap packages shouldn't be break API/ABI for updates, so in the vast majority of cases building against standard is fine and saves huge amounts of resources on obs.

That is true. But as we cannot have ABI compatibility with rust (not stable ABI and almost all is statically linked), I wonder if this will help to speed the update of the compiler in SLE and Leap. Neal Gompa wrote:

Rust in openSUSE is mostly community, driven within the Fedora Rust SIG (Yes, Fedora Rust runs the openSUSE Rust stack). The Rust compiler package does get semi-regular contributions from two SUSE employees, but the bulk of the Rust stuff is through the Fedora Rust SIG.

: ? is Luke Jones from Fedora? But in any case this is 100% true in the crates side. This is almost all done by the excellent Fedora Rust SIG. I really think that it is time for openSUSE to help here and with the upstream scripts from Fedora that is doing this magic. I want to do my part here. [1] https://build.opensuse.org/project/monitor/openSUSE:Factory:Staging:M? arch_x86_64=1&defaults=0&failed=1&repo_standard=1 -- SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nuremberg Germany (HRB 36809, AG Nürnberg) Managing Director: Felix Imendörffer

Hi! On 11/19/19 12:58 AM, William Brown wrote:

Rust as a language has a very aggressive release cycle compared to what we expect on a platform like SUSE - 6 weeks (https://github.com/rust-lang/rust/blob/master/RELEASES.md). In each cycle Rust releases new features, and despite the core language being "stable" and safe (which is great!) new features tend to be used very rapidly by library authors. For example the release from Version 1.34.0 (2019-04-11) contains convert::TryFrom, which is now in use by a large number of libraries.

Is that really that common? I have had a lot of interaction with Rust upstream through Debian and I was always told it's normally only the Rust compiler itself which uses new features in its own code which is why it always needs to be built with either its own version or the previous version, older or newer versions usually break. It sometimes happens that 3rd party Rust software projects are affected by it like you say, but I have observed that only occasionally unless the Rust compiler is really old. But maybe the situation is worse when using SLE due its long-term character. However, generally I think that Rust should slow down with frequent incompatible changes to the language itself as this prevents independent compiler implementations to keep up. mrustc is currently compatible with Rust 1.29.0 and from GCC upstream I have heard that the frequent changes are the main blocker for adopting a Rust frontend in GCC. Adrian N�����r��y隊Z)z{.��ZrF��x>�{.n�+������Ǩ��r��i�m��0��ޙ���������$j���0�����Ǩ�

On Tuesday 2019-11-19 13:02, Neal Gompa wrote:

The Rust compiler developers are smart guys, but they don't understand that bootstrapping Rust is not supposed to be so hard.

So then, what does smart mean actually? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

William Brown schrieb:

[...] Due to the fast release cycle, developers aggressively using new features, and the npm style micro dependency system we have a recipe for problems - if you are not using the latest stable compiler, it's extremely likely that your libraries, or their dependents may not build on your project. Which is exactly the issue I have run into where rust on fedora 30 and opensuse leap 15.1 are simply too old to support the features that have been used in the last 6 months by library developers.

From what I can tell Rust has two use cases in the stable distros. The first one is that it's needed by other system components. Regularly updated, more or less leaf packages Firefox as but unfortunately also as low as librsvg. For packages like the latter we most likely do not want Rust to change very six weeks, especially when that kind of use case spreads. So for that it would be better to upgrade the system Rust only for service packs ie once per year. The other use case is developers that want the latest and greatest for their own purposes. For that it should be possible to create a parallel installable Rust package and regularly provide newer versions for it. That works for Leap via regular maintenance updates as well as SLE via Package Hub. I've been told that Rust is meant to be pretty good at backwards compatibility so it's probably sufficient to provide one additional Rust package that follows upstream releases, instead of a new one for all kinds of versions like eg Go does. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer HRB 247165 (AG München) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 19-11-19 16 h 44, Ludwig Nussel wrote:

William Brown schrieb:

...

[...] Due to the fast release cycle, developers aggressively using new features, and the npm style micro dependency system we have a recipe for problems - if you are not using the latest stable compiler, it's extremely likely that your libraries, or their dependents may not build on your project. Which is exactly the issue I have run into where rust on fedora 30 and opensuse leap 15.1 are simply too old to support the features that have been used in the last 6 months by library developers.

From what I can tell Rust has two use cases in the stable distros. The first one is that it's needed by other system components. Regularly updated, more or less leaf packages Firefox as but unfortunately also as low as librsvg. For packages like the latter we most likely do not want Rust to change very six weeks, especially when that kind of use case spreads. So for that it would be better to upgrade the system Rust only for service packs ie once per year.

The other use case is developers that want the latest and greatest for their own purposes. For that it should be possible to create a parallel installable Rust package and regularly provide newer versions for it. That works for Leap via regular maintenance updates as well as SLE via Package Hub. I've been told that Rust is meant to be pretty good at backwards compatibility so it's probably sufficient to provide one additional Rust package that follows upstream releases, instead of a new one for all kinds of versions like eg Go does.

cu Ludwig

rustup[0] is what developers should use when developing in Rust. It's the official Rust toolchain installer. I packaged it in my home[1] and submitted a request[2] in hopes that it would get into Factory, but in the end, I didn't get much feedback beside from Neal Gompa. I decided to revoke the request. As you can see in the request, I talked about rust and rustup coexisting, but this would need a bit more work. I didn't look into this, but I'm willing to do it. It definitely seems to be possible as documented[3] upstream. [0]: https://github.com/rust-lang/rustup [1]: https://build.opensuse.org/package/show/home:dmarcoux/rustup.rs [2]: https://build.opensuse.org/request/show/684463 [3]: https://github.com/rust-lang/rustup#working-with-distribution-rust-packages -- Dany Marcoux Full Stack Web Developer - Open Build Service SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 2019-11-19 17:34, Dany Marcoux wrote:

rustup[0] is what developers should use when developing in Rust. It's the official Rust toolchain installer.

I guess that your idea is to have a system rust compiler installed via a package, and a user rust compiler installed via rustup? I see the appealing of this.

I packaged it in my home[1] and submitted a request[2] in hopes that it would get into Factory, but in the end, I didn't get much feedback beside from Neal Gompa. I decided to revoke the request. As you can see in the request, I talked about rust and rustup coexisting, but this would need a bit more work.

I think that this is critical. PIP is available in openSUSE as a mean to install Python packages, but executing it as root will install a system wide Python package and will break your system. There is some parallelism with rustup here, but as PIP is distributed in openSUSE, rustup can also be.

I didn't look into this, but I'm willing to do it. It definitely seems to be possible as documented[3] upstream.

-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 20 Nov 2019, at 05:57, John Paul Adrian Glaubitz wrote:

On 11/19/19 7:15 PM, Jan Engelhardt wrote:

...

An alternative would be to devote the manned resources to not grow minute dependencies on large stacks. (That ship has probably sailed in terms of firefox-in-the-distro, (...)

Unless Firefox has changed since the last time I touched the codebase, I'm pretty sure that it has all the Rust crates vendored which it needs. So, Firefox alone shouldn't be a problem.

I think Rust projects will always tend to vendoring, because the huge micro-dependency style system that exists means we'll never keep up trying to package these. Kanidm has ~200 crate indirect dependencies on it's own, I can't imagine how many firefox has. It's a waste of our time to try to package all 200 when we could be putting that time into other things instead. This essentially means we are always at the whims of those vendored dependencies and what they require, and that is highly likely to be the "latest" stable compiler. I think we should aim to: * Have rustc/cargo move with the upstream cadence, with automation as suggested into all SLE/LEAP/TW/Other * Do not attempt to package crates - only package leaves IE consuming projects * Improve our rust macros and packaging guides around this topic. As much as we want to wring our hands and slow down rust, and do things "our way" we can't - we have to follow and adapt in this case. So where do we start to start to achieve this?

Adrian ��N�งฒๆ์rธ�y้��Z)z{.ฑ๊ZrF �x>บ{.nว+�ทจฅ้์บวจฎ#<^_NSEDR_^#<่r#<^_NSEDR_^#<ํiหm#<^_NSEDR_^#<๊0#<^_NSEDR_^#<๊��จฅข�งฒ๋�ฅง$jง#<^_NSEDR_^#<๊0#<^_NSEDR_^#<๊่ฅ้์บวจ

— Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Wed, 20 Nov 2019, William Brown wrote:

...

On 20 Nov 2019, at 05:57, John Paul Adrian Glaubitz wrote:

On 11/19/19 7:15 PM, Jan Engelhardt wrote:

...

An alternative would be to devote the manned resources to not grow minute dependencies on large stacks. (That ship has probably sailed in terms of firefox-in-the-distro, (...)

Unless Firefox has changed since the last time I touched the codebase, I'm pretty sure that it has all the Rust crates vendored which it needs. So, Firefox alone shouldn't be a problem.

I think Rust projects will always tend to vendoring, because the huge micro-dependency style system that exists means we'll never keep up trying to package these. Kanidm has ~200 crate indirect dependencies on it's own, I can't imagine how many firefox has. It's a waste of our time to try to package all 200 when we could be putting that time into other things instead. This essentially means we are always at the whims of those vendored dependencies and what they require, and that is highly likely to be the "latest" stable compiler.

I think we should aim to:

* Have rustc/cargo move with the upstream cadence, with automation as suggested into all SLE/LEAP/TW/Other * Do not attempt to package crates - only package leaves IE consuming projects * Improve our rust macros and packaging guides around this topic.

As much as we want to wring our hands and slow down rust, and do things "our way" we can't - we have to follow and adapt in this case.

I wonder whether for packages requiring rust it would be most sensible to package up all dependences into the package itself - that at least ensures a "stable" set of dependences for individual packages even if it means that a dependence might exist twice in different versions in different packages. Not sure if that also means packaging up their own rust as some package up their own clang/llvm ... Richard. -- Richard Biener SUSE Software Solutions Germany GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany; GF: Felix Imendörffer; HRB 36809 (AG Nuernberg)

On 19-11-20 00 h 55, William Brown wrote:

...

On 20 Nov 2019, at 03:04, Dany Marcoux wrote:

On 19-11-19 16 h 44, Ludwig Nussel wrote:

...

William Brown schrieb:

...

[...] Due to the fast release cycle, developers aggressively using new features, and the npm style micro dependency system we have a recipe for problems - if you are not using the latest stable compiler, it's extremely likely that your libraries, or their dependents may not build on your project. Which is exactly the issue I have run into where rust on fedora 30 and opensuse leap 15.1 are simply too old to support the features that have been used in the last 6 months by library developers. From what I can tell Rust has two use cases in the stable distros. The first one is that it's needed by other system components. Regularly updated, more or less leaf packages Firefox as but unfortunately also as low as librsvg. For packages like the latter we most likely do not want Rust to change very six weeks, especially when that kind of use case spreads. So for that it would be better to upgrade the system Rust only for service packs ie once per year.

The other use case is developers that want the latest and greatest for their own purposes. For that it should be possible to create a parallel installable Rust package and regularly provide newer versions for it. That works for Leap via regular maintenance updates as well as SLE via Package Hub. I've been told that Rust is meant to be pretty good at backwards compatibility so it's probably sufficient to provide one additional Rust package that follows upstream releases, instead of a new one for all kinds of versions like eg Go does.

cu Ludwig

rustup[0] is what developers should use when developing in Rust. It's the

official Rust toolchain installer. I packaged it in my home[1] and submitted

a request[2] in hopes that it would get into Factory, but in the end, I didn't

get much feedback beside from Neal Gompa. I decided to revoke the request. As

you can see in the request, I talked about rust and rustup coexisting, but this

would need a bit more work. I didn't look into this, but I'm willing to do it.

It definitely seems to be possible as documented[3] upstream. This does not address the issue - how do we get from development to a system-buildable and shippable project. This is key here, we need to think about that full experience from development to deployment - not just development.

Right now, we have nothing for the development part, unless we want to stick to whatever rust version openSUSE provides (which is a bad developer experience). This is what my proposal would address.

...

[0]: https://github.com/rust-lang/rustup

[1]: https://build.opensuse.org/package/show/home:dmarcoux/rustup.rs

[2]: https://build.opensuse.org/request/show/684463

[3]: https://github.com/rust-lang/rustup#working-with-distribution-rust-packages

-- Dany Marcoux Full Stack Web Developer - Open Build Service

SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer

-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

— Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server SUSE Labs

-- Dany Marcoux Full Stack Web Developer - Open Build Service SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 19-11-20 09 h 06, William Brown wrote:

...

On 20 Nov 2019, at 18:35, Dany Marcoux wrote:

On 19-11-20 00 h 55, William Brown wrote:

...

...

On 20 Nov 2019, at 03:04, Dany Marcoux wrote:

On 19-11-19 16 h 44, Ludwig Nussel wrote:

...

William Brown schrieb:

...

[...] Due to the fast release cycle, developers aggressively using new features, and the npm style micro dependency system we have a recipe for problems - if you are not using the latest stable compiler, it's extremely likely that your libraries, or their dependents may not build on your project. Which is exactly the issue I have run into where rust on fedora 30 and opensuse leap 15.1 are simply too old to support the features that have been used in the last 6 months by library developers. From what I can tell Rust has two use cases in the stable distros. The first one is that it's needed by other system components. Regularly updated, more or less leaf packages Firefox as but unfortunately also as low as librsvg. For packages like the latter we most likely do not want Rust to change very six weeks, especially when that kind of use case spreads. So for that it would be better to upgrade the system Rust only for service packs ie once per year.

The other use case is developers that want the latest and greatest for their own purposes. For that it should be possible to create a parallel installable Rust package and regularly provide newer versions for it. That works for Leap via regular maintenance updates as well as SLE via Package Hub. I've been told that Rust is meant to be pretty good at backwards compatibility so it's probably sufficient to provide one additional Rust package that follows upstream releases, instead of a new one for all kinds of versions like eg Go does.

cu Ludwig

rustup[0] is what developers should use when developing in Rust. It's the

official Rust toolchain installer. I packaged it in my home[1] and submitted

a request[2] in hopes that it would get into Factory, but in the end, I didn't

get much feedback beside from Neal Gompa. I decided to revoke the request. As

you can see in the request, I talked about rust and rustup coexisting, but this

would need a bit more work. I didn't look into this, but I'm willing to do it.

It definitely seems to be possible as documented[3] upstream. This does not address the issue - how do we get from development to a system-buildable and shippable project. This is key here, we need to think about that full experience from development to deployment - not just development.

Right now, we have nothing for the development part, unless we want to stick to

whatever rust version openSUSE provides (which is a bad developer experience).

This is what my proposal would address. We can not seperate deverloper from the deployment - else we fragment the experience and end up back here. No, the solution for how we ship and develop must be the same.

In the end, rustup still remains the official way to develop in Rust. Not providing rustup is counter-intuitive. I understand your point, but I'm more interested in the developer experience. Anyway, I see that this is going nowhere in regards to rustup so I'm out. I'll keep following what is discussed here. Have a good day.

...

...

...

[0]: https://github.com/rust-lang/rustup

[1]: https://build.opensuse.org/package/show/home:dmarcoux/rustup.rs

[2]: https://build.opensuse.org/request/show/684463

[3]: https://github.com/rust-lang/rustup#working-with-distribution-rust-packages

-- Dany Marcoux Full Stack Web Developer - Open Build Service

SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer

-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org — Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server SUSE Labs

-- Dany Marcoux Full Stack Web Developer - Open Build Service

SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer

— Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server SUSE Labs

-- Dany Marcoux Full Stack Web Developer - Open Build Service SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 11/20/19 6:35 PM, Dany Marcoux wrote:

On 19-11-20 00 h 55, William Brown wrote:

...

...

On 20 Nov 2019, at 03:04, Dany Marcoux wrote:

On 19-11-19 16 h 44, Ludwig Nussel wrote:

...

William Brown schrieb:

...

[...] Due to the fast release cycle, developers aggressively using new features, and the npm style micro dependency system we have a recipe for problems - if you are not using the latest stable compiler, it's extremely likely that your libraries, or their dependents may not build on your project. Which is exactly the issue I have run into where rust on fedora 30 and opensuse leap 15.1 are simply too old to support the features that have been used in the last 6 months by library developers.  From what I can tell Rust has two use cases in the stable distros. The first one is that it's needed by other system components. Regularly updated, more or less leaf packages Firefox as but unfortunately also as low as librsvg. For packages like the latter we most likely do not want Rust to change very six weeks, especially when that kind of use case spreads. So for that it would be better to upgrade the system Rust only for service packs ie once per year.

The other use case is developers that want the latest and greatest for their own purposes. For that it should be possible to create a parallel installable Rust package and regularly provide newer versions for it. That works for Leap via regular maintenance updates as well as SLE via Package Hub. I've been told that Rust is meant to be pretty good at backwards compatibility so it's probably sufficient to provide one additional Rust package that follows upstream releases, instead of a new one for all kinds of versions like eg Go does.

cu Ludwig

rustup[0] is what developers should use when developing in Rust. It's the

official Rust toolchain installer. I packaged it in my home[1] and submitted

a request[2] in hopes that it would get into Factory, but in the end, I didn't

get much feedback beside from Neal Gompa. I decided to revoke the request. As

you can see in the request, I talked about rust and rustup coexisting, but this

would need a bit more work. I didn't look into this, but I'm willing to do it.

It definitely seems to be possible as documented[3] upstream. This does not address the issue - how do we get from development to a system-buildable and shippable project. This is key here, we need to think about that full experience from development to deployment - not just development.

Right now, we have nothing for the development part, unless we want to stick to

whatever rust version openSUSE provides (which is a bad developer experience).

This is what my proposal would address.

To a large extent if you can get the developer experience right and constantly enough updating in tumbleweed it can then be provided in places like SLE via packagehub without a huge amount of extra effort. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B

...

...

This essentially means we are always at the whims of those vendored dependencies and what they require, and that is highly likely to be the "latest" stable compiler.

I think we should aim to:

* Have rustc/cargo move with the upstream cadence, with automation as suggested into all SLE/LEAP/TW/Other

+1, I think that this is a good starting point.

Neal, what do you think of the idea from Ludwig of having a system rust compiler (more stable and conservative, used to build all the openSUSE rings), and a updated version of the compiler co-installable for developers.

It's not a _terrible_ idea. The only problem with it is that Firefox forces Rust to move forward anyway. That's why RHEL doesn't do this right now, either.

This isn't viable - the moment I, a developer, take my project from development -> packaging, I'm now on an older compiler that my dependencies won't compile with. They have to be the same version, and they have to both move at the rate upstream rust is moving as many dependencies really do move at that cadence too. This is even more important when you remember that rust doesn't distinguish security releases from feature ones, so an update to a dependency for security, may require newer compiler versions.

...

...

As much as we want to wring our hands and slow down rust, and do things "our way" we can't - we have to follow and adapt in this case.

So where do we start to start to achieve this?

You are making very hard questions : ))

Yes - because I want kanidm in opensuse at some point, and 389-ds is about to pick up rust as a requirement which will absolutely affect future leap versions. I want (need?) this solved. :)

...

I am personally in a phase that I try to have a view of what we want, and what are the good decisions. Fedora and Debian communities are spending a lot of thinking here, so before moving I want to have more information.

The assumption here is that Rust needs to be slowed down. That's not the case at all. You can move fast _without_ breaking things. The problem is that today there are no Rust compiler and language developers who understand this and help factor that into the future development of the stack.

I think that again, Rust has different targets (containers, static applications, continual integration and release) than an OS packaging system (shared libs, C, patched backports). It's not that they don't understand as much as they actively chose to follow a different path. Which is why we are struggling. The mental model of how we did things with C based programs doesn't apply. For years Red Hat / Fedora have tried to for python to be packaged like it's C, when it's not (which has left pip/pypi in bad places) and causes python on a system to be missing many libraries that do exist. This is why python as a community exclusively recommends virtualenv, not OS/system python. We'll probably end up in the exact same place with rust (cargo vs zypper in crate-foo). So we can either keep fighting against it, and people will just use cargo + vendoring anyway, or we can accept that the ship has sailed and try to work with it instead. For example: * https://build.opensuse.org/package/show/openSUSE:Leap:15.0/cargo * https://build.opensuse.org/package/show/home:gladiac/rav1e * https://build.opensuse.org/package/show/home:luke_nukem:rust_apps/nushell * https://build.opensuse.org/package/show/openSUSE:Leap:15.2/ripgrep

from M Matz:

...

That's a downside to QA process, not the packaging issue anymore. As far as I know we are discussing packaging right now, aren't we?

William (rightly) wants to have something realistic for the distro. Packaging (as in merely creating and submitting packages easily) is only one part, it can't be seen in isolation. If you can submit a thousand packages in 10 minutes doesn't really matter, if those thousand packages then don't land in Factory, or only six months after submission. The whole thing needs to work on all levels.

So the more I'm reading into this thread the more I think that my conclusion here is: * We should have rust/cargo's release cadence seperate from the release cycles of a distro (ie fedora modularity in a way, or tw style) * Packaging crates to rpms is not feasible or scalable, and we should not attempt it - cargo already has vendor and all the packages I've seen in obs today use vendored dependencies. * We should instead focus on packaging "edge" programs that use rust IE firefox, ripgrep. * Focus pkg QA efforts on the edge packages * We could consider integration of tools like cargo-audit to scan for security issues in packages (similar to clamav in obs already) to help ensure vendored deps are "up to date". * Improve the rpm spec docs related to rust/cargo to help make it easier for people to give us packaged programs. — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On 21 Nov 2019, at 09:45, John Paul Adrian Glaubitz wrote:

Hi!

On 11/20/19 11:26 PM, William Brown wrote:

...

...

The assumption here is that Rust needs to be slowed down. That's not the case at all. You can move fast _without_ breaking things. The problem is that today there are no Rust compiler and language developers who understand this and help factor that into the future development of the stack.

I think that again, Rust has different targets (containers, static applications, continual integration and release) than an OS packaging system (shared libs, C, patched backports). It's not that they don't understand as much as they actively chose to follow a different path.

But packages like 389-ds and librsvg are anything but typical container packages which is the main problem here. People don't necessarily criticize that Rust is fast moving target at the moment. They criticize that people start using Rust for system-critical components while Rust is a moving target.

Rust won't slow down though, and has no intent too. I think we have to accept it's fast moving.

...

Which is why we are struggling. The mental model of how we did things with C based programs doesn't apply.

Using stable APIs and not updating hundreds of packages just because you need to install a newer version of Firefox or some security updates isn't the mental model of C. It's the mental model of sanity.

We simply can't simply use the Rust release model for enterprise distributions because no matter what Rust upstream will claim: New upstream versions will always bring regressions, in one way or another. Changing command line options or configuration file formats is already a regression in production use. No paying customer really wants to see an update breaking their whole server setup. It's simply not acceptable. Imagine SLE running in a critical system at a bank on an IBM zSeries mainframe responsible for millions of customers and then you start upgrade 250+ packages for a security fix and break the whole database.

We can complain all we like here, but firefox is using rust, and other packages will use it too. We have to be able to adapt here ...

...

For years Red Hat / Fedora have tried to for python to be packaged like it's C, when it's not (which has left pip/pypi in bad places) and causes python on a system to be missing many libraries that do exist.

Again, this isn't a language issue. This is upstream projects not understanding how software is used in production environments. If 90% of the developers of an upstream projects are running distributions like Fedora Rawhide, Arch Linux or Tumbleweed, they naturally don't understand why you wouldn't want all packages to be bleeding edge all the time. But this isn't something you can use in a production environment with hundreds, thousands or even millions of customers.

...

This is why python as a community exclusively recommends virtualenv, not OS/system python.

So, how exactly do you expect SUSE or RedHat being able to support a paying customer when their whole Python environment has been installed through pip?

They don't offer them support in these cases. This in mind SUSE/RH do support 'whole applications' like openshift, 389-ds or jboss, and "how those applications are composed" doesn't really factor into it, and they support them still.

Our engineers simply can't support a software environment that we have no control over. The main reason why software is being packaged in distribution besides convenience is being able to provide something that can be supported by the distribution vendor. If we just give our customers a minimal base system and just let them install all their software through cargo, pip, npm and what not, we won't be able to provide support for their environment. We might as well drop the concept of a distribution then. But that's when enterprise customers will return to proprietary systems like Windows Server or AIX.

...

We'll probably end up in the exact same place with rust (cargo vs zypper in crate-foo). So we can either keep fighting against it, and people will just use cargo + vendoring anyway, or we can accept that the ship has sailed and try to work with it instead.

As long as we want to provide a distribution that we can provide support for, we will have to stick to the "old model" and that ship isn't going to sail anywhere.

And that's fine, but we have to be able to support more than just that old model else we will be left behind.

...

So the more I'm reading into this thread the more I think that my conclusion here is:

* We should have rust/cargo's release cadence seperate from the release cycles of a distro (ie fedora modularity in a way, or tw style) * Packaging crates to rpms is not feasible or scalable, and we should not attempt it - cargo already has vendor and all the packages I've seen in obs today use vendored dependencies. * We should instead focus on packaging "edge" programs that use rust IE firefox, ripgrep. * Focus pkg QA efforts on the edge packages * We could consider integration of tools like cargo-audit to scan for security issues in packages (similar to clamav in obs already) to help ensure vendored deps are "up to date". * Improve the rpm spec docs related to rust/cargo to help make it easier for people to give us packaged programs.

It could also happen that certain software projects like librsvg or 389-ds are being forked in their last non-Rust version to avoid the bootstrap issues. It's not that projects haven't been forked in the past over such fundamental project philosophy issues (XFree86/X.Org, libav/ffmpeg, OpenOffice/LibreOffice, eglibc/glibc, ecgs/gcc etc).

I don't think the hammer method that some projects are currently using to push Rust is going to be successful in the long-term.

As the person who is adding rust to 389-ds, the maintainer of it at suse, and part of the upstream team (the rh people are also on board with this), I don't think I'll be forking "the last non-rust version" as that would be a bit self defeating ... We need productive solutions, not complaining that rust is different to "how it's always been". — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Alright, I'm going to come out and say it. You need to pull that stick out of your rear.

This comment is extremely inappropriate regardless of the situation, and it has no place on a mailing list or in any communication to any persons. I've made my argument already in the thread and I think we are just going in a circle at the moment. I'm going to step back and think about this topic for a while before I say more. — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Wednesday, November 20, 2019 11:26:41 PM CET William Brown wrote:

...

...

Neal, what do you think of the idea from Ludwig of having a system rust compiler (more stable and conservative, used to build all the openSUSE rings), and a updated version of the compiler co-installable for developers.> It's not a _terrible_ idea. The only problem with it is that Firefox forces Rust to move forward anyway. That's why RHEL doesn't do this right now, either.

This isn't viable - the moment I, a developer, take my project from development -> packaging, I'm now on an older compiler that my dependencies won't compile with.

I understand. This is the reason of having two version of the compiler in Leap / SLE and one single version in TW. One version will be the system compiler, associated to the service pack release, or in the case of Leap, to the subversion number. This is the compiler used to build the distribution and will be updated in each SPx, based on some criteria that balance change and stability. Because Rust is very strong in the backward compatibility, I think that we can expect that in each SP / Leap release, the system compiler will be updated to the last version. For developers, there will be a more update version available officially, that is moving at the same peace that upstream. This is the only one that developers will use, and a `zypper up` will take care of updating it in the distribution. But we cannot expect that the system compiler will be updated at the same speed that upstream, because this requires that every 6 weeks, a routinary `zypper up` will update the compiler, but also the user applications that are using rust. This goes against the stability guarantees and reproducibility that SLE and Leap offer to the users. Imagine that I use the new compiler in my SLE to get my Firefox, and I get a different binary that the one distributed with the system.

They have to be the same version, and they have to both move at the rate upstream rust is moving as many dependencies really do move at that cadence too. This is even more important when you remember that rust doesn't distinguish security releases from feature ones, so an update to a dependency for security, may require newer compiler versions.

Yes, exactly. You will be using the compiler for developers.

...

The assumption here is that Rust needs to be slowed down. That's not the case at all. You can move fast _without_ breaking things. The problem is that today there are no Rust compiler and language developers who understand this and help factor that into the future development of the stack.

I think that again, Rust has different targets (containers, static applications, continual integration and release) than an OS packaging system (shared libs, C, patched backports). It's not that they don't understand as much as they actively chose to follow a different path.

Which is why we are struggling. The mental model of how we did things with C based programs doesn't apply. For years Red Hat / Fedora have tried to for python to be packaged like it's C, when it's not (which has left pip/pypi in bad places) and causes python on a system to be missing many libraries that do exist. This is why python as a community exclusively recommends virtualenv, not OS/system python. We'll probably end up in the exact same place with rust (cargo vs zypper in crate-foo). So we can either keep fighting against it, and people will just use cargo + vendoring anyway, or we can accept that the ship has sailed and try to work with it instead.

No, there is a difference. In Python you need to install the library before it can be used, as is an interpreted language, and in C you need to install the library as is dynamically linked to the application. The proposal of packaging the crates is not about using `zypper in rust- regex`, is about using `BuildRequires: rust-regex` during the creation of `kanidm`. No crate will be usually installed in the system, as the RPM only contains the source code. A change in `rust-regex` in OBS will trigger a rebuild of `kanidm`, as will be any update of any crate that belongs to the transitive closure of the dependencies of your package. This avoid vendoring, and avoid the security implications related with it. But during the daily development of kanidm, you are free to use cargo to install the crates in the user space. During the packaging phase of kanidm, the last version of the crate needs to be available in OBS before it can be compiled, tho. So multiversioning is something that we need to address in OBS, with the same expectations that cargo have.

...

from M Matz:

...

That's a downside to QA process, not the packaging issue anymore. As far as I know we are discussing packaging right now, aren't we?

William (rightly) wants to have something realistic for the distro. Packaging (as in merely creating and submitting packages easily) is only one part, it can't be seen in isolation. If you can submit a thousand packages in 10 minutes doesn't really matter, if those thousand packages then don't land in Factory, or only six months after submission. The whole thing needs to work on all levels.

So the more I'm reading into this thread the more I think that my conclusion here is:

* We should have rust/cargo's release cadence seperate from the release cycles of a distro (ie fedora modularity in a way, or tw style)

IMHO we do not mix the real problem of the review process, with the pace of upgrade. If we need to upgrade a lot of packages, and we have the tools to do that, lets fix the review process. I am sure that this way we help more. I am in favor of moving fast, but again, there are reasons in SLE / Leap (reproducibility build, stability guarantees) that are not related with the review process problem.

* Packaging crates to rpms is not feasible or scalable, and we should not attempt it - cargo already has vendor and all the packages I've seen in obs today use vendored dependencies.

Packaging crates is is feasible. Is not scalable because the review process, not because we cannot update all the packages every day. and use it the in the spec file as a BuildRequires as is expected (and as is implemented right now on Fedora) We can use vendoring now, until we fix those problems, but we need to aim to the better solution.

* We should instead focus on packaging "edge" programs that use rust IE firefox, ripgrep.

For TW, sure.

* Focus pkg QA efforts on the edge packages

* We could consider integration of tools like cargo-audit to scan for security issues in packages (similar to clamav in obs already) to help ensure vendored deps are "up to date".

Something like this is a good idea, like the feature that github provides, that is pointing to you that have a requirement on a library that have doing only this integration is not fixing the vendoring issue.

* Improve the rpm spec docs related to rust/cargo to help make it easier for people to give us packaged programs.

SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nuremberg (HRB 36809, AG Nürnberg) Managing Director: Felix Imendörffer

On 22 Nov 2019, at 03:04, Alberto Planas Dominguez wrote:

On Thursday, November 21, 2019 8:40:19 AM CET Richard Biener wrote:

[cut a lot here]

...

From the start of the thread I remember there were two conflicting things - first packaging where vendoring seems to work and while it's not ideal it's probably going to be what we need to do due to being "slow". I interpret your above sentences as that this is intended to work this way.

Then there's the need of a developer using Rust (but not necessarily packaging software for a distro) who wants the latest and greatest. I've suggested we provide means to easily bootstrap such upstream Rust - and you've indicated that yes, indeed, thats supported and even the way it is supposed to work, but there's no openSUSE "package" to start this (in Factor).

What is wrong on the idea of having two compilers: the system one used for the distribution, and the one that can be updated and expected to be used by developers.

Another idea is to have one single compiler, the system-wide one, and provide a rustup binary that the developer can use to install rust and cargo on the users $HOME. Dany Marcoux already submitted the package, but later was closed.

I personally prefer the 2 compiler idea, tho, but this is irrelevant.

The 2 compiler idea is kind of a problem? I have a version on my machine that I was developing with (1.39.0), and encountered version (1.36.0) on leap which didn't have a feature I required. What I'm thinking is important is how can we encourage developers to want to package their code for suse. Developers will use whatever is latest in crates.io (and sometimes wont have a choice due to indirect dependencies), and as I've mentioned, there is a strong trend for those dependencies to always rely on latest stable features. Having a 4 month old (or older) version will become a blocker to adding software (or rust security updates) to suse :(

[more cut]

...

Indeed. We're not going to make Rust "go away". Vendoring works fine enough for "leafs" like Firefox but indeed having the rust stack in core components imposes some scalability issues and people should think twice before introducing a rust dependency into, say, systemd. But obviously those who do the work decide, and as alwaks talk is cheap.

I am sorry but... vendoring is not working. OBS is not aware of what is inside, so any security update in any component in the closure can hurt us a lot.

I'd suggest we wont be able to stop people vendoring because the rate of flux in crates.io is so high, continually adding and packaging more crates is going to be a high resource cost on people. If I had to submit 200 rpm's to suse just to submit kanidm, I'd probably give up at the sheer thought of that effort on me and others. Security updates become a requirement of the projects submitting to ensure their vendored dependencies are up to date. I think that we can detect this with cargo audit as a build step in the spec instead, similar to how we use clamav. I guess the feeling I'm getting overall from this thread and peoples input is the idea of "support" and what that is, and how it works, and what goes into that. SUSE/SLE/Distros have always been about supporting "layers", such as a kernel or tls library. They are updated dynamically and the higher layers can inherit their changes seamlessly. This encourages backporting, patches etc. They also have C compilers that have *rapidly* changed in the last two decades, which while making code "faster", has also introduced (sometimes breaking) changes when you recompile the same code. This unpredictability makes us want to stick to a known compiler for a version so that all the layers are consistent. It means you have to have these layers work for multiple applications at the same time, via ABI's that are "stable". This promotes us keeping one version of a layer part that is supported for a long time. Rust, firefox, docker and others, have become about supporting "applications". The interface is no longer between layers, but the user-facing parts, and the application as a whole is a "static linked" whole stack that is supported as is. The composition of that static application is the responsibility of the project, who is expected to re-build and re-release often. Rust has no ABI or guarantees about it, shifting this to compile time rather than link time to ensure you consume the API correctly. A single project can have multiple versions of the same crate in it's dep stack (rand is a key example here, in large projects it's been observed to have 4 different api versions). This in mind, due to Rust's guarantees many of the memory safety issues that heavily affect C do not exist, meaning there are less stability and security releases in comparison, and that static stack is much more reliable as a whole, lending again to supporting the "application" as a whole unit rather than supporting the "layers" that make it up. I guess maybe we should think about Rust applications the same way we think about docker applications, they have much more in common as staticly linked units than say comparing Rust to C. — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Fri, 22 Nov 2019, William Brown wrote:

...

On 22 Nov 2019, at 03:04, Alberto Planas Dominguez wrote:

On Thursday, November 21, 2019 8:40:19 AM CET Richard Biener wrote:

[cut a lot here]

...

From the start of the thread I remember there were two conflicting things - first packaging where vendoring seems to work and while it's not ideal it's probably going to be what we need to do due to being "slow". I interpret your above sentences as that this is intended to work this way.

Then there's the need of a developer using Rust (but not necessarily packaging software for a distro) who wants the latest and greatest. I've suggested we provide means to easily bootstrap such upstream Rust - and you've indicated that yes, indeed, thats supported and even the way it is supposed to work, but there's no openSUSE "package" to start this (in Factor).

What is wrong on the idea of having two compilers: the system one used for the distribution, and the one that can be updated and expected to be used by developers.

Another idea is to have one single compiler, the system-wide one, and provide a rustup binary that the developer can use to install rust and cargo on the users $HOME. Dany Marcoux already submitted the package, but later was closed.

I personally prefer the 2 compiler idea, tho, but this is irrelevant.

The 2 compiler idea is kind of a problem? I have a version on my machine that I was developing with (1.39.0), and encountered version (1.36.0) on leap which didn't have a feature I required.

Of course this happens with C++ language features evolving as well. And we _do_ have multiple(!) GCC C++ compilers around, also specifically to address this problem. So I see no problem with having multiple rust compilers around. Richard. -- Richard Biener SUSE Software Solutions Germany GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany; GF: Felix Imendörffer; HRB 36809 (AG Nuernberg)

Am 26.11.19 um 01:18 schrieb William Brown:

It's also worth bearing in mind that rust does not have most of the issues which makes C developers so anxious or conservative around versioning and releases.

If you read https://semver.org/, you will find that it doesn't mention C or C++ at all. (It was written by Ruby developers.) Versioning is used for software written in almost any language you can imagine. I don't understand how the language differences between C or C++ and Rust should affect versioning and releases. Some users want stable releases with minimal backports, because they don't want any changes that could disrupt their operation. Whether a language has less or no undefined behavior is almost completely irrelevant there. Sure, you are protecting against some low-level errors, but no language (unless it requires you to formally prove your algorithms) will stop you from introducing high-level errors or just changing behavior in a way that some users might not want. Even in such a language, you would probably want to version the specification. Stability does not just mean "no new bugs", it's more about "no changes in behavior". That's why we give users the choice between Tumbleweed and Leap. It's not necessarily that Tumbleweed is horribly buggy or has regressions all the time, but there are configuration changes, interface changes, removed packages, and so on. As a Tumbleweed user, I have to occasionally fix something in the system, even without any bugs.

As mentioned, rust is closer as an experience to docker than C.

Isolated parts ("leaf" packages) aren't what anybody is worrying about, it's parts written in Rust on which other parts of the system depend. Although actually leaf packages don't really exist: there can always be dependencies outside of the distribution. As an example, users might have scripts on their system that stop working with newer versions of some software. Long story short: keeping stable interfaces is about much more than the ABI compatibility a C or C++ developer might be worried about. That's why versioning is used across languages in pretty much the same way. Best regards, Aaron -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org