Comment # 18 on bug 1192067 from William Brown

(In reply to Hans-Peter Jansen from comment #16)
> Thanks guys, this is one of the most helpful bug records ever.

You are most welcome! We always try to help here :) 

> 
> @Aaron, @William, @Wolfgang: one thing, that makes me nervous is the errors
> behavior. In Firefox, it seems to be sandbox related, but no real failure
> there, apart from some hints of failing access to /dev/random. Why/how does
> this break specific areas like WebRTC only, while most of the rest is acting
> fine?
> 
> Without understanding, what and how it is going wrong, we'll have a hard
> time to look for similar issues in other packages.

Code generation bugs are hard to detect at runtime. There are absolutely things
you can do as a developer in your project to try to detect anomalies at run
time like this, but not every project has written things so defensively. So I
think it's honestly hard to actually make this "fail better" in this situation
without projects making defensive changes. 

> 
> 1. llvm is very slow to compile, so when compiling rustc / cargo is good to
> use the llvm from the system to gain some speed.

Correct - but we can speed it up with the use of sccache in OBS, which I have
been trying to have included in our OBS instances (if you know any OBS
admins/maintainers, please get them to contact me because I can't seem to get
in touch with any of them ...)

> 
> 2. rustc is expected to work with some range of llvm versions

"yes-ish". Even if it works, we still need to focus on "what received the most
testing". Software always is reflective of, and works best in the way it was
developed, and for rust that means lock-stepping llvm to rust versions, because
that's how it's developed.

From now, I intend to pair rust versions to llvm versions rather than allowing
it to change on us mid-release of a rust cycle like happened here, which
introduced this chaos.

The reason we have to build llvm in leap15.3 is just because llvm13 isn't
available there yet, else we could use the distro llvm. I can't remember the
exact details about llvm and out-of-band updates in 15.3 but if 13 becomes
available we can swap to that potentially. 

> I think that they are fair points, and TW can help in detecting early issues.
> 
> But still I think that there should be an outcome of this bug, like working
> on enabling the tests inside the %check section on the rust package.

That's an option, but it's also another time consuming option. OBS already is
excruciatingly slow to build rust taking 2hrs or more - and that's with distro
llvm, not the bundled llvm. Enabling the tests will likely only make this
worst. (Contrast my home lab, where I can build rust in 40 mins with cold
sccache and 10 minutes with hot cache). 

we could do the checks just on package submit time instead as an option, but
then we'd only be testing on x86_64 (for now, I intend to get aarch64 hardware
soon). 

Another option to consider too is extended openqa testing of firefox *as well*
as the rust checks. Clearly this was missed and every time we see a failure
like this, we can potential grow openqa to detect issues like this. 

Anyway, the rust1.55 updates are accepted and are in the Factory/Tumbleweed
pipeline at the moment, and once that's accepted, I have staged the 1.55
updates for 15.3 in SLE, which will then flow out to leap. So I think this bug
should probably switch focus to proactive steps to detect and prevent issues
like this.