09.08.2019 22:40, Michal Suchánek пишет:

On Fri, 09 Aug 2019 17:57:02 +0200 Takashi Iwai wrote: Hello,

...

Hi,

- The firmware files are split into a bunch of sub-packages (e.g. kernel-firmware-iwlwifi, kernel-firmware-media, etc), so that user can install only a small set of firmware files. Each firmware file is compressed in XZ format, hence these packages are supported only by 5.3 or later kernel. And each package contains Supplements rpm tag for module aliases that are generated from our current and past kernel binaries, so that the sub-package your hardware requires may be installed automatically via zypper install-recommends.

I think importing aliases from different package is fragile and prone to getting outdated. What is the advantage?

I understood it firmware packages should Supplement hardware aliases so firmware is automatically pulled in if matching hardware is present.

Clearly the packages containing kernel modules know their aliases and required firmware at build time, and the firmware packages know the firmware they contain at build time too. So providing/recommending the firmware files should give working drivers so long as a package that contains the firmware is available.

Most drivers are in main kernel package. What is the point of splitting firmware into subpackages if all these subpackages will be pulled in by main kernel anyway?

With the necessity to shuffle data between different packages the thing is going to break, especially if we ever want to use one firmware package on multiple codestreams.

Thanks

Michal

-- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Sat, 10 Aug 2019 09:31:02 +0200 Takashi Iwai wrote:

On Fri, 09 Aug 2019 21:40:30 +0200, Michal Suchánek wrote:

...

On Fri, 09 Aug 2019 17:57:02 +0200 Takashi Iwai wrote: Hello,

...

Hi,

- The firmware files are split into a bunch of sub-packages (e.g. kernel-firmware-iwlwifi, kernel-firmware-media, etc), so that user can install only a small set of firmware files. Each firmware file is compressed in XZ format, hence these packages are supported only by 5.3 or later kernel. And each package contains Supplements rpm tag for module aliases that are generated from our current and past kernel binaries, so that the sub-package your hardware requires may be installed automatically via zypper install-recommends.

I think importing aliases from different package is fragile and prone to getting outdated. What is the advantage?

The package dependency. kernel-firmware itself is noarch, and yet we need the modaliases from all kernel archs and flavors. If we extract the modaliases at the build time, we'd need to fetch all these at each build.

...

Clearly the packages containing kernel modules know their aliases and required firmware at build time, and the firmware packages know the firmware they contain at build time too. So providing/recommending the firmware files should give working drivers so long as a package that contains the firmware is available.

Yes, this direction of the firmware requirement would work as well. OTOH, a downside is that this will bring all firmwares of the installed driver modules that aren't used -- e.g. when you install kernel-default, it'd drag effectively all kernel firmware files even if you don't use all of them.

In anyway, both mechanisms aren't mutual exclusive, we can implement both for completeness.

...

With the necessity to shuffle data between different packages the thing is going to break, especially if we ever want to use one firmware package on multiple codestreams.

Well, we may provide an option:

- Install firmware per recommends from kernel binary: this will install lots of unneeded files but safer. This can be the default behavior of the installer. Almost equivalent with the current state, too (kernel-default recommends kernel-firmware).

- Install firmware per modaliases: the system would set up the appropriate locks to kernel-firmware and kernel-firmware-all, and prune the unneeded firmware packages. Maybe this is a feature implemented on YaST, I don't know whether we can do by any other smart method.

For this you don't need the modaliases in the firmware, though. You will know which modules are used by modalias and infer which firmware is needed from module requirements. I suppose writing a separate tool for this is OK. yast can integrate it if desired. Thanks Michal -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Sat, 10 Aug 2019 10:35:18 +0200, Takashi Iwai wrote:

On Sat, 10 Aug 2019 10:04:36 +0200, Michal Suchánek wrote:

...

On Sat, 10 Aug 2019 09:31:02 +0200 Takashi Iwai wrote:

...

On Fri, 09 Aug 2019 21:40:30 +0200, Michal Suchánek wrote:

...

On Fri, 09 Aug 2019 17:57:02 +0200 Takashi Iwai wrote: Hello,

...

Hi,

- The firmware files are split into a bunch of sub-packages (e.g. kernel-firmware-iwlwifi, kernel-firmware-media, etc), so that user can install only a small set of firmware files. Each firmware file is compressed in XZ format, hence these packages are supported only by 5.3 or later kernel. And each package contains Supplements rpm tag for module aliases that are generated from our current and past kernel binaries, so that the sub-package your hardware requires may be installed automatically via zypper install-recommends.

I think importing aliases from different package is fragile and prone to getting outdated. What is the advantage?

The package dependency. kernel-firmware itself is noarch, and yet we need the modaliases from all kernel archs and flavors. If we extract the modaliases at the build time, we'd need to fetch all these at each build.

...

Clearly the packages containing kernel modules know their aliases and required firmware at build time, and the firmware packages know the firmware they contain at build time too. So providing/recommending the firmware files should give working drivers so long as a package that contains the firmware is available.

Yes, this direction of the firmware requirement would work as well. OTOH, a downside is that this will bring all firmwares of the installed driver modules that aren't used -- e.g. when you install kernel-default, it'd drag effectively all kernel firmware files even if you don't use all of them.

In anyway, both mechanisms aren't mutual exclusive, we can implement both for completeness.

...

With the necessity to shuffle data between different packages the thing is going to break, especially if we ever want to use one firmware package on multiple codestreams.

Well, we may provide an option:

- Install firmware per recommends from kernel binary: this will install lots of unneeded files but safer. This can be the default behavior of the installer. Almost equivalent with the current state, too (kernel-default recommends kernel-firmware).

- Install firmware per modaliases: the system would set up the appropriate locks to kernel-firmware and kernel-firmware-all, and prune the unneeded firmware packages. Maybe this is a feature implemented on YaST, I don't know whether we can do by any other smart method.

For this you don't need the modaliases in the firmware, though. You will know which modules are used by modalias and infer which firmware is needed from module requirements. I suppose writing a separate tool for this is OK. yast can integrate it if desired.

Right, it's about the timing when the modalias <-> firmware mapping needs to be generated.

Rethinking on the suggested idea, I believe the biggest obstacle is how to trigger this new post-installation. For the new way, we'll need: - Read modules.alias or whatever and list up the modules that match with the running system configuration (PCI, USB, etc). - Read modinfo output (or some static mapping for extra) for the listed modules and install the firmware files; the firmware files can be found as firmware($FILE) Provides of kernel-firmware-* package rpms. And I have no idea whether whole this procedure can be integrated into any existing package installation phase. Or, would we schedule another script at post(trans) and there we'll rerun zypper again for the requested firmware files? Also I don't know what's the easiest implementation for the first item. Any existing framework for that? thanks, Takashi -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Sat, 10 Aug 2019 15:41:02 +0200 Takashi Iwai wrote:

On Sat, 10 Aug 2019 10:35:18 +0200, Takashi Iwai wrote:

...

On Sat, 10 Aug 2019 10:04:36 +0200, Michal Suchánek wrote:

...

On Sat, 10 Aug 2019 09:31:02 +0200 Takashi Iwai wrote:

...

On Fri, 09 Aug 2019 21:40:30 +0200, Michal Suchánek wrote:

...

On Fri, 09 Aug 2019 17:57:02 +0200 Takashi Iwai wrote: Hello,

...

Hi,

- The firmware files are split into a bunch of sub-packages (e.g. kernel-firmware-iwlwifi, kernel-firmware-media, etc), so that user can install only a small set of firmware files. Each firmware file is compressed in XZ format, hence these packages are supported only by 5.3 or later kernel. And each package contains Supplements rpm tag for module aliases that are generated from our current and past kernel binaries, so that the sub-package your hardware requires may be installed automatically via zypper install-recommends.

I think importing aliases from different package is fragile and prone to getting outdated. What is the advantage?

The package dependency. kernel-firmware itself is noarch, and yet we need the modaliases from all kernel archs and flavors. If we extract the modaliases at the build time, we'd need to fetch all these at each build.

...

Clearly the packages containing kernel modules know their aliases and required firmware at build time, and the firmware packages know the firmware they contain at build time too. So providing/recommending the firmware files should give working drivers so long as a package that contains the firmware is available.

Yes, this direction of the firmware requirement would work as well. OTOH, a downside is that this will bring all firmwares of the installed driver modules that aren't used -- e.g. when you install kernel-default, it'd drag effectively all kernel firmware files even if you don't use all of them.

In anyway, both mechanisms aren't mutual exclusive, we can implement both for completeness.

...

With the necessity to shuffle data between different packages the thing is going to break, especially if we ever want to use one firmware package on multiple codestreams.

Well, we may provide an option:

- Install firmware per recommends from kernel binary: this will install lots of unneeded files but safer. This can be the default behavior of the installer. Almost equivalent with the current state, too (kernel-default recommends kernel-firmware).

- Install firmware per modaliases: the system would set up the appropriate locks to kernel-firmware and kernel-firmware-all, and prune the unneeded firmware packages. Maybe this is a feature implemented on YaST, I don't know whether we can do by any other smart method.

For this you don't need the modaliases in the firmware, though. You will know which modules are used by modalias and infer which firmware is needed from module requirements. I suppose writing a separate tool for this is OK. yast can integrate it if desired.

Right, it's about the timing when the modalias <-> firmware mapping needs to be generated.

Rethinking on the suggested idea, I believe the biggest obstacle is how to trigger this new post-installation. For the new way, we'll need:

- Read modules.alias or whatever and list up the modules that match with the running system configuration (PCI, USB, etc).

- Read modinfo output (or some static mapping for extra) for the listed modules and install the firmware files; the firmware files can be found as firmware($FILE) Provides of kernel-firmware-* package rpms.

Why do you need to do anything this complicated? If the module is installed it will pull in the firmware. If you customize the list of modules to install (eg. you list modules by modalias, append that to kernel-default-base.spec, and rebuild it) then you get potentially less firmware files. If not you get everything by default. Thanks Michal -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Fri, 16 Aug 2019 12:17:30 +0200, Michal Suchánek wrote:

On Fri, 16 Aug 2019 12:11:26 +0200 Michal Suchánek wrote:

...

On Sat, 10 Aug 2019 15:41:02 +0200 Takashi Iwai wrote:

...

On Sat, 10 Aug 2019 10:35:18 +0200, Takashi Iwai wrote:

...

On Sat, 10 Aug 2019 10:04:36 +0200, Michal Suchánek wrote:

...

On Sat, 10 Aug 2019 09:31:02 +0200 Takashi Iwai wrote:

...

On Fri, 09 Aug 2019 21:40:30 +0200, Michal Suchánek wrote: > > On Fri, 09 Aug 2019 17:57:02 +0200 > Takashi Iwai wrote: > Hello, > > Hi, > > > > - The firmware files are split into a bunch of sub-packages (e.g. > > kernel-firmware-iwlwifi, kernel-firmware-media, etc), so that user > > can install only a small set of firmware files. > > Each firmware file is compressed in XZ format, hence these packages > > are supported only by 5.3 or later kernel. > > And each package contains Supplements rpm tag for module aliases > > that are generated from our current and past kernel binaries, so > > that the sub-package your hardware requires may be installed > > automatically via zypper install-recommends. > > I think importing aliases from different package is fragile and prone > to getting outdated. What is the advantage?

The package dependency. kernel-firmware itself is noarch, and yet we need the modaliases from all kernel archs and flavors. If we extract the modaliases at the build time, we'd need to fetch all these at each build.

> Clearly the packages containing kernel modules know their aliases and > required firmware at build time, and the firmware packages know the > firmware they contain at build time too. So providing/recommending the > firmware files should give working drivers so long as a package that > contains the firmware is available.

Yes, this direction of the firmware requirement would work as well. OTOH, a downside is that this will bring all firmwares of the installed driver modules that aren't used -- e.g. when you install kernel-default, it'd drag effectively all kernel firmware files even if you don't use all of them.

In anyway, both mechanisms aren't mutual exclusive, we can implement both for completeness.

> With the necessity to shuffle data between different packages the thing > is going to break, especially if we ever want to use one firmware > package on multiple codestreams.

Well, we may provide an option:

- Install firmware per recommends from kernel binary: this will install lots of unneeded files but safer. This can be the default behavior of the installer. Almost equivalent with the current state, too (kernel-default recommends kernel-firmware).

- Install firmware per modaliases: the system would set up the appropriate locks to kernel-firmware and kernel-firmware-all, and prune the unneeded firmware packages. Maybe this is a feature implemented on YaST, I don't know whether we can do by any other smart method.

For this you don't need the modaliases in the firmware, though. You will know which modules are used by modalias and infer which firmware is needed from module requirements. I suppose writing a separate tool for this is OK. yast can integrate it if desired.

Right, it's about the timing when the modalias <-> firmware mapping needs to be generated.

Rethinking on the suggested idea, I believe the biggest obstacle is how to trigger this new post-installation. For the new way, we'll need:

- Read modules.alias or whatever and list up the modules that match with the running system configuration (PCI, USB, etc).

- Read modinfo output (or some static mapping for extra) for the listed modules and install the firmware files; the firmware files can be found as firmware($FILE) Provides of kernel-firmware-* package rpms.

Why do you need to do anything this complicated? If the module is installed it will pull in the firmware. If you customize the list of modules to install (eg. you list modules by modalias, append that to kernel-default-base.spec, and rebuild it) then you get potentially less firmware files. If not you get everything by default.

Or in other words where do we do any selection of modules to install, at all? The only case is

- select kernel-default, kernel-default-base or any other kernel-deafult-foo - select any additional KMP to install

In any of these cases the package you select for installation can depend on required firmware on its own without any additional scripting, shuffling modaliases between packages, manually created tables that are bound to get outdated, etc.

Or what am I missing?

As mentioned in the previous post, there is no mapping between device info and firmware files provided in a form that can be used by resolver at the point you install these kernel modules. Takashi -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Fri, 16 Aug 2019 15:32:26 +0200, Michal Suchánek wrote:

On Fri, 16 Aug 2019 13:34:58 +0200 Takashi Iwai wrote:

...

On Fri, 16 Aug 2019 12:17:30 +0200, Michal Suchánek wrote:

...

On Fri, 16 Aug 2019 12:11:26 +0200 Michal Suchánek wrote:

...

On Sat, 10 Aug 2019 15:41:02 +0200 Takashi Iwai wrote:

...

On Sat, 10 Aug 2019 10:35:18 +0200, Takashi Iwai wrote:

...

On Sat, 10 Aug 2019 10:04:36 +0200, Michal Suchánek wrote: > > On Sat, 10 Aug 2019 09:31:02 +0200 > Takashi Iwai wrote: > > > On Fri, 09 Aug 2019 21:40:30 +0200, > > Michal Suchánek wrote: > > > > > > On Fri, 09 Aug 2019 17:57:02 +0200 > > > Takashi Iwai wrote: > > > Hello, > > > > Hi, > > > > > > > > - The firmware files are split into a bunch of sub-packages (e.g. > > > > kernel-firmware-iwlwifi, kernel-firmware-media, etc), so that user > > > > can install only a small set of firmware files. > > > > Each firmware file is compressed in XZ format, hence these packages > > > > are supported only by 5.3 or later kernel. > > > > And each package contains Supplements rpm tag for module aliases > > > > that are generated from our current and past kernel binaries, so > > > > that the sub-package your hardware requires may be installed > > > > automatically via zypper install-recommends. > > > > > > I think importing aliases from different package is fragile and prone > > > to getting outdated. What is the advantage? > > > > The package dependency. kernel-firmware itself is noarch, and yet we > > need the modaliases from all kernel archs and flavors. If we extract > > the modaliases at the build time, we'd need to fetch all these at each > > build. > > > > > Clearly the packages containing kernel modules know their aliases and > > > required firmware at build time, and the firmware packages know the > > > firmware they contain at build time too. So providing/recommending the > > > firmware files should give working drivers so long as a package that > > > contains the firmware is available. > > > > Yes, this direction of the firmware requirement would work as well. > > OTOH, a downside is that this will bring all firmwares of the > > installed driver modules that aren't used -- e.g. when you install > > kernel-default, it'd drag effectively all kernel firmware files even > > if you don't use all of them. > > > > In anyway, both mechanisms aren't mutual exclusive, we can implement > > both for completeness. > > > > > With the necessity to shuffle data between different packages the thing > > > is going to break, especially if we ever want to use one firmware > > > package on multiple codestreams. > > > > Well, we may provide an option: > > > > - Install firmware per recommends from kernel binary: > > this will install lots of unneeded files but safer. This can be the > > default behavior of the installer. Almost equivalent with the > > current state, too (kernel-default recommends kernel-firmware). > > > > - Install firmware per modaliases: > > the system would set up the appropriate locks to kernel-firmware and > > kernel-firmware-all, and prune the unneeded firmware packages. > > Maybe this is a feature implemented on YaST, I don't know whether we > > can do by any other smart method. > > For this you don't need the modaliases in the firmware, though. You will > know which modules are used by modalias and infer which firmware is > needed from module requirements. I suppose writing a separate tool for > this is OK. yast can integrate it if desired.

Right, it's about the timing when the modalias <-> firmware mapping needs to be generated.

Rethinking on the suggested idea, I believe the biggest obstacle is how to trigger this new post-installation. For the new way, we'll need:

- Read modules.alias or whatever and list up the modules that match with the running system configuration (PCI, USB, etc).

- Read modinfo output (or some static mapping for extra) for the listed modules and install the firmware files; the firmware files can be found as firmware($FILE) Provides of kernel-firmware-* package rpms.

Why do you need to do anything this complicated? If the module is installed it will pull in the firmware. If you customize the list of modules to install (eg. you list modules by modalias, append that to kernel-default-base.spec, and rebuild it) then you get potentially less firmware files. If not you get everything by default.

Or in other words where do we do any selection of modules to install, at all? The only case is

- select kernel-default, kernel-default-base or any other kernel-deafult-foo - select any additional KMP to install

In any of these cases the package you select for installation can depend on required firmware on its own without any additional scripting, shuffling modaliases between packages, manually created tables that are bound to get outdated, etc.

Or what am I missing?

As mentioned in the previous post, there is no mapping between device info and firmware files provided in a form that can be used by resolver at the point you install these kernel modules.

It is, in the dependencies of the package containing the kernel modules.

Sorry, I don't get your point. Suppose you're installing kernel-default.rpm on your laptop and you want to install only the firmware for WiFi chip on the machine. How would the installer find it out only from the package information? thanks, Takashi -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Fri, 16 Aug 2019 15:44:55 +0200, Michal Suchánek wrote:

On Fri, 16 Aug 2019 15:39:54 +0200 Takashi Iwai wrote:

...

On Fri, 16 Aug 2019 15:32:26 +0200, Michal Suchánek wrote:

...

On Fri, 16 Aug 2019 13:34:58 +0200 Takashi Iwai wrote:

...

On Fri, 16 Aug 2019 12:17:30 +0200, Michal Suchánek wrote:

...

On Fri, 16 Aug 2019 12:11:26 +0200 Michal Suchánek wrote:

...

On Sat, 10 Aug 2019 15:41:02 +0200 Takashi Iwai wrote:

> On Sat, 10 Aug 2019 10:35:18 +0200, > Takashi Iwai wrote: > > > > On Sat, 10 Aug 2019 10:04:36 +0200, > > Michal Suchánek wrote: > > > > > > On Sat, 10 Aug 2019 09:31:02 +0200 > > > Takashi Iwai wrote: > > > > > > > On Fri, 09 Aug 2019 21:40:30 +0200, > > > > Michal Suchánek wrote: > > > > > > > > > > On Fri, 09 Aug 2019 17:57:02 +0200 > > > > > Takashi Iwai wrote: > > > > > Hello, > > > > > > Hi, > > > > > > > > > > > > - The firmware files are split into a bunch of sub-packages (e.g. > > > > > > kernel-firmware-iwlwifi, kernel-firmware-media, etc), so that user > > > > > > can install only a small set of firmware files. > > > > > > Each firmware file is compressed in XZ format, hence these packages > > > > > > are supported only by 5.3 or later kernel. > > > > > > And each package contains Supplements rpm tag for module aliases > > > > > > that are generated from our current and past kernel binaries, so > > > > > > that the sub-package your hardware requires may be installed > > > > > > automatically via zypper install-recommends. > > > > > > > > > > I think importing aliases from different package is fragile and prone > > > > > to getting outdated. What is the advantage? > > > > > > > > The package dependency. kernel-firmware itself is noarch, and yet we > > > > need the modaliases from all kernel archs and flavors. If we extract > > > > the modaliases at the build time, we'd need to fetch all these at each > > > > build. > > > > > > > > > Clearly the packages containing kernel modules know their aliases and > > > > > required firmware at build time, and the firmware packages know the > > > > > firmware they contain at build time too. So providing/recommending the > > > > > firmware files should give working drivers so long as a package that > > > > > contains the firmware is available. > > > > > > > > Yes, this direction of the firmware requirement would work as well. > > > > OTOH, a downside is that this will bring all firmwares of the > > > > installed driver modules that aren't used -- e.g. when you install > > > > kernel-default, it'd drag effectively all kernel firmware files even > > > > if you don't use all of them. > > > > > > > > In anyway, both mechanisms aren't mutual exclusive, we can implement > > > > both for completeness. > > > > > > > > > With the necessity to shuffle data between different packages the thing > > > > > is going to break, especially if we ever want to use one firmware > > > > > package on multiple codestreams. > > > > > > > > Well, we may provide an option: > > > > > > > > - Install firmware per recommends from kernel binary: > > > > this will install lots of unneeded files but safer. This can be the > > > > default behavior of the installer. Almost equivalent with the > > > > current state, too (kernel-default recommends kernel-firmware). > > > > > > > > - Install firmware per modaliases: > > > > the system would set up the appropriate locks to kernel-firmware and > > > > kernel-firmware-all, and prune the unneeded firmware packages. > > > > Maybe this is a feature implemented on YaST, I don't know whether we > > > > can do by any other smart method. > > > > > > For this you don't need the modaliases in the firmware, though. You will > > > know which modules are used by modalias and infer which firmware is > > > needed from module requirements. I suppose writing a separate tool for > > > this is OK. yast can integrate it if desired. > > > > Right, it's about the timing when the modalias <-> firmware mapping > > needs to be generated. > > Rethinking on the suggested idea, I believe the biggest obstacle is > how to trigger this new post-installation. For the new way, we'll > need: > > - Read modules.alias or whatever and list up the modules that match > with the running system configuration (PCI, USB, etc). > > - Read modinfo output (or some static mapping for extra) for the > listed modules and install the firmware files; > the firmware files can be found as firmware($FILE) Provides of > kernel-firmware-* package rpms.

Why do you need to do anything this complicated? If the module is installed it will pull in the firmware. If you customize the list of modules to install (eg. you list modules by modalias, append that to kernel-default-base.spec, and rebuild it) then you get potentially less firmware files. If not you get everything by default.

Or in other words where do we do any selection of modules to install, at all? The only case is

- select kernel-default, kernel-default-base or any other kernel-deafult-foo - select any additional KMP to install

In any of these cases the package you select for installation can depend on required firmware on its own without any additional scripting, shuffling modaliases between packages, manually created tables that are bound to get outdated, etc.

Or what am I missing?

As mentioned in the previous post, there is no mapping between device info and firmware files provided in a form that can be used by resolver at the point you install these kernel modules.

It is, in the dependencies of the package containing the kernel modules.

Sorry, I don't get your point.

Suppose you're installing kernel-default.rpm on your laptop and you want to install only the firmware for WiFi chip on the machine. How would the installer find it out only from the package information?

And you are running what on the laptop when the installer is running? Do you not have a kernel?

The installer doesn't have to load the module at the installation time.

And why would you want to do that, anyway? If you replace the WiFi card you lose connectivity.

This action is what many users have demanded for long time. The selected firmware installation is the only way to reduce the installation size significantly, after all. Ifsomething missing in future, of course you'll need either net or media access. But it's often not that fatal. If there is a net or media access, all you need is to run "zypper install-recommends".

What you might want to do is put drivers like liquidio or amdgpu that have huge firmware into a separate package and not install them at all if you do not need them.

Well, it's exactly the goal of the new kernel-firmware packaging... Don't get me wrong: I don't push this installation mode as default. But at least it allows users to do that (only manually, so far). thanks, Takashi -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Fri, 16 Aug 2019 16:13:07 +0200, Michal Suchánek wrote:

On Fri, 16 Aug 2019 15:58:47 +0200 Takashi Iwai wrote:

...

On Fri, 16 Aug 2019 15:44:55 +0200, Michal Suchánek wrote:

...

On Fri, 16 Aug 2019 15:39:54 +0200 Takashi Iwai wrote:

...

On Fri, 16 Aug 2019 15:32:26 +0200, Michal Suchánek wrote:

...

On Fri, 16 Aug 2019 13:34:58 +0200 Takashi Iwai wrote:

...

On Fri, 16 Aug 2019 12:17:30 +0200, Michal Suchánek wrote: > > On Fri, 16 Aug 2019 12:11:26 +0200 > Michal Suchánek wrote: > > > On Sat, 10 Aug 2019 15:41:02 +0200 > > Takashi Iwai wrote: > > > > > On Sat, 10 Aug 2019 10:35:18 +0200, > > > Takashi Iwai wrote: > > > > > > > > On Sat, 10 Aug 2019 10:04:36 +0200, > > > > Michal Suchánek wrote: > > > > > > > > > > On Sat, 10 Aug 2019 09:31:02 +0200 > > > > > Takashi Iwai wrote: > > > > > > > > > > > On Fri, 09 Aug 2019 21:40:30 +0200, > > > > > > Michal Suchánek wrote: > > > > > > > > > > > > > > On Fri, 09 Aug 2019 17:57:02 +0200 > > > > > > > Takashi Iwai wrote: > > > > > > > Hello, > > > > > > > > Hi, > > > > > > > > > > > > > > > > - The firmware files are split into a bunch of sub-packages (e.g. > > > > > > > > kernel-firmware-iwlwifi, kernel-firmware-media, etc), so that user > > > > > > > > can install only a small set of firmware files. > > > > > > > > Each firmware file is compressed in XZ format, hence these packages > > > > > > > > are supported only by 5.3 or later kernel. > > > > > > > > And each package contains Supplements rpm tag for module aliases > > > > > > > > that are generated from our current and past kernel binaries, so > > > > > > > > that the sub-package your hardware requires may be installed > > > > > > > > automatically via zypper install-recommends. > > > > > > > > > > > > > > I think importing aliases from different package is fragile and prone > > > > > > > to getting outdated. What is the advantage? > > > > > > > > > > > > The package dependency. kernel-firmware itself is noarch, and yet we > > > > > > need the modaliases from all kernel archs and flavors. If we extract > > > > > > the modaliases at the build time, we'd need to fetch all these at each > > > > > > build. > > > > > > > > > > > > > Clearly the packages containing kernel modules know their aliases and > > > > > > > required firmware at build time, and the firmware packages know the > > > > > > > firmware they contain at build time too. So providing/recommending the > > > > > > > firmware files should give working drivers so long as a package that > > > > > > > contains the firmware is available. > > > > > > > > > > > > Yes, this direction of the firmware requirement would work as well. > > > > > > OTOH, a downside is that this will bring all firmwares of the > > > > > > installed driver modules that aren't used -- e.g. when you install > > > > > > kernel-default, it'd drag effectively all kernel firmware files even > > > > > > if you don't use all of them. > > > > > > > > > > > > In anyway, both mechanisms aren't mutual exclusive, we can implement > > > > > > both for completeness. > > > > > > > > > > > > > With the necessity to shuffle data between different packages the thing > > > > > > > is going to break, especially if we ever want to use one firmware > > > > > > > package on multiple codestreams. > > > > > > > > > > > > Well, we may provide an option: > > > > > > > > > > > > - Install firmware per recommends from kernel binary: > > > > > > this will install lots of unneeded files but safer. This can be the > > > > > > default behavior of the installer. Almost equivalent with the > > > > > > current state, too (kernel-default recommends kernel-firmware). > > > > > > > > > > > > - Install firmware per modaliases: > > > > > > the system would set up the appropriate locks to kernel-firmware and > > > > > > kernel-firmware-all, and prune the unneeded firmware packages. > > > > > > Maybe this is a feature implemented on YaST, I don't know whether we > > > > > > can do by any other smart method. > > > > > > > > > > For this you don't need the modaliases in the firmware, though. You will > > > > > know which modules are used by modalias and infer which firmware is > > > > > needed from module requirements. I suppose writing a separate tool for > > > > > this is OK. yast can integrate it if desired. > > > > > > > > Right, it's about the timing when the modalias <-> firmware mapping > > > > needs to be generated. > > > > > > Rethinking on the suggested idea, I believe the biggest obstacle is > > > how to trigger this new post-installation. For the new way, we'll > > > need: > > > > > > - Read modules.alias or whatever and list up the modules that match > > > with the running system configuration (PCI, USB, etc). > > > > > > - Read modinfo output (or some static mapping for extra) for the > > > listed modules and install the firmware files; > > > the firmware files can be found as firmware($FILE) Provides of > > > kernel-firmware-* package rpms. > > > > Why do you need to do anything this complicated? If the module is > > installed it will pull in the firmware. If you customize the list of > > modules to install (eg. you list modules by modalias, append that to > > kernel-default-base.spec, and rebuild it) then you get potentially less > > firmware files. If not you get everything by default. > > Or in other words where do we do any selection of modules to install, > at all? The only case is > > - select kernel-default, kernel-default-base or any other > kernel-deafult-foo > - select any additional KMP to install > > In any of these cases the package you select for installation can > depend on required firmware on its own without any additional > scripting, shuffling modaliases between packages, manually created > tables that are bound to get outdated, etc. > > Or what am I missing?

As mentioned in the previous post, there is no mapping between device info and firmware files provided in a form that can be used by resolver at the point you install these kernel modules.

It is, in the dependencies of the package containing the kernel modules.

Sorry, I don't get your point.

Suppose you're installing kernel-default.rpm on your laptop and you want to install only the firmware for WiFi chip on the machine. How would the installer find it out only from the package information?

And you are running what on the laptop when the installer is running? Do you not have a kernel?

The installer doesn't have to load the module at the installation time.

...

And why would you want to do that, anyway? If you replace the WiFi card you lose connectivity.

This action is what many users have demanded for long time. The selected firmware installation is the only way to reduce the installation size significantly, after all.

Selected, yes. Selected per device, no.

...

Ifsomething missing in future, of course you'll need either net or media access. But it's often not that fatal. If there is a net or media access, all you need is to run "zypper install-recommends".

It will be like Windows. Your network card died, you replace it, and you do not have the firmware for the new card. You can plug in an USB network card but don't have the firmware for that either. So in the end you will frantically search for a thumb drive, download some package using the web browser on another computer with no security verification whatsoever and manually force-feed it to the defunct system. Great, isn't it?

You have the rescue media in that case. It's like Windows, so why not? Users are used to that ;) (And how often does it happen nowadays? Most of cheap USB network devices run fine without firmware...) In anyway, we won't mandate this installation scenario. The default should be still install-all mode.

And of course "zypper install-recommends" cannot fix it. If it did the firmware would be there to start with because installing recommends is the default.

The Supplements with modalias would still match even if you lock out kernel-firmware. So the kernel-firmware-xxx sub package would match and be installed accordingly via zypper inr.

...

...

What you might want to do is put drivers like liquidio or amdgpu that have huge firmware into a separate package and not install them at all if you do not need them.

Well, it's exactly the goal of the new kernel-firmware packaging...

So it needs to be matched with kernel module packaging.

Yes, and the frequency of the modalias changes isn't that high, so we can follow manually. It was the reason I took this approach for now. I know this is pretty sub-optimal, so no need to go deeper wrt this point.

...

Don't get me wrong: I don't push this installation mode as default. But at least it allows users to do that (only manually, so far).

You can manually do it without supplement tables. Remove firmware you don't want, lock it, done.

And how can a novice user know which firmware file, and more important,y which corresponding kernel-firmware-xxx package is required and which not? The manual inspection isn't always trivial.

Does not need to be integrated into the installer, either. It's not like a freshly installed system is supposed to be so tight on disk space that a few hundred MB extra firmware files won't fit.

I can't agree with that; a few hundreds MB is a few hundreds MB, not a nut. It's still huge. And, comparing with other distros regarding this area (storage saving), I'm afraid that we are pretty much behind them.

How will you update anything like that?

The update is a totally different scenario and it'll just keep the whole packages (via meta package) unless explicitly commanded.

Embedded systems and appliances might be a different story but you have pre-built images for those where the firmware might be locked out to start with.

Yes. And, as repeatedly mentioned, I'm not sticking with the current approach with modalias. If any other better way is provided, we can switch to it immediately and drop these static mapping. But currently there is no such mechanism, and we still need a way to let users know which firmware to install *automagically*. So the modalias still remains the kernel-firmware side for the time being. As an alternative, I can think of extending the kernel binary / KMP to provide the mapping -- if zypper / rpm allows something like: Recommends: firmware(xyz) if modalias(pci:xxx) This should be able to be extracted at the build time. thanks, Takashi -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Fri, 16 Aug 2019 17:53:23 +0200, Michal Suchánek wrote:

On Fri, 16 Aug 2019 16:50:53 +0200 Takashi Iwai wrote:

...

...

You can manually do it without supplement tables. Remove firmware you don't want, lock it, done.

And how can a novice user know which firmware file, and more important,y which corresponding kernel-firmware-xxx package is required and which not? The manual inspection isn't always trivial.

That's where I see room for improvement. Providing commandline tool that does the inspection would be nice.

Yes, such a helper would be needed if we'd go for other than install-recommends. The modalias implementation was taken just because "it works now as-is".

...

...

Does not need to be integrated into the installer, either. It's not like a freshly installed system is supposed to be so tight on disk space that a few hundred MB extra firmware files won't fit.

I can't agree with that; a few hundreds MB is a few hundreds MB, not a nut. It's still huge.

And, comparing with other distros regarding this area (storage saving), I'm afraid that we are pretty much behind them.

That's where a firmware cleanup tool comes in. If space is a concern you can do it. If not you need not be bothered.

I think the "cleanup" approach is rather dangerous. It may remove too much if it doesn't know what is actually needed. For example, not all modules show the firmware information, and if we do get rid of all unneeded files, such hidden requirement might be gone unexpectedly. A manual package lock might work, but it's too ugly. OTOH, the do-install-what-you-want approach won't remove anything unexpectedly. The drawback, of course, is that it might miss something. But you can install in addition manually but without lock.

It can be integrated into yast and whatnot if the interface is sane and there is user demand for the feature.

Right, the tooling is needed and it's missing. That's the reason I still keep the manual modalias in the new kernel-firmware package *FOR NOW*. That is, the modalias stuff may be dropped later once when other better way is provided. thanks, Takashi -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Samstag, 17. August 2019 04:12:41 CEST Larry Finger wrote:

On 8/16/19 12:57 PM, Takashi Iwai wrote:

...

On Fri, 16 Aug 2019 17:53:23 +0200,

Michal Suchánek wrote:

...

On Fri, 16 Aug 2019 16:50:53 +0200

Takashi Iwai wrote: [...]

How many packages are used for wifi firmware?

If this firmware is missing for a new wireless NIC, then recovery is not trivial.

The assertion made earlier was that modern, cheap USB wireless devices do not need external firmware. That does not match my experience. Of all the wireless devices that I have, only wireless G devices do not need external firmware, and I doubt that people are buying these for emergency backup.

Reinserting the quote:

...

...

(And how often does it happen nowadays? Most of cheap USB network devices run fine without firmware...)

^ Wired ethernet. The same is also true for any Android phone in tethering mode. The biggest offender for wifi firmware size probably is iwlwifi which is 106 MByte currently, b43 and rtlwifi are both ~1 MByte each. Kind regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen home: +49 241 53809034 mobile: +49 151 50412019

On 8/17/19 5:14 AM, Takashi Iwai wrote:

Yes, and I believe iwlwifi can be excluded from the mandatory list. That is, we may pick up the firmware files for USB ethernet / wifi devices that are needed "to be safe".

The Realtek are fairly small (750K compressed), and Atheros are small except for ath10k. We may split ath10k out of atheros subpkg, as ath10k is PCI-only. Mediatek is also not big (1.4M). Broadcom is only about brcmfmac, and b43 isn't included in the standard kernel-firmware package.

Firmware for b43 and b43legacy cannot be redistributed. For that reason, the package b43-fwcutter does not contain any firmware, but does have code to extract firmware from a number of binary files that Broadcom needs to allow to be distributed to meet the GPL requirements for a number of AP devices. This package is only 68 KiB, thus it could be included with every laptop installation.

Some USB WiFi and Ethernet firmware are currently in network and platform subpackages, and these may be moved into a new usb-network subpackage, for example.

Then the biggest remaining is Marvell, mostly about mwifiex. It has a USB variant, so this needs to be included.

I like the idea of a usb-network subpackage. If you want any help picking the firmware for USB devices, please let me know. Larry -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Sat, 17 Aug 2019 04:12:41 +0200, Larry Finger wrote:

On 8/16/19 12:57 PM, Takashi Iwai wrote:

...

On Fri, 16 Aug 2019 17:53:23 +0200, Michal Suchánek wrote:

...

On Fri, 16 Aug 2019 16:50:53 +0200 Takashi Iwai wrote:

...

...

You can manually do it without supplement tables. Remove firmware you don't want, lock it, done.

And how can a novice user know which firmware file, and more important,y which corresponding kernel-firmware-xxx package is required and which not? The manual inspection isn't always trivial.

That's where I see room for improvement. Providing commandline tool that does the inspection would be nice.

Yes, such a helper would be needed if we'd go for other than install-recommends. The modalias implementation was taken just because "it works now as-is".

...

...

...

Does not need to be integrated into the installer, either. It's not like a freshly installed system is supposed to be so tight on disk space that a few hundred MB extra firmware files won't fit.

I can't agree with that; a few hundreds MB is a few hundreds MB, not a nut. It's still huge.

And, comparing with other distros regarding this area (storage saving), I'm afraid that we are pretty much behind them.

That's where a firmware cleanup tool comes in. If space is a concern you can do it. If not you need not be bothered.

I think the "cleanup" approach is rather dangerous. It may remove too much if it doesn't know what is actually needed. For example, not all modules show the firmware information, and if we do get rid of all unneeded files, such hidden requirement might be gone unexpectedly. A manual package lock might work, but it's too ugly.

OTOH, the do-install-what-you-want approach won't remove anything unexpectedly. The drawback, of course, is that it might miss something. But you can install in addition manually but without lock.

...

It can be integrated into yast and whatnot if the interface is sane and there is user demand for the feature.

Right, the tooling is needed and it's missing. That's the reason I still keep the manual modalias in the new kernel-firmware package *FOR NOW*. That is, the modalias stuff may be dropped later once when other better way is provided.

Takashi,

How many packages are used for wifi firmware?

If this firmware is missing for a new wireless NIC, then recovery is not trivial.

The assertion made earlier was that modern, cheap USB wireless devices do not need external firmware. That does not match my experience. Of all the wireless devices that I have, only wireless G devices do not need external firmware, and I doubt that people are buying these for emergency backup.

Hm, then maybe we can add USB WiFi firmware to laptop or such patterns as Recommends. Currently I've split per vendor and/or functionality, e.g. kernel-firmware-atheros, kernel-firmware-iwlwifi, kernel-firmware-realtek, etc. Each package size is as below: kernel-firmware-amdgpu 5.9M kernel-firmware-atheros 6.1M kernel-firmware-bluetooth 3.5M kernel-firmware-bnx2 9.4M kernel-firmware-brcm 8.8M kernel-firmware-chelsio 2.7M kernel-firmware-dpaa2 1.9M kernel-firmware-i915 1.3M kernel-firmware-intel 2.4M kernel-firmware-iwlwifi 22M kernel-firmware-liquidio 16M kernel-firmware-marvell 7.4M kernel-firmware-media 3.1M kernel-firmware-mediatek 1.4M kernel-firmware-mellanox 6.3M kernel-firmware-network 3.3M kernel-firmware-nfp 4.4M kernel-firmware-nvidia 737K kernel-firmware-platform 1.0M kernel-firmware-qlogic 11M kernel-firmware-radeon 2.3M kernel-firmware-realtek 750K kernel-firmware-serial 383K kernel-firmware-sound 2.9M kernel-firmware-ti 4.3M kernel-firmware-ueagle 786K The realtek one is pretty small (750KB) while iwlwifi is very large (22MB) and atheros is a bit largish (6MB). Also network subpkg might be wroth for default. Of course, it's possible to split / re-group some entities. thanks, Takashi -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Fri, 16 Aug 2019 19:57:36 +0200 Takashi Iwai wrote:

On Fri, 16 Aug 2019 17:53:23 +0200, Michal Suchánek wrote:

...

On Fri, 16 Aug 2019 16:50:53 +0200 Takashi Iwai wrote:

...

...

You can manually do it without supplement tables. Remove firmware you don't want, lock it, done.

And how can a novice user know which firmware file, and more important,y which corresponding kernel-firmware-xxx package is required and which not? The manual inspection isn't always trivial.

That's where I see room for improvement. Providing commandline tool that does the inspection would be nice.

Yes, such a helper would be needed if we'd go for other than install-recommends. The modalias implementation was taken just because "it works now as-is".

...

...

...

Does not need to be integrated into the installer, either. It's not like a freshly installed system is supposed to be so tight on disk space that a few hundred MB extra firmware files won't fit.

I can't agree with that; a few hundreds MB is a few hundreds MB, not a nut. It's still huge.

And, comparing with other distros regarding this area (storage saving), I'm afraid that we are pretty much behind them.

That's where a firmware cleanup tool comes in. If space is a concern you can do it. If not you need not be bothered.

I think the "cleanup" approach is rather dangerous. It may remove too much if it doesn't know what is actually needed.

Yes, it should allow selecting a firmware package that is not obviously needed but you want it anyway.

For example, not all modules show the firmware information,

Then these modules are broken and cannot work with such tooling. Adding the dependency manually in some supplements table just papers over the issue. Thanks Michal -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Fri, 16 Aug 2019 12:11:26 +0200, Michal Suchánek wrote:

On Sat, 10 Aug 2019 15:41:02 +0200 Takashi Iwai wrote:

...

On Sat, 10 Aug 2019 10:35:18 +0200, Takashi Iwai wrote:

...

On Sat, 10 Aug 2019 10:04:36 +0200, Michal Suchánek wrote:

...

On Sat, 10 Aug 2019 09:31:02 +0200 Takashi Iwai wrote:

...

On Fri, 09 Aug 2019 21:40:30 +0200, Michal Suchánek wrote:

...

On Fri, 09 Aug 2019 17:57:02 +0200 Takashi Iwai wrote: Hello, > Hi, > > - The firmware files are split into a bunch of sub-packages (e.g. > kernel-firmware-iwlwifi, kernel-firmware-media, etc), so that user > can install only a small set of firmware files. > Each firmware file is compressed in XZ format, hence these packages > are supported only by 5.3 or later kernel. > And each package contains Supplements rpm tag for module aliases > that are generated from our current and past kernel binaries, so > that the sub-package your hardware requires may be installed > automatically via zypper install-recommends.

I think importing aliases from different package is fragile and prone to getting outdated. What is the advantage?

The package dependency. kernel-firmware itself is noarch, and yet we need the modaliases from all kernel archs and flavors. If we extract the modaliases at the build time, we'd need to fetch all these at each build.

...

Clearly the packages containing kernel modules know their aliases and required firmware at build time, and the firmware packages know the firmware they contain at build time too. So providing/recommending the firmware files should give working drivers so long as a package that contains the firmware is available.

Yes, this direction of the firmware requirement would work as well. OTOH, a downside is that this will bring all firmwares of the installed driver modules that aren't used -- e.g. when you install kernel-default, it'd drag effectively all kernel firmware files even if you don't use all of them.

In anyway, both mechanisms aren't mutual exclusive, we can implement both for completeness.

...

With the necessity to shuffle data between different packages the thing is going to break, especially if we ever want to use one firmware package on multiple codestreams.

Well, we may provide an option:

- Install firmware per recommends from kernel binary: this will install lots of unneeded files but safer. This can be the default behavior of the installer. Almost equivalent with the current state, too (kernel-default recommends kernel-firmware).

- Install firmware per modaliases: the system would set up the appropriate locks to kernel-firmware and kernel-firmware-all, and prune the unneeded firmware packages. Maybe this is a feature implemented on YaST, I don't know whether we can do by any other smart method.

For this you don't need the modaliases in the firmware, though. You will know which modules are used by modalias and infer which firmware is needed from module requirements. I suppose writing a separate tool for this is OK. yast can integrate it if desired.

Right, it's about the timing when the modalias <-> firmware mapping needs to be generated.

Rethinking on the suggested idea, I believe the biggest obstacle is how to trigger this new post-installation. For the new way, we'll need:

- Read modules.alias or whatever and list up the modules that match with the running system configuration (PCI, USB, etc).

- Read modinfo output (or some static mapping for extra) for the listed modules and install the firmware files; the firmware files can be found as firmware($FILE) Provides of kernel-firmware-* package rpms.

Why do you need to do anything this complicated? If the module is installed it will pull in the firmware.

... by Recommends: kernel-firmware, and it enforces all firmware files.

If you customize the list of modules to install (eg. you list modules by modalias, append that to kernel-default-base.spec, and rebuild it) then you get potentially less firmware files.

No, the mapping between modalias and the corresponding firmware is completely missing in the package level. That's the whole point. The Supplements with modalias is essentially device-info -> f/w mapping.

If not you get everything by default.

Right. So, in the current situation, you'll get everything in anyway, nothing different from today. However, you can still do zypper addlock kernel-firmware kernel-firmware-all to avoid the catch-all packages. Then you'd need some mechanism to choose which firmware to install. The modalias is one of the ways that have been confirmed to work over a decade. The maintenance of the modalias is yet another question, yes, and as already mentioned, I'm not sticking with this. But, as of today, there is nothing better way than a static mapping. That said, the current static modalias is a stop-gap solution until the other better way becomes ready. thanks, Takashi -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org