Re: Hints on debugging driver issues with GW7401 board

Hi, Sorry for the late followup on this thread. Also, I guess I didn't realize that my responses didn't go to the MicroOS list. As it turns out, I was able to track down the main problem that was causing issues with PCIe and USB with the GW7401 board that I have been using. One of the key drivers, "imx_bus", which is a part of the standard "initrd", was missing a kernel module dependency, "governor-userspace". Thankfully, though, all of the needed drivers are pre-built with the openSUSE Tumbleweed/MicroOS kernel as kernel modules, I didn't really need a custom kernel to resolve the issues. After performing a fair amount of work to get the installer to work using the embedded eMMC on the board, I was able to then add the "governor-userspace" kernel module to the local "initrd" using dracut by creating a new file named "/etc/dracut.conf.d/00-custom.conf" with the following contents: add_drivers+=" governor-userspace " and then using "transactional-update shell" to run: dracut --force in the "transactional-update shell", exiting the shell, and rebooting. Ideally, the default "initrd" would include the "governor-userspace" kernel module by default as well as "usb-conn-gpio" so the GW7401 could boot the installer directly from a USB drive and then properly have access to USB and PCIe devices. What is the right way to request that these two drivers get added to the installer's and MicroOS's default initrd? If that is unlikely to happen, is there a convenient way to build the installer "DVD" image with these kernel modules included in the "initrd"? Thanks, Paul On Fri, Sep 15, 2023 at 9:01 AM Paul Graham wrote:

Thanks for the reply! My responses to your questions show up in-line below.

Paul

On Fri, Sep 15, 2023 at 12:30 AM Fabian Vogt wrote:

...

Hi,

Am Freitag, 15. September 2023, 03:47:08 CEST schrieb Paul Graham:

...

...

So, my questions are: 1. Do you have any hints or suggestions on debugging issues with device trees and kernel module loading at boot time? Gateworks demonstrated that the board works fine with a Linux 6.4.6 kernel with their .config file, but they compile most, if not all, of their drivers directly into the kernel. Anyway, I believe that using the openSUSE kernels will work if I can work through some device tree issues.

What's the issue precisely? Usually any driver that can be built as a module can be used just fine when loaded as a module, so inherently using =m shouldn't be an issue. Is this about specific platform drivers which don't have any autoloading/hardware matching configured?

The main issue I am having is that the USB and PCIe interfaces and peripherals connected to those buses are not accessible after boot. So, my primary question is, why are the required modules not loading at boot time? I believe that the device tree is being properly provided to U-Boot and the kernel at boot time since I see a few of the required driver kernel modules loaded after boot. For example, Gateworks has a specific driver for their management controller ("gateworks_gsc") that wouldn't get dynamically loaded at boot time (in my opinion) if there wasn't a device tree specifying the need for it. I also see fairly SoC-specific drivers like "phy_fsl_imx8m_pcie" and "phy_fsl_imx8mq_usb" being loaded as modules after boot. Again, a good sign that the device tree is making it to the kernel.

In saying that, I am still trying to confirm definitively that the device tree is being properly loaded since that could be the main source of issues. I have confirmed (so far) that all of the required drivers exist as pre-compiled modules on the MicroOS file system, so the driver modules should be available. Based on the modules that are loaded (as noted above) and my experience with the GW7200, I believe that the mechanism for providing the device tree to Linux is functioning correctly. For the GW7200, the same method results in all of the USB and PCIe devices working just fine despite being compiled as loadable modules.

For example, I was talking to Gateworks about being able to access USB storage from their Type C USB interface, they said the following are required:

CONFIG_USB_CONN_GPIO (required so their Type C USB connector can role switch) CONFIG_USB_UAS (I was trying to use USB storage) CONFIG_PHY_FSL_IMX8MQ_USB CONFIG_USB_DWC3

If I do an "lsmod" after boot, I only see "phy_fsl_imx8mq_usb" loaded from this list of modules. I can manually use "modprobe" to load "usb_conn_gpio", "uas", and "dwc3", but that hasn't been enough to get things working.

One thing that I know is a bit tricky with the NXP i.MX8M Mini/Plus SoCs that are used on these boards is that various things have to be configured (GPIO, etc.) so that the USB and PCIe devices are even properly powered. So, this is one reason why I think there might be some sort of issue or interaction with the device tree that I need to work out.

It still could be a subtle kernel configuration difference between Gateworks and openSUSE's kernels--I am still wading through all of the differences. For the most part, the openSUSE kernel builds many things as modules while Gateworks compiles the drivers they want right into the kernel binary. The openSUSE kernel has many more drivers enabled than Gateworks (of course), but, for the most part, if Gateworks has a driver built into the kernel, there is a module for it in the openSUSE kernel. I have found a few minor differences that I need to work with Gateworks to see if they are significant or not--cases where openSUSE doesn't enable a driver or disables a feature in the kernel that Gateworks enables.

Anyway, if you have any hints on resolving these issues, I am open to suggestions. I will continue to work with Gateworks as well.

My feeling, though, is that MicroOS has the needed drivers built as modules and that the mechanisms for loading the modules based on the device tree are working. Clearly, though, something must be misconfigured related to the device tree (or the kernel configuration) that is preventing things from loading properly. The device tree I am using is "imx8mp-venice-gw74xx.dtb" that comes from the "dtb-freescale-6.4.12-1.1.aarch64" package.

...

...

2. In the short term, if I needed to use a custom kernel for this board with MicroOS, is there any straightforward method to use it with the usual transactional-update process? We already have custom application packages that we host in a repository that MicroOS uses for installing and updating our packages. Could we just include our custom kernel package in our repository somehow despite the fact that MicroOS pushes out its own kernel package? It seems like the kernel is part of the group of packages that MicroOS installs as its base OS, so it isn't clear to me that I can just replace the kernel from MicroOS with my own.

You can. MicroOS will honour any repo and packaging changes you make. Ideally you set up builds on OBS based on the latest distro kernel to get updates, but this only works if the patches are very simple such that they don't need to be rebased manually all the time.

I agree and that makes a lot of sense. The more specific questions are:

1. Do I just call my kernel package something slightly different from what MicroOS does so that it only gets updated when I publish my own kernel package or is there a lot more to do? I would use the openSUSE kernel build tools to make the package (I have tried this out before), but I can't remember if the name of the kernel shows up as something different so that zypper and the other tools recognize it as being different from the stock MicroOS/Tumbleweed packages.

2. Also, what dependencies does MicroOS and its packages have on the kernel package? In other words, do I remove the standard kernel package and then install my own or do both need to be installed simultaneously to resolve some dependencies? Ideally, I would build my kernel using the MicroOS kernel source and package so it should be a drop-in replacement, but I could see that there could be confusion if there are two potential kernel packages with similar version numbers.

3. When you do a transactional-update of the system, will the update process handle the various things that must be taken care of every time for the atomic updates to work properly? E.g., building initrd files, setting the kernel to boot to for GRUB, etc. Again, if I am using the openSUSE kernel sources and packaging tools, I would expect this to just work.

Anyway, I guess my concern is, how to make my customized build of the openSUSE kernel package coexist with those expected and provided by MicroOS?

By the way, I think my custom kernel (if I need to go that route) will mainly be a build using the openSUSE kernel package but with a custom kernel configuration. This expectation is based on the fact that Gateworks's builds of the 6.4 kernels have full access to all of the peripherals when using Ubuntu. Further, their documentation states that all of the drivers needed for the board are in the upstream Linux kernel as of Linux 6.4. I guess I will see how that goes.

Thanks again!

Paul

...

Cheers, Fabian

...

Sorry for the long email and thanks again for your help!

Paul