Cameron, Thorsten, and Richard, Thanks for the feedback and updates. They are much appreciated. I have one additional question, at the moment, that is somewhat related. I am targeting an Arm64 board that needs a custom kernel to fully support all of the peripherals. What is the right strategy to build and use a custom kernel in a MicroOS context? I plan to patch the openSUSE kernel and try to build openSUSE-like kernel RPMs from that source tree. I am hoping that approach would provide a more seamless approach from an integration point of view, leveraging the openSUSE installation process to automatically generate the initrd, update the grub configuration, etc. I still need to figure out where to place the updated device tree as well, but I think I can figure that out. Currently the system is passing a device tree to Grub through bootefi within U-Boot. Assuming I can get all of that to work, I would probably host the custom kernel in a separate local repository, install the new kernel packages, and disable kernel package updates until, hopefully, the mainline kernel supports the Arm64 SoC completely. If you have any suggestions or insights, they would be greatly appreciated. Thanks again! Paul On Thu, Jul 22, 2021 at 3:31 PM Richard Brown wrote:

On Thu, 2021-07-22 at 09:17 +0200, Thorsten Kukuk wrote:

...

...

I understand that this is probably not compatible with the "transactional-update" tool on the receiving end, but it might be a nice way to push updates to systems, taking advantage of the fact that only snapshots are being sent and, ideally, they would represent only the differences from previous snapshots (a natural way of handling the deltas between two file systems).

If "btrfs send" isn't the right mechanism for pushing updates, is there a better mechanism?

The idea with "btrfs send/receive" is not new. The biggest problem I see is, that /etc is part of the root filesystem (else it wouldn't be possible to do a rollback of the configuration files, too), and this directory contains host specific data, which you should never share with other uses. Like the unique machine-id.

Yeah I've been playing around with this and found the same conclusion. But then this is not exactly any different from regular transactional- update, which has to jump through some hoops to handle /etc properly.

I have a half-finished proof of concept script which, in theory at least, should follow the below workflow:

1. ssh from host to node 2. create new target snapshot for node 3. create new target /etc overlay for nodes new snapshot 4. btrfs send/recieve new rootfs from host to node 5. copy/move /etc from new node rootfs to new /etc overlay on node 6. mount users /etc overlay over new overlay

This would still be is an 'imperfect' system compared to a typical rpm based transactional-update, as it's impossible to make the users config available for the rpm to process and possibly update, but would still provide an arrangement where distro config can be updated and rolled back.

Things like machine-id would always be in the users /etc overlay so preserved between updates.

-- Richard Brown Linux Distribution Engineer - Future Technology Team

Phone +4991174053-361 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D-90409 Nuernberg (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer

Cameron, Thanks for the advice. I will take a look at OBS for the kernel. I did look at it and kiwi for building custom images. I am working on a project that would like to use MicroOS on the Gateworks Venice boards (https://www.gateworks.com/products/industrial-single-board-computers/imx8-si...) or the Gateworks Newport boards (https://www.gateworks.com/products/industrial-single-board-computers/octeon-...). We have copies of both families of boards. The Venice GW7200 board (NXP i.MX8M Mini ARMv8 64-bit 1.6GHz Quad Core) runs the MicroOS installer nicely and works generally with MicroOS. The only catch that I found was that U-Boot as installed on the device does not like GPT disk labels and so I had to manually partition the drive to use an MS-DOS disk label, imitating the default MicroOS partitioning scheme--I used the expert mode partitioning tool to make it happen. The biggest issues with the GW7200 are that the USB support and PCIe support are not working. In interacting with the helpful engineers at Gateworks, apparently NXP's power management kernel patches for the USB and PCIe subsystems are not in the upstream kernel yet, but the fixes may land soon (5.14 or 5.15?). So, this situation may resolve itself relatively soon. In the meantime, we are trying to demonstrate the feasibility of using MicroOS on the GW7200 and we need USB support in the short term--part of the desire to get a custom kernel built and installed. Gateworks provides the complete, patched kernel source on GitHub, so there shouldn't be any concerns about proprietary information, thankfully. NXP's and Gateworks' intentions, I believe, are to have all of the kernel source to be upstreamed, if at all possible. Gateworks does mention that they are working to support SLES on this family of devices, but they haven't announced official support yet. Interestingly, the Newport series of boards are actually supported by SLES (15.2), but the MicroOS installer was very unstable running on a GW6200 board. There were several kernel oops messages and I was never really able to get all of the way into the installer. The Gateworks' kernel source for these Octeon-TX-based boards is also open-source and available on GitHub, but I don't know much about the efforts to upstream missing features. In the short term, we might have to maintain the kernel for these devices if we use them with MicroOS. It is good to know that SUSE can provide support for custom kernels through service agreements. I will pass that information onto the rest of the team. Finally, I should note that Gateworks does provide pre-built kernels for use with various Linux userland images, but I thought I would try to understand the full process of building the kernels in openSUSE to ensure better MicroOS integration and management. Sorry for the long e-mail, Paul On Wed, Aug 4, 2021 at 12:15 PM Cameron Seader wrote:

Hi Paul, What is this Arm64 board that needs a custom kernel? What peripherals are you speaking of? Are they proprietary or something you can share?

Building a separate project tree in the OBS might be a good way to get you there in terms of building a custom kernel tree which you could base off of tumbleweed latest kernels. It really depends on what your trying to accomplish here in terms of a product outcome. You of course could use your own Open Build Service on-premise as well if your worried about Intellectual property rights.

SUSE does have services agreements available to help with building such custom kernels and building proper maintenance where you would get the backing and support of an extended developer team at SUSE. We would need to explore this in detail if you are interested.

You can of course go this on your own and build this out in your own home directory in the OBS where the repos are built automatically through the automation available there. I'm pretty sure you could easily make this kernel available in a MicroOS tree, but would have to be built in using the kiwi process in the OBS if building an ISO or Image.

At first glance this is a lot and you would be maintaining quite a lot on your own, or duplicating a lot of work. I'm sure most of us would agree that there might be other ways that would be better.

Maybe Richard and Thorsten have some other ideas.

If it were me sure I would start with a separate project in my home OBS and start building.

Cameron Seader Technology Strategist

SUSE +1 208 420 2167 cs@suse.com

www.suse.com

________________________________________ From: Paul Graham Sent: Friday, July 30, 2021 5:03 PM To: Richard Brown Cc: kubic@lists.opensuse.org Subject: Re: Update questions for MicroOS deployed on a private network

Cameron, Thorsten, and Richard,

Thanks for the feedback and updates. They are much appreciated.

I have one additional question, at the moment, that is somewhat related. I am targeting an Arm64 board that needs a custom kernel to fully support all of the peripherals. What is the right strategy to build and use a custom kernel in a MicroOS context?

I plan to patch the openSUSE kernel and try to build openSUSE-like kernel RPMs from that source tree. I am hoping that approach would provide a more seamless approach from an integration point of view, leveraging the openSUSE installation process to automatically generate the initrd, update the grub configuration, etc. I still need to figure out where to place the updated device tree as well, but I think I can figure that out. Currently the system is passing a device tree to Grub through bootefi within U-Boot.

Assuming I can get all of that to work, I would probably host the custom kernel in a separate local repository, install the new kernel packages, and disable kernel package updates until, hopefully, the mainline kernel supports the Arm64 SoC completely.

If you have any suggestions or insights, they would be greatly appreciated.

Thanks again!

Paul

On Thu, Jul 22, 2021 at 3:31 PM Richard Brown wrote:

...

On Thu, 2021-07-22 at 09:17 +0200, Thorsten Kukuk wrote:

...

...

I understand that this is probably not compatible with the "transactional-update" tool on the receiving end, but it might be a nice way to push updates to systems, taking advantage of the fact that only snapshots are being sent and, ideally, they would represent only the differences from previous snapshots (a natural way of handling the deltas between two file systems).

If "btrfs send" isn't the right mechanism for pushing updates, is there a better mechanism?

The idea with "btrfs send/receive" is not new. The biggest problem I see is, that /etc is part of the root filesystem (else it wouldn't be possible to do a rollback of the configuration files, too), and this directory contains host specific data, which you should never share with other uses. Like the unique machine-id.

Yeah I've been playing around with this and found the same conclusion. But then this is not exactly any different from regular transactional- update, which has to jump through some hoops to handle /etc properly.

I have a half-finished proof of concept script which, in theory at least, should follow the below workflow:

1. ssh from host to node 2. create new target snapshot for node 3. create new target /etc overlay for nodes new snapshot 4. btrfs send/recieve new rootfs from host to node 5. copy/move /etc from new node rootfs to new /etc overlay on node 6. mount users /etc overlay over new overlay

This would still be is an 'imperfect' system compared to a typical rpm based transactional-update, as it's impossible to make the users config available for the rpm to process and possibly update, but would still provide an arrangement where distro config can be updated and rolled back.

Things like machine-id would always be in the users /etc overlay so preserved between updates.

-- Richard Brown Linux Distribution Engineer - Future Technology Team

Phone +4991174053-361 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D-90409 Nuernberg (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer