Hi, On Wed, Jul 21, Paul Graham wrote:

Hi,

I had a couple of questions that I was hoping someone could give me some guidance or opinions on.

First, I have a use case where I would like to run some embedded systems with MicroOS on a private network. Do you have any guidance on how I should mirror various repositories so I can update those machines regularly while they are on the private network?

Cameron gave you already some hints. Since MicroOS is using the Tumbleweed repositories, which are really big, there is another way: - Download the new ISO image - loopback mount it - Export it via http/https/ftp/nfs/... You can automate the download of the image and the setup easily with a shell script.

...

From what I can tell, the repos that are being used to perform software installs and updates can be found in /etc/zypp/repos.d. If I had mirrors for the repositories listed in that directory and updated

I have run across tools like Pulp (https://pulpproject.org/) that might help in creating the mirrors, but I haven't tried it out yet. the .repo files in /etc/zypp/repos.d to point to my mirrors, do I need to do anything else? Any pointers to documentation or other suggestions are welcomed.

Should be enough.

The second question is related. If I wanted to push updates to the individual systems rather than have them pull updates from repository mirrors, would it be possible to use a single "golden" system for managing the configuration for a collection of identical systems and then use "btrfs send" to update that collection of systems with updated software for the OS? In other words, I could see how you might perform an update on the "golden" system and then use the new snapshot that was created by the "transactional-update" tool in a "btrfs send" operation to push the changes to other systems.

I understand that this is probably not compatible with the "transactional-update" tool on the receiving end, but it might be a nice way to push updates to systems, taking advantage of the fact that only snapshots are being sent and, ideally, they would represent only the differences from previous snapshots (a natural way of handling the deltas between two file systems).

If "btrfs send" isn't the right mechanism for pushing updates, is there a better mechanism?

The idea with "btrfs send/receive" is not new. The biggest problem I see is, that /etc is part of the root filesystem (else it wouldn't be possible to do a rollback of the configuration files, too), and this directory contains host specific data, which you should never share with other uses. Like the unique machine-id. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg)

Cameron, Thorsten, and Richard, Thanks for the feedback and updates. They are much appreciated. I have one additional question, at the moment, that is somewhat related. I am targeting an Arm64 board that needs a custom kernel to fully support all of the peripherals. What is the right strategy to build and use a custom kernel in a MicroOS context? I plan to patch the openSUSE kernel and try to build openSUSE-like kernel RPMs from that source tree. I am hoping that approach would provide a more seamless approach from an integration point of view, leveraging the openSUSE installation process to automatically generate the initrd, update the grub configuration, etc. I still need to figure out where to place the updated device tree as well, but I think I can figure that out. Currently the system is passing a device tree to Grub through bootefi within U-Boot. Assuming I can get all of that to work, I would probably host the custom kernel in a separate local repository, install the new kernel packages, and disable kernel package updates until, hopefully, the mainline kernel supports the Arm64 SoC completely. If you have any suggestions or insights, they would be greatly appreciated. Thanks again! Paul On Thu, Jul 22, 2021 at 3:31 PM Richard Brown wrote:

On Thu, 2021-07-22 at 09:17 +0200, Thorsten Kukuk wrote:

...

...

I understand that this is probably not compatible with the "transactional-update" tool on the receiving end, but it might be a nice way to push updates to systems, taking advantage of the fact that only snapshots are being sent and, ideally, they would represent only the differences from previous snapshots (a natural way of handling the deltas between two file systems).

If "btrfs send" isn't the right mechanism for pushing updates, is there a better mechanism?

The idea with "btrfs send/receive" is not new. The biggest problem I see is, that /etc is part of the root filesystem (else it wouldn't be possible to do a rollback of the configuration files, too), and this directory contains host specific data, which you should never share with other uses. Like the unique machine-id.

Yeah I've been playing around with this and found the same conclusion. But then this is not exactly any different from regular transactional- update, which has to jump through some hoops to handle /etc properly.

I have a half-finished proof of concept script which, in theory at least, should follow the below workflow:

1. ssh from host to node 2. create new target snapshot for node 3. create new target /etc overlay for nodes new snapshot 4. btrfs send/recieve new rootfs from host to node 5. copy/move /etc from new node rootfs to new /etc overlay on node 6. mount users /etc overlay over new overlay

This would still be is an 'imperfect' system compared to a typical rpm based transactional-update, as it's impossible to make the users config available for the rpm to process and possibly update, but would still provide an arrangement where distro config can be updated and rolled back.

Things like machine-id would always be in the users /etc overlay so preserved between updates.

-- Richard Brown Linux Distribution Engineer - Future Technology Team

Phone +4991174053-361 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D-90409 Nuernberg (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer

Cameron, Thanks for the advice. I will take a look at OBS for the kernel. I did look at it and kiwi for building custom images. I am working on a project that would like to use MicroOS on the Gateworks Venice boards (https://www.gateworks.com/products/industrial-single-board-computers/imx8-si...) or the Gateworks Newport boards (https://www.gateworks.com/products/industrial-single-board-computers/octeon-...). We have copies of both families of boards. The Venice GW7200 board (NXP i.MX8M Mini ARMv8 64-bit 1.6GHz Quad Core) runs the MicroOS installer nicely and works generally with MicroOS. The only catch that I found was that U-Boot as installed on the device does not like GPT disk labels and so I had to manually partition the drive to use an MS-DOS disk label, imitating the default MicroOS partitioning scheme--I used the expert mode partitioning tool to make it happen. The biggest issues with the GW7200 are that the USB support and PCIe support are not working. In interacting with the helpful engineers at Gateworks, apparently NXP's power management kernel patches for the USB and PCIe subsystems are not in the upstream kernel yet, but the fixes may land soon (5.14 or 5.15?). So, this situation may resolve itself relatively soon. In the meantime, we are trying to demonstrate the feasibility of using MicroOS on the GW7200 and we need USB support in the short term--part of the desire to get a custom kernel built and installed. Gateworks provides the complete, patched kernel source on GitHub, so there shouldn't be any concerns about proprietary information, thankfully. NXP's and Gateworks' intentions, I believe, are to have all of the kernel source to be upstreamed, if at all possible. Gateworks does mention that they are working to support SLES on this family of devices, but they haven't announced official support yet. Interestingly, the Newport series of boards are actually supported by SLES (15.2), but the MicroOS installer was very unstable running on a GW6200 board. There were several kernel oops messages and I was never really able to get all of the way into the installer. The Gateworks' kernel source for these Octeon-TX-based boards is also open-source and available on GitHub, but I don't know much about the efforts to upstream missing features. In the short term, we might have to maintain the kernel for these devices if we use them with MicroOS. It is good to know that SUSE can provide support for custom kernels through service agreements. I will pass that information onto the rest of the team. Finally, I should note that Gateworks does provide pre-built kernels for use with various Linux userland images, but I thought I would try to understand the full process of building the kernels in openSUSE to ensure better MicroOS integration and management. Sorry for the long e-mail, Paul On Wed, Aug 4, 2021 at 12:15 PM Cameron Seader wrote:

Hi Paul, What is this Arm64 board that needs a custom kernel? What peripherals are you speaking of? Are they proprietary or something you can share?

Building a separate project tree in the OBS might be a good way to get you there in terms of building a custom kernel tree which you could base off of tumbleweed latest kernels. It really depends on what your trying to accomplish here in terms of a product outcome. You of course could use your own Open Build Service on-premise as well if your worried about Intellectual property rights.

SUSE does have services agreements available to help with building such custom kernels and building proper maintenance where you would get the backing and support of an extended developer team at SUSE. We would need to explore this in detail if you are interested.

You can of course go this on your own and build this out in your own home directory in the OBS where the repos are built automatically through the automation available there. I'm pretty sure you could easily make this kernel available in a MicroOS tree, but would have to be built in using the kiwi process in the OBS if building an ISO or Image.

At first glance this is a lot and you would be maintaining quite a lot on your own, or duplicating a lot of work. I'm sure most of us would agree that there might be other ways that would be better.

Maybe Richard and Thorsten have some other ideas.

If it were me sure I would start with a separate project in my home OBS and start building.

Cameron Seader Technology Strategist

SUSE +1 208 420 2167 cs@suse.com

www.suse.com

________________________________________ From: Paul Graham Sent: Friday, July 30, 2021 5:03 PM To: Richard Brown Cc: kubic@lists.opensuse.org Subject: Re: Update questions for MicroOS deployed on a private network

Cameron, Thorsten, and Richard,

Thanks for the feedback and updates. They are much appreciated.

I have one additional question, at the moment, that is somewhat related. I am targeting an Arm64 board that needs a custom kernel to fully support all of the peripherals. What is the right strategy to build and use a custom kernel in a MicroOS context?

I plan to patch the openSUSE kernel and try to build openSUSE-like kernel RPMs from that source tree. I am hoping that approach would provide a more seamless approach from an integration point of view, leveraging the openSUSE installation process to automatically generate the initrd, update the grub configuration, etc. I still need to figure out where to place the updated device tree as well, but I think I can figure that out. Currently the system is passing a device tree to Grub through bootefi within U-Boot.

Assuming I can get all of that to work, I would probably host the custom kernel in a separate local repository, install the new kernel packages, and disable kernel package updates until, hopefully, the mainline kernel supports the Arm64 SoC completely.

If you have any suggestions or insights, they would be greatly appreciated.

Thanks again!

Paul

On Thu, Jul 22, 2021 at 3:31 PM Richard Brown wrote:

...

On Thu, 2021-07-22 at 09:17 +0200, Thorsten Kukuk wrote:

...

...

I understand that this is probably not compatible with the "transactional-update" tool on the receiving end, but it might be a nice way to push updates to systems, taking advantage of the fact that only snapshots are being sent and, ideally, they would represent only the differences from previous snapshots (a natural way of handling the deltas between two file systems).

If "btrfs send" isn't the right mechanism for pushing updates, is there a better mechanism?

The idea with "btrfs send/receive" is not new. The biggest problem I see is, that /etc is part of the root filesystem (else it wouldn't be possible to do a rollback of the configuration files, too), and this directory contains host specific data, which you should never share with other uses. Like the unique machine-id.

Yeah I've been playing around with this and found the same conclusion. But then this is not exactly any different from regular transactional- update, which has to jump through some hoops to handle /etc properly.

I have a half-finished proof of concept script which, in theory at least, should follow the below workflow:

1. ssh from host to node 2. create new target snapshot for node 3. create new target /etc overlay for nodes new snapshot 4. btrfs send/recieve new rootfs from host to node 5. copy/move /etc from new node rootfs to new /etc overlay on node 6. mount users /etc overlay over new overlay

This would still be is an 'imperfect' system compared to a typical rpm based transactional-update, as it's impossible to make the users config available for the rpm to process and possibly update, but would still provide an arrangement where distro config can be updated and rolled back.

Things like machine-id would always be in the users /etc overlay so preserved between updates.

-- Richard Brown Linux Distribution Engineer - Future Technology Team

Phone +4991174053-361 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D-90409 Nuernberg (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer