[yast-devel] Storage Redesign
Hi, my hackweek project was to evaluate using the boost graph library (BGL) in libstorage. For me the project was interesting and successful. I have documented it at https://github.com/aschnell/libstorage-bgl-eval/wiki. The code is also available there. So now I would like to redesign the storage part of YaST. Here are the main steps required: - Extend libstorage-bgl-eval to fully support disks, partitions and simply filesystems (probing, committing, testmodes, testsuite). - Try to make a compatibility layer. - Make the new API robust against API and ABI changes. - Export new API to Ruby (help would be good) for use by all YaST modules. - Get a simply installation working (two partitions, ext4 and swap). - Extend libstorage-bgl-eval to current libstorage functionality. Maybe drop a few features, e.g. loop-back devices (once used for encryption). - Make the new API suit the needs of all YaST modules (input from other developers required). If required add utility functions. - Make all code in YaST use new API (task for several developers). - Make expert partitioner use new features, e.g. use disks without partition table for filesystems. - Rewrite storage proposal (specification required!). - Finally drop target-map (as decided during workshop). - Drop compatibility layer. In the end yast2-storage would ideally only contain dialogs and no logic for other YaST modules (since all of that moved to libstorage). But if some utility functions are easier to implement in Ruby they can stay there. I assume that altogether the redesigning will take more than half a year. Anyway I think it is required to keep the code maintainable and be prepared for new feature requests. Some existing features request might already be implemented along the way (e.g. https://fate.suse.com/316251). ciao Arvin -- Arvin Schnell, <aschnell@suse.de> Senior Software Engineer, Research & Development SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Fri, 24 Oct 2014 14:53:06 +0200 Arvin Schnell <aschnell@suse.de> wrote:
Hi,
Hi Arvin, thanks for sharing, I post below some comments.
my hackweek project was to evaluate using the boost graph library (BGL) in libstorage. For me the project was interesting and successful. I have documented it at https://github.com/aschnell/libstorage-bgl-eval/wiki. The code is also available there.
So now I would like to redesign the storage part of YaST. Here are the main steps required:
- Extend libstorage-bgl-eval to fully support disks, partitions and simply filesystems (probing, committing, testmodes, testsuite).
- Try to make a compatibility layer.
yes, compatibility layer is needed to smooth migration.
- Make the new API robust against API and ABI changes.
Sounds good, question is how?
- Export new API to Ruby (help would be good) for use by all YaST modules.
I think it would be very cool if we have in ruby nice object oriented API. I can help you with ruby bindings. I think it is usually better to have combination of C bindings and ruby code that make usage in ruby smoother.
- Get a simply installation working (two partitions, ext4 and swap).
- Extend libstorage-bgl-eval to current libstorage functionality. Maybe drop a few features, e.g. loop-back devices (once used for encryption).
- Make the new API suit the needs of all YaST modules (input from other developers required). If required add utility functions.
I think that if only one module need specific functionality, then it should do it itself. ( like bootloader do setting pmbr flag for disks ). If more modules need it, then it make sense to have it in libstorage.
- Make all code in YaST use new API (task for several developers).
For this step we really need some storage setup factory, so e.g. in bootloader I can replace old mocks of target maps with new stuff and change code.
- Make expert partitioner use new features, e.g. use disks without partition table for filesystems.
- Rewrite storage proposal (specification required!).
- Finally drop target-map (as decided during workshop).
That will be great step especially together with object oriented API. Then we can have something like Disk.find( bios_id: "0x80").find_partition(mount_to: "/boot").sw_raid? in bootloader.
- Drop compatibility layer.
In the end yast2-storage would ideally only contain dialogs and no logic for other YaST modules (since all of that moved to libstorage). But if some utility functions are easier to implement in Ruby they can stay there.
I think utility can be in ruby bindings of libstorage if it is easier to do it there.
I assume that altogether the redesigning will take more than half a year. Anyway I think it is required to keep the code maintainable and be prepared for new feature requests. Some existing features request might already be implemented along the way (e.g. https://fate.suse.com/316251).
ciao Arvin
Josef -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Fri, Oct 31, 2014 at 08:00:20AM +0100, Josef Reidinger wrote:
my hackweek project was to evaluate using the boost graph library (BGL) in libstorage. For me the project was interesting and successful. I have documented it at https://github.com/aschnell/libstorage-bgl-eval/wiki. The code is also available there.
So now I would like to redesign the storage part of YaST. Here are the main steps required:
- Make the new API robust against API and ABI changes.
Sounds good, question is how?
At least export (via header files) only what really required, e.g. use the d-ptr pattern (e.g. http://qt-project.org/wiki/Dpointer).
- Export new API to Ruby (help would be good) for use by all YaST modules.
I think it would be very cool if we have in ruby nice object oriented API. I can help you with ruby bindings. I think it is usually better to have combination of C bindings and ruby code that make usage in ruby smoother.
I prefer to not write manual bindings. On the other hand I haven't checked what swig allows here.
- Make all code in YaST use new API (task for several developers).
For this step we really need some storage setup factory, so e.g. in bootloader I can replace old mocks of target maps with new stuff and change code.
The current libstorage already has support to not probe the system and instead read the setup from files. This will of course stay and even be extended (to cover all features, e.g. multipath).
- Finally drop target-map (as decided during workshop).
That will be great step especially together with object oriented API. Then we can have something like Disk.find( bios_id: "0x80").find_partition(mount_to: "/boot").sw_raid? in bootloader.
Well, I'm not used to such a "find" API so I cannot say whether it will like that way. Also so far libstorage does not know the bios-id, hwinfo provides that informations and it's "merged" in Ruby into the target-map. ciao Arvin -- Arvin Schnell, <aschnell@suse.de> Senior Software Engineer, Research & Development SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 10.11.2014 11:29, Arvin Schnell wrote:
- Finally drop target-map (as decided during workshop).
That will be great step especially together with object oriented API. Then we can have something like Disk.find( bios_id: "0x80").find_partition(mount_to: "/boot").sw_raid? in bootloader.
Well, I'm not used to such a "find" API so I cannot say whether it will like that way.
Also so far libstorage does not know the bios-id, hwinfo provides that informations and it's "merged" in Ruby into the target-map.
I'd like if the new API was designed to suit as much as possible to its users, such as Installer, Updater, Bootloader, Storage and maybe even some other SUSE products. Simple to use and understandable to as many developers as possible. This can't be achieved without open discussion with all the involved parties. And this is actually how I understand Arvin's e-mail and this whole thread. Thanks! Lukas -- Lukas Ocilka, Systems Management (Yast) Team Leader Cloud & Systems Management Department, SUSE Linux -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Fri, Nov 14, 2014 at 04:09:20PM +0100, Lukas Ocilka wrote:
On 10.11.2014 11:29, Arvin Schnell wrote:
- Finally drop target-map (as decided during workshop).
That will be great step especially together with object oriented API. Then we can have something like Disk.find( bios_id: "0x80").find_partition(mount_to: "/boot").sw_raid? in bootloader.
Well, I'm not used to such a "find" API so I cannot say whether it will like that way.
Also so far libstorage does not know the bios-id, hwinfo provides that informations and it's "merged" in Ruby into the target-map.
I'd like if the new API was designed to suit as much as possible to its users, such as Installer, Updater, Bootloader, Storage and maybe even some other SUSE products. Simple to use and understandable to as many developers as possible.
So far the only other project I know of using libstorage in some CIM provider. But even there I haven't heard of for some years. Apart from that I can remember ideas about using libstorage for kiwi but I cannot say whether those came from development or product management. Anyway, it would make the task more complicated - although also more interesting.
This can't be achieved without open discussion with all the involved parties. And this is actually how I understand Arvin's e-mail and this whole thread.
I think discussing a API should start with a proposal. Maybe I can present something in a few weeks. Regards, Arvin -- Arvin Schnell, <aschnell@suse.de> Senior Software Engineer, Research & Development SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 14.11.2014 16:49, Arvin Schnell wrote:
So far the only other project I know of using libstorage in some CIM provider. But even there I haven't heard of for some years.
"Another" libstorage users via Yast Storage: - Installation - Upgrade - Bootloader - Network (decision based on storage type) - Software (Windows partition, space calculations, USB-based media) - AutoYast - NFS Client - S390 - Country (Windows Partition) - REIPL - Multipath - ReaR - Kdump - Registration - Live Installer - AddOns (floppies) - Tune - iSCSI Client(?) - FCoE Client(?) These should be taken into consideration even when designing lower level API. All of them have a bit or completely different requirements and use-cases.
This can't be achieved without open discussion with all the involved parties. And this is actually how I understand Arvin's e-mail and this whole thread.
I think discussing a API should start with a proposal. Maybe I can present something in a few weeks.
That's one way, another way is getting the overview first and then making a proposal. Some might even not remember they use Storage for something. Anyway, making a proposal and checking whether it fits all needs is good as well. Thanks Lukas -- Lukas Ocilka, Systems Management (Yast) Team Leader Cloud & Systems Management Department, SUSE Linux -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Fri, Oct 24, 2014 at 02:53:06PM +0200, Arvin Schnell wrote:
Hi,
my hackweek project was to evaluate using the boost graph library (BGL) in libstorage. For me the project was interesting and successful. I have documented it at https://github.com/aschnell/libstorage-bgl-eval/wiki. The code is also available there.
Thanks for the write up! Some questions about the diagrams in the document: - why do nodes have numbers which are not unique? - what do [f] [l] [fl] mean? - the node colors seem to mean node type; the last graph should be much more colorful, shouldn't it? -- Martin Vidner, Cloud & Systems Management Team http://en.opensuse.org/User:Mvidner Kuracke oddeleni v restauraci je jako fekalni oddeleni v bazenu
On Wed, Nov 05, 2014 at 01:51:47PM +0100, Martin Vidner wrote:
On Fri, Oct 24, 2014 at 02:53:06PM +0200, Arvin Schnell wrote:
Hi,
my hackweek project was to evaluate using the boost graph library (BGL) in libstorage. For me the project was interesting and successful. I have documented it at https://github.com/aschnell/libstorage-bgl-eval/wiki. The code is also available there.
Thanks for the write up!
Some questions about the diagrams in the document:
- why do nodes have numbers which are not unique?
Each device objects has a unique id to identify it across graphs (the device name is not usable for that, see comment in source code). This id is displayed in the action graph, but since a single object may need more actions (e.g. create partition and set type) the id is not unique in that graph.
- what do [f] [l] [fl] mean?
First and last action for a device object.
- the node colors seem to mean node type; the last graph should be much more colorful, shouldn't it?
Well, currently the colors are: - green -> create (or format for filesystems) - red -> delele - blue -> modify - gray -> nop Sure more colors can be added ;) ciao Arvin -- Arvin Schnell, <aschnell@suse.de> Senior Software Engineer, Research & Development SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 10.11.2014 11:10, Arvin Schnell wrote:
Each device objects has a unique id to identify it across graphs (the device name is not usable for that, see comment in source code). This id is displayed in the action graph, but since a single object may need more actions (e.g. create partition and set type) the id is not unique in that graph.
First [f] and last [l] action for a device object.
Well, currently the colors are: - green -> create (or format for filesystems) - red -> delele - blue -> modify - gray -> nop
Thanks for explanation, it would be great if you could add this explanation to your document as mostly everyone who will read the document might have the very same questions, e.g., PM/PjM. Thanks & Bye Lukas -- Lukas Ocilka, Systems Management (Yast) Team Leader Cloud & Systems Management Department, SUSE Linux -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
participants (4)
-
Arvin Schnell -
Josef Reidinger -
Lukas Ocilka -
Martin Vidner