Hack Week summary: Rust-based D-Installer CLI
Hi all, As a Hack Week project, Josef and I started to work on a Rust-based command line interface[1] for D-Installer. My primary purpose was to determine whether it is a good language for building this tool. Moreover, I wanted to put my (limited) knowledge of the language into practice. TL;DR; the experience was really positive. Although it is far from finished, we managed to build a minimal CLI that allows setting/getting configuration values (using YAML and JSON as output formats). You can find more details in the Running section[2] of the README. Now, I will comment on some aspects that I found interesting. I hope you enjoy the reading :-) # The main obstacle Funny enough, the main "problem" we found was unrelated to Rust. It was, instead, the gap between our new CLI design (+ the auto- installation specification) and the D-Bus API. To fill the gap, we decided to keep an in-memory representation of D-Installer's settings (see the Settings and related structs[3]) and implement a separate mechanism to read/write those settings (see the Store struct[4]). # Procedural macros We found out that to set or get those configuration values, we needed to determine the name of the method/function to call at runtime. That's trivial with Ruby, but not with Rust without writing one line per each possible function. So we implemented a proc-macro[5] that generates the code for us. Beware that these macros are different to C/C++ ones and, among other things, they are written in Rust and give you access to the AST. # Abstraction level Rust cannot offer the same abstraction level as Ruby, period. But it has an expressive type system and powerful tools for meta-programming if needed. So when writing this CLI, I felt quite comfortable with the level of abstraction. # Error messages are helpful Most of the time, compiler error messages make sense. Apart from an explanation, it usually gives you some hints to solve the issues. However, if the error implies lifetimes, generics and so on, it can be hard to understand what is happening behind the scenes. # Extensive documentation Apart from "the book"[6], the standard library documentation is rather complete. However, it can be intimidating if you do not know the basis. # Closing thoughts As said before, the experience was really positive and I enjoyed a lot working on this project. There were a few things that we wanted to try, like calling Rust code from Ruby or using the async support, but we ran out of time. My main question now is: should we use this as the base for our D- Installer CLI or should we go back to our Ruby-based implementation? Regards, Imo [1] https://hackweek.opensuse.org/22/projects/rewrite-the-d-installer [2] https://github.com/imobachgs/dinstaller-rs#running [3] https://github.com/imobachgs/dinstaller-rs/blob/update-readme/dinstaller-lib... [4] https://github.com/imobachgs/dinstaller-rs/blob/update-readme/dinstaller-lib... [5] https://github.com/imobachgs/dinstaller-rs/blob/update-readme/dinstaller-der... [6] https://doc.rust-lang.org/book/ [7] https://www.rust-lang.org/tools -- Imobach González Sosa SUSE Linux LLC
On 2/6/23 11:40, Imobach Gonzalez Sosa wrote:
Hi all,
As a Hack Week project, Josef and I started to work on a Rust-based command line interface[1] for D-Installer. My primary purpose was to determine whether it is a good language for building this tool. Moreover, I wanted to put my (limited) knowledge of the language into practice.
TL;DR; the experience was really positive. Although it is far from finished, we managed to build a minimal CLI that allows setting/getting configuration values (using YAML and JSON as output formats). You can find more details in the Running section[2] of the README.
Now, I will comment on some aspects that I found interesting. I hope you enjoy the reading :-)
# The main obstacle
Funny enough, the main "problem" we found was unrelated to Rust. It was, instead, the gap between our new CLI design (+ the auto- installation specification) and the D-Bus API. To fill the gap, we decided to keep an in-memory representation of D-Installer's settings (see the Settings and related structs[3]) and implement a separate mechanism to read/write those settings (see the Store struct[4]).
# Procedural macros
We found out that to set or get those configuration values, we needed to determine the name of the method/function to call at runtime. That's trivial with Ruby, but not with Rust without writing one line per each possible function. So we implemented a proc-macro[5] that generates the code for us. Beware that these macros are different to C/C++ ones and, among other things, they are written in Rust and give you access to the AST.
# Abstraction level
Rust cannot offer the same abstraction level as Ruby, period. But it has an expressive type system and powerful tools for meta-programming if needed. So when writing this CLI, I felt quite comfortable with the level of abstraction.
# Error messages are helpful
Most of the time, compiler error messages make sense. Apart from an explanation, it usually gives you some hints to solve the issues. However, if the error implies lifetimes, generics and so on, it can be hard to understand what is happening behind the scenes.
# Extensive documentation
Apart from "the book"[6], the standard library documentation is rather complete. However, it can be intimidating if you do not know the basis.
# Closing thoughts
As said before, the experience was really positive and I enjoyed a lot working on this project. There were a few things that we wanted to try, like calling Rust code from Ruby or using the async support, but we ran out of time.
My main question now is: should we use this as the base for our D- Installer CLI or should we go back to our Ruby-based implementation?
Regards, Imo
[1] https://hackweek.opensuse.org/22/projects/rewrite-the-d-installer [2] https://github.com/imobachgs/dinstaller-rs#running [3] https://github.com/imobachgs/dinstaller-rs/blob/update-readme/dinstaller-lib... [4] https://github.com/imobachgs/dinstaller-rs/blob/update-readme/dinstaller-lib... [5] https://github.com/imobachgs/dinstaller-rs/blob/update-readme/dinstaller-der... [6] https://doc.rust-lang.org/book/ [7] https://www.rust-lang.org/tools
Thanks for the summary! Very interesting, indeed. About deciding whether to continue with Rust or not, I would put several aspects on the balance: * Do we gain any use cases? Probably yes. For example, if the CLI is able to run commands remotely, then we don't need ruby deps in the system. * Is Rust a good option for other future migrations? Right now D-Installer uses a lot YaST logic, but we could start implementing new features in a separate Rust based backend. * In case of continuing with the Rust based CLI, more people in the team should start learning Rust. I think this is a positive thing, just commenting to make clear we don't want silos anymore. * Is there any other options to consider? Maybe we could do the exercise of implementing something similar with other modern language, for example Go, just to compare. * Having a team that speaks Ruby, is Rust the best migration? If not, are there good reasons to pay the effort? And for sure there are more things to consider. BTW, I am all for learning Rust. I am only expressing that officially migrating to another language is a difficult decision ;). Thanks a lot Imo and Josef. Great work! Regards, Iván -- José Iván López González YaST Team at SUSE LINUX GmbH IRC: jilopez
El lun, 06-02-2023 a las 12:14 +0000, José Iván López González escribió:
[..]
Thanks for the summary! Very interesting, indeed. About deciding whether to continue with Rust or not, I would put several aspects on the balance:
You are welcome!
* Do we gain any use cases? Probably yes. For example, if the CLI is able to run commands remotely, then we don't need ruby deps in the system.
+1
* Is Rust a good option for other future migrations? Right now D-Installer uses a lot YaST logic, but we could start implementing new features in a separate Rust based backend.
Yes, that's true. Moving some services to Rust might help to reduce disk space and memory consumption. But although D-Installer uses a fraction of the available YaST code, it does not look realistic to me to replace our full Ruby-based codebase (think about storage-ng). Actually, I am talking only about the CLI for a reason: it is a part of D-Installer which offers some space for research. And, if we fail, we can go back to a Ruby-based solution without much hassle. But it is true that, in the future, we might reimplement some services. About your question, I think yes, it is a good option ;-)
* In case of continuing with the Rust based CLI, more people in the team should start learning Rust. I think this is a positive thing, just commenting to make clear we don't want silos anymore.
Yes, sure!
* Is there any other options to consider? Maybe we could do the exercise of implementing something similar with other modern language, for example Go, just to compare.
I have my personal opinion on the subject, but let's keep that for another discussion :-) For me it is fine, but we should stop at some point :-)
* Having a team that speaks Ruby, is Rust the best migration? If not, are there good reasons to pay the effort?
That's the main point IMHO. Does it bring any advantage? I think yes (memory consumption, no runtime dependencies, etc.). But if that's not clear, we should stay as we are right now.
And for sure there are more things to consider. BTW, I am all for learning Rust. I am only expressing that officially migrating to another language is a difficult decision ;).
Yes, I think it deserves a longer discussion. But it is good that you brought those points.
Thanks a lot Imo and Josef. Great work!
Thanks! ;-) Regards, Imo -- Imobach González Sosa SUSE Linux LLC
On Mon, 6 Feb 2023 12:14:06 +0000 José Iván López González <jlopez@suse.de> wrote:
On 2/6/23 11:40, Imobach Gonzalez Sosa wrote:
Hi all, <snip>
Thanks for the summary! Very interesting, indeed. About deciding whether to continue with Rust or not, I would put several aspects on the balance:
* Do we gain any use cases? Probably yes. For example, if the CLI is able to run commands remotely, then we don't need ruby deps in the system.
Well, for me the best use case for CLI is having iguana that has only dinstaller backend container and static build rust CLI as the least memory consuming solution for something like RPi or where memory size matters. With ruby CLI it is a lot of dependencies which probably ends up in own container so much more space occupied.
* Is Rust a good option for other future migrations? Right now D-Installer uses a lot YaST logic, but we could start implementing new features in a separate Rust based backend.
Well, this really depends if we see the language as future. I for sure would not hesitate to write some dbus service in rust. Advantage is that it does not bring dependencies.
* In case of continuing with the Rust based CLI, more people in the team should start learning Rust. I think this is a positive thing, just commenting to make clear we don't want silos anymore.
Yeap, that is for sure and it is advantage of rust. It is popular language similar like ruby, so it is not problem to hire someone if needed to replace someone.
* Is there any other options to consider? Maybe we could do the exercise of implementing something similar with other modern language, for example Go, just to compare.
Well, let me compare it from my POV. Rust is for me good language for serious work where correctness matters ( I do not do any speed or memory comparison, but I expect rust to be a better or at least on par with ruby ). If I compare it to ruby I have to say, I get more often errors from compiler ( quite good one, with templating it is more challenging but definitively better then from templates in g++ ), but significantly less often resulting binary. So compiler catch for me quite lot issues I overlook or do not know in advance. So ruby is for me better with prototyping quickly, but for robust and less buggy result rust looks better for me. If I compare to C even with new standard rust is definitively more safe environment and easier to work with its memory model. If I compare to C++ I think that C++ can do with some libraries and new standard almost all what rust can do, but for me a big issue with C++ is its objects, which makes it hard to bind to other languages and swig is not answer for me. Also as smart pointers are not default, it is still possible to mess with memory. Comparison to go is the hardest for me, because I just touch go briefly and read just some intro to it. But overall I learn more quickly rust, but I have no direct goal for go program. And that reminds me one really nice advantage of rust for newcommers. Really great IDE support thats to LSP. I use vscode and I have to say that support is great. It shows all documentation, it shows all types even those dynamically interfered ( something like C++ auto ), debugger support and also formatter/hints why it does not build. So even if you are not familiar with method names it really smartly shows available ones including its documentation and parameters.
* Having a team that speaks Ruby, is Rust the best migration? If not, are there good reasons to pay the effort?
That is good question and sadly I do not know the best answer, but as said I think also ruby has its place. In ruby you can really quickly prototype what you want, do it on target system and play with it. With rust it is different purpose, but for me it is really good language if you have some POC that you want to transform to stable software. Overall I really enjoy this hackweek project and it was good experience. Josef
And for sure there are more things to consider. BTW, I am all for learning Rust. I am only expressing that officially migrating to another language is a difficult decision ;).
Thanks a lot Imo and Josef. Great work!
Regards,
Iván
Hi, I forgot to mention, that also during hackweek I see notice on ruby weekly about better rust and ruby integration and looks really good. Just see this blog post: https://bundler.io/blog/2023/01/31/rust-gem-skeleton.html Josef
El mié, 08-02-2023 a las 13:34 +0100, josef Reidinger escribió:
Hi,
Hi!
I forgot to mention, that also during hackweek I see notice on ruby weekly about better rust and ruby integration and looks really good. Just see this blog post: https://bundler.io/blog/2023/01/31/rust-gem-skeleton.html
It sounds interesting. I knew about Rutie[1], but I did not try it. It took my attention that they can work in both ways (Rust<->Ruby). Something from the learning & innovation time... :-) [1] https://github.com/danielpclark/rutie
Josef
Regards, Imo -- Imobach González Sosa SUSE Linux LLC
participants (3)
-
Imobach Gonzalez Sosa -
josef Reidinger -
José Iván López González