[yast-devel] YaST Installer Memory Consumption (the Ruby part)
Hi all, Ruby 2.1+ allows tracing object allocations, see more details in this article [1]. I enabled it and dumped the Ruby memory usage data in a SLES12-SP2 Beta3 installation. It is related to this Trello card [2] and this bug [3]. You can find more details and the dump in my Gist [4], here is a copy of the most interesting part: ## The Result A quick scan found out that the place which consumes most memory is /usr/share/YaST2/lib/installation/ssh_config_file.rb:45:STRING: count: 4, total size: 274056 which loads the SSH keys and configuration from the previous installation and needs about 270kB memory. The question is we can optimize it better, the SSH keys actually need to be stored somewhere as the target partition will be reformatted... The only optimization could be probably possible when the user selects to not copy the keys. In that case we could drop the loaded keys when the installation starts. But that's not the default so this improvement would actually help only in some cases... ## TODO You can collect a different statistics from the data, e.g. the place which creates the most objects, which methods, check the object "age" (the GC generation value), etc... [1] http://tmm1.net/ruby21-objspace/ [2] https://trello.com/c/QJ2PHxjs [3] https://bugzilla.suse.com/show_bug.cgi?id=974601 [4] https://gist.github.com/lslezak/6b99026b7ea9e163cdfd5a6d6d0aca6c -- Best Regards Ladislav Slezák Yast Developer ------------------------------------------------------------------------ SUSE LINUX, s.r.o. e-mail: lslezak@suse.cz Lihovarská 1060/12 tel: +420 284 028 960 190 00 Prague 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz/ -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 07/07/2016 07:38 PM, Ladislav Slezak wrote:
Hi all,
Ruby 2.1+ allows tracing object allocations, see more details in this article [1].
I enabled it and dumped the Ruby memory usage data in a SLES12-SP2 Beta3 installation.
It is related to this Trello card [2] and this bug [3].
You can find more details and the dump in my Gist [4], here is a copy of the most interesting part:
## The Result
A quick scan found out that the place which consumes most memory is
/usr/share/YaST2/lib/installation/ssh_config_file.rb:45:STRING: count: 4, total size: 274056
which loads the SSH keys and configuration from the previous installation and needs about 270kB memory.
The question is we can optimize it better, the SSH keys actually need to be stored somewhere as the target partition will be reformatted...
A quite trivial modification would be to change SshConfigFile#content and SshKey::KeyFile#content to store the content of the files in a temporary file and not in memory. It should be completely harmless, since that code is new, sane and covered by unit tests. https://github.com/yast/yast-installation/blob/fd94cd1477a4807a8fee6b50c95a9... https://github.com/yast/yast-installation/blob/fd94cd1477a4807a8fee6b50c95a9...
The only optimization could be probably possible when the user selects to not copy the keys. In that case we could drop the loaded keys when the installation starts. But that's not the default so this improvement would actually help only in some cases...
## TODO
You can collect a different statistics from the data, e.g. the place which creates the most objects, which methods, check the object "age" (the GC generation value), etc...
[1] http://tmm1.net/ruby21-objspace/ [2] https://trello.com/c/QJ2PHxjs [3] https://bugzilla.suse.com/show_bug.cgi?id=974601 [4] https://gist.github.com/lslezak/6b99026b7ea9e163cdfd5a6d6d0aca6c
-- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 07/11/2016 01:12 PM, Ancor Gonzalez Sosa wrote:
On 07/07/2016 07:38 PM, Ladislav Slezak wrote:
Hi all,
Ruby 2.1+ allows tracing object allocations, see more details in this article [1].
I enabled it and dumped the Ruby memory usage data in a SLES12-SP2 Beta3 installation.
It is related to this Trello card [2] and this bug [3].
You can find more details and the dump in my Gist [4], here is a copy of the most interesting part:
## The Result
A quick scan found out that the place which consumes most memory is
/usr/share/YaST2/lib/installation/ssh_config_file.rb:45:STRING: count: 4, total size: 274056
which loads the SSH keys and configuration from the previous installation and needs about 270kB memory.
The question is we can optimize it better, the SSH keys actually need to be stored somewhere as the target partition will be reformatted...
A quite trivial modification would be to change SshConfigFile#content and SshKey::KeyFile#content to store the content of the files in a temporary file and not in memory. It should be completely harmless, since that code is new, sane and covered by unit tests.
Imo just made me notice that a temporary file is memory as well. Never mind. I said nothing. ;-) Cheers.
https://github.com/yast/yast-installation/blob/fd94cd1477a4807a8fee6b50c95a9...
https://github.com/yast/yast-installation/blob/fd94cd1477a4807a8fee6b50c95a9...
The only optimization could be probably possible when the user selects to not copy the keys. In that case we could drop the loaded keys when the installation starts. But that's not the default so this improvement would actually help only in some cases...
## TODO
You can collect a different statistics from the data, e.g. the place which creates the most objects, which methods, check the object "age" (the GC generation value), etc...
[1] http://tmm1.net/ruby21-objspace/ [2] https://trello.com/c/QJ2PHxjs [3] https://bugzilla.suse.com/show_bug.cgi?id=974601 [4] https://gist.github.com/lslezak/6b99026b7ea9e163cdfd5a6d6d0aca6c
-- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 11.7.2016 v 13:17 Ancor Gonzalez Sosa napsal(a):
Imo just made me notice that a temporary file is memory as well. Never mind. I said nothing. ;-)
Yes, until the installer formats and mounts the /mnt target everything is in memory. The trick is that / is located in a RAM disk, i.e. saving to a file eats RAM... My idea is to compress the loaded data, in my test the ~270kB SSH data was gzipped to ~17kB. That's a pretty high compression ratio. 'xz' would be even a bit more efficient (~13kB). But gzip is much easier to use in Ruby, it is built-in: At read: require "zlib" compressed = Zlib::Deflate.deflate(read_data) [Run the installation] At write: write_data = Zlib::Inflate.inflate(compressed) Imo what do you think about it? -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Tuesday, July 12, 2016 3:08:21 PM WEST Ladislav Slezak wrote:
My idea is to compress the loaded data, in my test the ~270kB SSH data was gzipped to ~17kB. That's a pretty high compression ratio.
Nice.
'xz' would be even a bit more efficient (~13kB). But gzip is much easier to use in Ruby, it is built-in:
That's a plus.
At read:
require "zlib" compressed = Zlib::Deflate.deflate(read_data)
[Run the installation]
At write:
write_data = Zlib::Inflate.inflate(compressed)
Imo what do you think about it?
It sounds good. But it will depend in how much memory is used to make the compression (as you said in a status call IIRC). Regards, Imo -- Imobach González Sosa YaST team at SUSE LINUX GmbH
Dne 13.7.2016 v 12:16 Imobach González Sosa napsal(a):
It sounds good. But it will depend in how much memory is used to make the compression (as you said in a status call IIRC).
My comment was mainly about the `xz` compression tool, according to this [1] document it takes roughly 10 times more RAM than gzip. But the Zlib Ruby compression uses the gzip algorithm so it should be OK regarding to the memory consumption. More over the compression would happend quite early in the installation workflow, at that time there should be still enough memory. The most memory critical part is usually when starting the installation (during partitioning and package installation). I did some other space optimizations and Steffen removed duplicates from inst-sys which saved several megabytes of RAM so this SSH config compression is not that important anymore. But as I wrote before it's really trivial to implement it so I'd look at it again later... Lada [1] http://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_... -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Mon, Jul 11, 2016 at 01:17:54PM +0200, Ancor Gonzalez Sosa wrote:
On 07/11/2016 01:12 PM, Ancor Gonzalez Sosa wrote:
On 07/07/2016 07:38 PM, Ladislav Slezak wrote:
A quick scan found out that the place which consumes most memory is
/usr/share/YaST2/lib/installation/ssh_config_file.rb:45:STRING: count: 4, total size: 274056
which loads the SSH keys and configuration from the previous installation and needs about 270kB memory.
Is that the largest allocation, or the largest persistent (leaked) allocation? IMHO we do not need to care if the memory is freed soon enough.
The question is we can optimize it better, the SSH keys actually need to be stored somewhere as the target partition will be reformatted...
A quite trivial modification would be to change SshConfigFile#content and SshKey::KeyFile#content to store the content of the files in a temporary file and not in memory. It should be completely harmless, since that code is new, sane and covered by unit tests.
Imo just made me notice that a temporary file is memory as well. Never mind. I said nothing. ;-)
But a temporary file can be deleted and the memory reused (I hope). Which may be easier than convincing Ruby to let go of all the references. -- Martin Vidner, YaST Team http://en.opensuse.org/User:Mvidner Kuracke oddeleni v restauraci je jako fekalni oddeleni v bazenu
Dne 27.7.2016 v 16:06 Martin Vidner napsal(a):
which loads the SSH keys and configuration from the previous installation and needs about 270kB memory.
Is that the largest allocation, or the largest persistent (leaked) allocation? IMHO we do not need to care if the memory is freed soon enough.
It is not the largest allocation, it is just a complete memory dump in a specific point - just before starting the installation, at the installation confirmation dialog. To find the largest allocation you would need to add some code. And that code could potentially change the memory allocation, remember Heisenberg... AFAIK all information collected at this point is kept until the end of the installation. (Maybe something is dropped later in the inst_finish step but that does not matter, the most memory critical steps are partitioning and package installation.)
But a temporary file can be deleted and the memory reused (I hope). Which may be easier than convincing Ruby to let go of all the references.
Actually yes, but it depends how much is the Ruby GC aggressive in freeing unused memory... Theoretically the space occupied by a temporary file should be reused faster but I'm not sure if the potential gain is worth the effort. And we already did some memory optimizations... -- Ladislav Slezák YaST Developer lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
participants (4)
-
Ancor Gonzalez Sosa -
Imobach González Sosa -
Ladislav Slezak -
Martin Vidner