"background idle" syncing of filesystems
Hello, I recently upgraded RAM on my Desktop computer (running openSUSE Leap 15.4) to 512GiB ECC DDR5. It's a great speed up, as I do a lot of disk I/O (home filesystem). But if it comes to syncing filesystes, it takes up to twenty minutes to sync them. Hence shutdown/reboot takes ages. (Hardware is a SAS Hardware RAID 1 (megaraid_sas with 2 2TB SAS HDDs)). is it possible to do a regular sync of the filesystem in idle I/O class, e.g. a crontab "ionice -c 3 sync", or would it cause performance issues or some sort of locks? I do not want to loose the speedup by async mounts and buffer/caching. But also worried about possible data loss, e.g. on AC fail. Thank you, regards Paul
On Tue, May 16, 2023 at 11:33 AM Paul Neuwirth via openSUSE Users <users@lists.opensuse.org> wrote:
Hello,
I recently upgraded RAM on my Desktop computer (running openSUSE Leap 15.4) to 512GiB ECC DDR5. It's a great speed up, as I do a lot of disk I/O (home filesystem). But if it comes to syncing filesystes, it takes up to twenty minutes to sync them. Hence shutdown/reboot takes ages. (Hardware is a SAS Hardware RAID 1 (megaraid_sas with 2 2TB SAS HDDs)).
is it possible to do a regular sync of the filesystem in idle I/O class, e.g. a crontab "ionice -c 3 sync", or would it cause performance issues or some sort of locks?
By default the kernel should flush old dirty pages (change more than 30 seconds ago), the kernel task to do it is called every 5 seconds. This is controlled by /proc/sys/vm/dirty_expire_centisecs and /proc/sys/vm/dirty_writeback_centisecs. If the system accumulates a lot of dirty pages, they are probably constantly changing.
I do not want to loose the speedup by async mounts and buffer/caching. But also worried about possible data loss, e.g. on AC fail.
Default is to start background flushing of dirty pages to disk when they grow over 10% of available memory. With 512GiB this means about 50G. You may consider lowering this limit so background flushing happens more often. See /proc/sys/vm/dirty_background_bytes and /proc/sys/vm/dirty_background_ratio. There are also global dirty_bytes and dirty_ratio which will force synchronous writes. If some program generates a lot of dirty data, it may be possible to run it in its own cgroup and restrict the total amount of available memory. The above vm sysctls should be applied proportionally to cgroup memory. I think it needs cgroup v2.
On Tuesday 2023-05-16 11:21, Andrei Borzenkov wrote:
Date: Tue, 16 May 2023 11:21:33 From: Andrei Borzenkov <arvidjaar@gmail.com> To: users@lists.opensuse.org Subject: Re: "background idle" syncing of filesystems
On Tue, May 16, 2023 at 11:33 AM Paul Neuwirth via openSUSE Users <users@lists.opensuse.org> wrote:
Hello,
I recently upgraded RAM on my Desktop computer (running openSUSE Leap 15.4) to 512GiB ECC DDR5. It's a great speed up, as I do a lot of disk I/O (home filesystem). But if it comes to syncing filesystes, it takes up to twenty minutes to sync them. Hence shutdown/reboot takes ages. (Hardware is a SAS Hardware RAID 1 (megaraid_sas with 2 2TB SAS HDDs)).
is it possible to do a regular sync of the filesystem in idle I/O class, e.g. a crontab "ionice -c 3 sync", or would it cause performance issues or some sort of locks?
By default the kernel should flush old dirty pages (change more than 30 seconds ago), the kernel task to do it is called every 5 seconds. This is controlled by /proc/sys/vm/dirty_expire_centisecs and /proc/sys/vm/dirty_writeback_centisecs. If the system accumulates a lot of dirty pages, they are probably constantly changing.
I do not want to loose the speedup by async mounts and buffer/caching. But also worried about possible data loss, e.g. on AC fail.
Default is to start background flushing of dirty pages to disk when they grow over 10% of available memory. With 512GiB this means about 50G. You may consider lowering this limit so background flushing happens more often. See /proc/sys/vm/dirty_background_bytes and /proc/sys/vm/dirty_background_ratio. There are also global dirty_bytes and dirty_ratio which will force synchronous writes.
If some program generates a lot of dirty data, it may be possible to run it in its own cgroup and restrict the total amount of available memory. The above vm sysctls should be applied proportionally to cgroup memory. I think it needs cgroup v2.
Thank you for that information, I thus monitored nr_dirty and nr_dirty_threshold in /proc/vmstat. the highest value for nr_dirty I got so far is as following: nr_dirty 7686 nr_dirty_threshold 23977156 it seems "dirty" data is being written quite regularly to disk. But why does "sync" (or a sysrq emergency sync) take ages then on shutdown? maybe I also need to monitor that over a longer period of time. currently a "sync" is finished in parts of a second. regards, paul
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 El 2023-05-16 a las 11:42 +0200, Paul Neuwirth via openSUSE Users escribió:
On Tuesday 2023-05-16 11:21, Andrei Borzenkov wrote:
On Tue, May 16, 2023 at 11:33 AM Paul Neuwirth via openSUSE Users <> wrote:
Hello,
...
Thank you for that information, I thus monitored nr_dirty and nr_dirty_threshold in /proc/vmstat. the highest value for nr_dirty I got so far is as following:
nr_dirty 7686 nr_dirty_threshold 23977156
it seems "dirty" data is being written quite regularly to disk. But why does "sync" (or a sysrq emergency sync) take ages then on shutdown?
maybe I also need to monitor that over a longer period of time. currently a "sync" is finished in parts of a second.
Look at the top part of top: top - 12:11:36 up 31 days, 12:29, 2 users, load average: 1,14, 1,07, 1,07 Tasks: 815 total, 2 running, 813 sleeping, 0 stopped, 0 zombie %Cpu(s): 10,5 us, 2,0 sy, 0,0 ni, 87,2 id, 0,3 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 65778600 total, 16972240 free, 32940940 used, 16763984 buff/cache KiB Swap: 10485760+total, 88519704 free, 16337892 used. 32837660 avail Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 10975 cer 20 0 7909248 4,723g 16952 59296 R 86,14 7,528 2822:45 Isolated Web Co 9988 cer 20 0 10,407g 1,291g 157136 1,225g S 8,581 2,058 3107:45 firefox 9260 cer 20 0 49512 38316 2052 136 S 4,950 0,058 1783:15 atop or free: cer@Telcontar:~> free -h total used free shared buff/cache available Mem: 62Gi 31Gi 16Gi 170Mi 15Gi 31Gi Swap: 99Gi 15Gi 84Gi cer@Telcontar:~> When I have not powered off or hibernated my machine for days, the "free" ram gets down to 1 gig or so, and the rest is on "buff/cache". You can see that this moment 25% my ram is there, and another 25% is free. Side note (unrelated): some web page is behaving itself very badly. I can use "top" to kill that "Isolated Web Co..." or open "about:performance" in a firefox tab to see what tab is the culprit. - -- Cheers, Carlos E. R. (from openSUSE 15.4 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCZGNYvRwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVIoEAn0lj9vEVEaJLxExAASix hDfHE7gpAKCPFmYGNTrs2vLJ+xBRefsXfTxXZQ== =Xw+6 -----END PGP SIGNATURE-----
On Tue, May 16, 2023 at 12:42 PM Paul Neuwirth <mail@paul-neuwirth.nl> wrote:
But why does "sync" (or a sysrq emergency sync) take ages then on shutdown?
Are you sure it is sync? What happens if you call the "sync" command (should wait until sync is completed) - does it take as much time? Do you see disk activity when waiting on shutdown?
On Tuesday 2023-05-16 12:51, Andrei Borzenkov wrote:
Date: Tue, 16 May 2023 12:51:24 From: Andrei Borzenkov <arvidjaar@gmail.com> To: users@lists.opensuse.org Subject: Re: "background idle" syncing of filesystems
On Tue, May 16, 2023 at 12:42 PM Paul Neuwirth <mail@paul-neuwirth.nl> wrote:
But why does "sync" (or a sysrq emergency sync) take ages then on shutdown?
Are you sure it is sync? What happens if you call the "sync" command (should wait until sync is completed) - does it take as much time? Do you see disk activity when waiting on shutdown?
sync currently finishes in less then a second, but system uptime is not quite long yet, and currently no high i/o activities, 33GiB buff/cache (at shutdown it's in the 400's). When I had those problems I ran some things like btrfs deduplication, btrfs defragmentation/compression, working on snapshots, etc. But at shutdown it's definitely the syncing. systemd/kernel/init or whoever is even trying to kill "sd_sync", I also tried sysrq emergency sync, never became the message "emergeny sync complete". last steps I see in the kernel debug output is trying to sync. ultimately it finishes and the pc is shutdown/rebooted. All the while the disk activity led of the pc is illuminated.
On 2023-05-16 13:04, Paul Neuwirth via openSUSE Users wrote:
On Tuesday 2023-05-16 12:51, Andrei Borzenkov wrote:
On Tue, May 16, 2023 at 12:42 PM Paul Neuwirth <> wrote:
But why does "sync" (or a sysrq emergency sync) take ages then on shutdown?
Are you sure it is sync? What happens if you call the "sync" command (should wait until sync is completed) - does it take as much time? Do you see disk activity when waiting on shutdown?
sync currently finishes in less then a second, but system uptime is not quite long yet, and currently no high i/o activities, 33GiB buff/cache (at shutdown it's in the 400's). When I had those problems I ran some things like btrfs deduplication, btrfs defragmentation/compression, working on snapshots, etc. But at shutdown it's definitely the syncing. systemd/kernel/init or whoever is even trying to kill "sd_sync", I also tried sysrq emergency sync, never became the message "emergeny sync complete". last steps I see in the kernel debug output is trying to sync. ultimately it finishes and the pc is shutdown/rebooted. All the while the disk activity led of the pc is illuminated.
Is it hard disk, or ssd? I had a similar problem some time ago, with only 8 GiB IIRC. Sync took ages. I pinpointed the issue to a single partition that held my local usenet proxy server. When I migrated that partition to ssd, the issue disappeared. Also affected by it being reiserfs. -- Cheers / Saludos, Carlos E. R. (from 15.4 x86_64 at Telcontar)
On Tuesday 2023-05-16 13:10, Carlos E. R. wrote:
Date: Tue, 16 May 2023 13:10:09 From: Carlos E. R. <robin.listas@telefonica.net> To: users@lists.opensuse.org Subject: Re: "background idle" syncing of filesystems
On 2023-05-16 13:04, Paul Neuwirth via openSUSE Users wrote:
On Tuesday 2023-05-16 12:51, Andrei Borzenkov wrote:
On Tue, May 16, 2023 at 12:42 PM Paul Neuwirth <> wrote:
But why does "sync" (or a sysrq emergency sync) take ages then on shutdown?
Are you sure it is sync? What happens if you call the "sync" command (should wait until sync is completed) - does it take as much time? Do you see disk activity when waiting on shutdown?
sync currently finishes in less then a second, but system uptime is not quite long yet, and currently no high i/o activities, 33GiB buff/cache (at shutdown it's in the 400's). When I had those problems I ran some things like btrfs deduplication, btrfs defragmentation/compression, working on snapshots, etc. But at shutdown it's definitely the syncing. systemd/kernel/init or whoever is even trying to kill "sd_sync", I also tried sysrq emergency sync, never became the message "emergeny sync complete". last steps I see in the kernel debug output is trying to sync. ultimately it finishes and the pc is shutdown/rebooted. All the while the disk activity led of the pc is illuminated.
Is it hard disk, or ssd? hard disk(s), it's a megaraid_sas sas hardware raid level 1. 2TB
I had a similar problem some time ago, with only 8 GiB IIRC. Sync took ages. I pinpointed the issue to a single partition that held my local usenet proxy server. When I migrated that partition to ssd, the issue disappeared. Also affected by it being reiserfs.
-- Cheers / Saludos,
Carlos E. R. (from 15.4 x86_64 at Telcontar)
On Tue, May 16, 2023 at 2:04 PM Paul Neuwirth <mail@paul-neuwirth.nl> wrote:
On Tuesday 2023-05-16 12:51, Andrei Borzenkov wrote:
Date: Tue, 16 May 2023 12:51:24 From: Andrei Borzenkov <arvidjaar@gmail.com> To: users@lists.opensuse.org Subject: Re: "background idle" syncing of filesystems
On Tue, May 16, 2023 at 12:42 PM Paul Neuwirth <mail@paul-neuwirth.nl> wrote:
But why does "sync" (or a sysrq emergency sync) take ages then on shutdown?
Are you sure it is sync? What happens if you call the "sync" command (should wait until sync is completed) - does it take as much time? Do you see disk activity when waiting on shutdown?
sync currently finishes in less then a second, but system uptime is not quite long yet, and currently no high i/o activities, 33GiB buff/cache (at shutdown it's in the 400's). When I had those problems I ran some
So if you try "sync" just before you are going to shut down your system - does it take so much time? There is still not much evidence that it is filesystem sync that slows things down. It could also be some (IO intensive) task that runs as part of shutdown.
things like btrfs deduplication, btrfs defragmentation/compression, working on snapshots, etc. But at shutdown it's definitely the syncing. systemd/kernel/init or whoever is even trying to kill "sd_sync", I also tried sysrq emergency sync, never became the message "emergeny sync complete". last steps I see in the kernel debug output is trying to sync. ultimately it finishes and the pc is shutdown/rebooted. All the while the disk activity led of the pc is illuminated.
Paul Neuwirth via openSUSE Users wrote:
(Hardware is a SAS Hardware RAID 1 (megaraid_sas with 2 2TB SAS HDDs)).
Wild guess: It tries to do an fstrim on the volume, and the megaraid_sas does not pass that through. I have the same issue with one here, that trim isn't supported.
On Wednesday 2023-05-17 15:01, Pit Suetterlin via openSUSE Users wrote:
Date: Wed, 17 May 2023 15:01:18 From: Pit Suetterlin via openSUSE Users <users@lists.opensuse.org> Reply-To: Pit Suetterlin <pit@astro.su.se> To: users@lists.opensuse.org Subject: Re: "background idle" syncing of filesystems
Paul Neuwirth via openSUSE Users wrote:
(Hardware is a SAS Hardware RAID 1 (megaraid_sas with 2 2TB SAS HDDs)).
Wild guess:
It tries to do an fstrim on the volume, and the megaraid_sas does not pass that through. I have the same issue with one here, that trim isn't supported.
interesting theory. any way to prove it? and if in doubt, how to disable? thank you and I could recreate the issue, after 6 days uptime with high I/O. from /proc/vmstat: nr_dirty 571064 nr_dirty_threshold 24894434 nr_dirty_background_threshold 12432019 from top: MiB Mem : 515896.2+total, 6047.250 free, 15834.02+used, 501647.4+buff/cache and finally: # date; sync; date Tue May 23 05:49:06 AM CEST 2023 Tue May 23 05:59:44 AM CEST 2023 -> syncing took > 10 minutes. regards, Paul Neuwirth
On 23.05.2023 07:02, Paul Neuwirth via openSUSE Users wrote:
and I could recreate the issue, after 6 days uptime with high I/O.
from /proc/vmstat: nr_dirty 571064
That is about 22GiB (4K pages)
nr_dirty_threshold 24894434 nr_dirty_background_threshold 12432019
from top: MiB Mem : 515896.2+total, 6047.250 free, 15834.02+used, 501647.4+buff/cache
and finally: # date; sync; date Tue May 23 05:49:06 AM CEST 2023 Tue May 23 05:59:44 AM CEST 2023
-> syncing took > 10 minutes.
This is appr. 32MiB/s. Assuming small random IO (which is also duplicated due to mirroring) I would not call it something extraordinary with two SATA disks. If this data is on btrfs, you have additional metadata which adds to IO randomness (and the amount of metadata could be quite large as well). So it is quite normal for your workload. The only way to mitigate it is to lower thresholds to force syncing of dirty pages earlier. What you observe is the typical buffer bloat. If you let data to accumulate you need to wait for data to drain.
On Tuesday 2023-05-23 06:35, Andrei Borzenkov wrote:
Date: Tue, 23 May 2023 06:35:45 From: Andrei Borzenkov <arvidjaar@gmail.com> To: users@lists.opensuse.org Subject: Re: "background idle" syncing of filesystems
On 23.05.2023 07:02, Paul Neuwirth via openSUSE Users wrote:
and I could recreate the issue, after 6 days uptime with high I/O.
from /proc/vmstat: nr_dirty 571064
That is about 22GiB (4K pages)
nr_dirty_threshold 24894434 nr_dirty_background_threshold 12432019
from top: MiB Mem : 515896.2+total, 6047.250 free, 15834.02+used, 501647.4+buff/cache
and finally: # date; sync; date Tue May 23 05:49:06 AM CEST 2023 Tue May 23 05:59:44 AM CEST 2023
-> syncing took > 10 minutes.
This is appr. 32MiB/s. Assuming small random IO (which is also duplicated due to mirroring) I would not call it something extraordinary with two SATA disks. If this data is on btrfs, you have additional metadata which adds to IO randomness (and the amount of metadata could be quite large as well).
So it is quite normal for your workload. The only way to mitigate it is to lower thresholds to force syncing of dirty pages earlier.
What you observe is the typical buffer bloat. If you let data to accumulate you need to wait for data to drain.
thanks for figuring this out. As part of my original post, would lowering /proc/sys/vm/dirty_background_bytes or /proc/sys/vm/dirty_background_ratio affect system performance noticeably? Paul Neuwirth
On Tue, May 23, 2023 at 8:53 AM Paul Neuwirth <mail@paul-neuwirth.nl> wrote:
On Tuesday 2023-05-23 06:35, Andrei Borzenkov wrote:
Date: Tue, 23 May 2023 06:35:45 From: Andrei Borzenkov <arvidjaar@gmail.com> To: users@lists.opensuse.org Subject: Re: "background idle" syncing of filesystems
On 23.05.2023 07:02, Paul Neuwirth via openSUSE Users wrote:
and I could recreate the issue, after 6 days uptime with high I/O.
from /proc/vmstat: nr_dirty 571064
That is about 22GiB (4K pages)
nr_dirty_threshold 24894434 nr_dirty_background_threshold 12432019
from top: MiB Mem : 515896.2+total, 6047.250 free, 15834.02+used, 501647.4+buff/cache
and finally: # date; sync; date Tue May 23 05:49:06 AM CEST 2023 Tue May 23 05:59:44 AM CEST 2023
-> syncing took > 10 minutes.
This is appr. 32MiB/s. Assuming small random IO (which is also duplicated due
Sorry, I miscalculated. It is 3.5MiB/s. It sounds small, but the usual sustained random IO rate for SATA disks is around 50 - 70 IOPS (IO operations per second), so it depends on IO size.
to mirroring) I would not call it something extraordinary with two SATA disks. If this data is on btrfs, you have additional metadata which adds to IO randomness (and the amount of metadata could be quite large as well).
So it is quite normal for your workload. The only way to mitigate it is to lower thresholds to force syncing of dirty pages earlier.
What you observe is the typical buffer bloat. If you let data to accumulate you need to wait for data to drain.
thanks for figuring this out. As part of my original post, would lowering /proc/sys/vm/dirty_background_bytes or /proc/sys/vm/dirty_background_ratio affect system performance noticeably?
Define "system performance". Yes, lowering the dirty background threshold may help to reduce sync time during shutdown. But you may suddenly find out that now drives are kept more busy and it impacts normal application performance.
On Tuesday 2023-05-23 09:18, Andrei Borzenkov wrote:
Date: Tue, 23 May 2023 09:18:19 From: Andrei Borzenkov <arvidjaar@gmail.com> To: users@lists.opensuse.org Subject: Re: "background idle" syncing of filesystems
On Tue, May 23, 2023 at 8:53 AM Paul Neuwirth <mail@paul-neuwirth.nl> wrote:
On Tuesday 2023-05-23 06:35, Andrei Borzenkov wrote:
Date: Tue, 23 May 2023 06:35:45 From: Andrei Borzenkov <arvidjaar@gmail.com> To: users@lists.opensuse.org Subject: Re: "background idle" syncing of filesystems
On 23.05.2023 07:02, Paul Neuwirth via openSUSE Users wrote:
and I could recreate the issue, after 6 days uptime with high I/O.
from /proc/vmstat: nr_dirty 571064
That is about 22GiB (4K pages)
nr_dirty_threshold 24894434 nr_dirty_background_threshold 12432019
from top: MiB Mem : 515896.2+total, 6047.250 free, 15834.02+used, 501647.4+buff/cache
and finally: # date; sync; date Tue May 23 05:49:06 AM CEST 2023 Tue May 23 05:59:44 AM CEST 2023
-> syncing took > 10 minutes.
This is appr. 32MiB/s. Assuming small random IO (which is also duplicated due
Sorry, I miscalculated. It is 3.5MiB/s. It sounds small, but the usual sustained random IO rate for SATA disks is around 50 - 70 IOPS (IO operations per second), so it depends on IO size.
maybe this represents my real problem. I triggered a sync and watched iostat. I/O Rate is around 1.2MB/s, between 40 and 46 tps (transactions per second?) Harddisks are SAS 2TB (SPL-3), IBM-XIV, ST2000NM0023 Controller 02:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2008 [Falcon] (rev 03) SMART data is fine, except for elevated temperature for one HDD (60°C) is that really what to expect? btw. I/O wait increased a lot during sync (which just finished after > 20 minutes). Some pplications got totally inresponsive, X froze for several minutes, load average went from ~1 to 35 (having 32 cpu). not nice :-/
to mirroring) I would not call it something extraordinary with two SATA disks. If this data is on btrfs, you have additional metadata which adds to IO randomness (and the amount of metadata could be quite large as well).
So it is quite normal for your workload. The only way to mitigate it is to lower thresholds to force syncing of dirty pages earlier.
What you observe is the typical buffer bloat. If you let data to accumulate you need to wait for data to drain.
thanks for figuring this out. As part of my original post, would lowering /proc/sys/vm/dirty_background_bytes or /proc/sys/vm/dirty_background_ratio affect system performance noticeably?
Define "system performance". Yes, lowering the dirty background threshold may help to reduce sync time during shutdown. But you may suddenly find out that now drives are kept more busy and it impacts normal application performance.
Paul Neuwirth PaNe Foto Paul Neuwirth Postfach 45 04 54 80904 MüNCHEN DEUTSCHLAND Fax: +49 89 35819624 https://www.swabian.net/ UST-IdNr. (VAT): DE314867715
On 2023-05-24 08:26, Paul Neuwirth via openSUSE Users wrote:
On Tuesday 2023-05-23 09:18, Andrei Borzenkov wrote:
Date: Tue, 23 May 2023 09:18:19 From: Andrei Borzenkov <arvidjaar@gmail.com> To: users@lists.opensuse.org Subject: Re: "background idle" syncing of filesystems
On Tue, May 23, 2023 at 8:53 AM Paul Neuwirth <mail@paul-neuwirth.nl> wrote:
On Tuesday 2023-05-23 06:35, Andrei Borzenkov wrote:
Date: Tue, 23 May 2023 06:35:45 From: Andrei Borzenkov <arvidjaar@gmail.com> To: users@lists.opensuse.org Subject: Re: "background idle" syncing of filesystems
On 23.05.2023 07:02, Paul Neuwirth via openSUSE Users wrote:
and I could recreate the issue, after 6 days uptime with high I/O.
from /proc/vmstat: nr_dirty 571064
That is about 22GiB (4K pages)
nr_dirty_threshold 24894434 nr_dirty_background_threshold 12432019
from top: MiB Mem : 515896.2+total, 6047.250 free, 15834.02+used, 501647.4+buff/cache
and finally: # date; sync; date Tue May 23 05:49:06 AM CEST 2023 Tue May 23 05:59:44 AM CEST 2023
-> syncing took > 10 minutes.
This is appr. 32MiB/s. Assuming small random IO (which is also duplicated due
Sorry, I miscalculated. It is 3.5MiB/s. It sounds small, but the usual sustained random IO rate for SATA disks is around 50 - 70 IOPS (IO operations per second), so it depends on IO size.
maybe this represents my real problem. I triggered a sync and watched iostat. I/O Rate is around 1.2MB/s, between 40 and 46 tps (transactions per second?) Harddisks are SAS 2TB (SPL-3), IBM-XIV, ST2000NM0023 Controller 02:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2008 [Falcon] (rev 03) SMART data is fine, except for elevated temperature for one HDD (60°C)
is that really what to expect?
btw. I/O wait increased a lot during sync (which just finished after > 20 minutes). Some pplications got totally inresponsive, X froze for several minutes, load average went from ~1 to 35 (having 32 cpu). not nice :-/
to mirroring) I would not call it something extraordinary with two SATA disks. If this data is on btrfs, you have additional metadata which adds to IO randomness (and the amount of metadata could be quite large as well).
So it is quite normal for your workload. The only way to mitigate it is to lower thresholds to force syncing of dirty pages earlier.
What you observe is the typical buffer bloat. If you let data to accumulate you need to wait for data to drain.
thanks for figuring this out. As part of my original post, would lowering /proc/sys/vm/dirty_background_bytes or /proc/sys/vm/dirty_background_ratio affect system performance noticeably?
Define "system performance". Yes, lowering the dirty background threshold may help to reduce sync time during shutdown. But you may suddenly find out that now drives are kept more busy and it impacts normal application performance.
Maybe what is needed is something that write to disk old cached entries, in background, so that they don't accumulate. -- Cheers / Saludos, Carlos E. R. (from 15.4 x86_64 at Telcontar)
participants (4)
-
Andrei Borzenkov -
Carlos E. R. -
Paul Neuwirth -
Pit Suetterlin