[Bug 838475] New: Querying any service status takes up a long time with the combination btrfs + systemd + journald
https://bugzilla.novell.com/show_bug.cgi?id=838475 https://bugzilla.novell.com/show_bug.cgi?id=838475#c0 Summary: Querying any service status takes up a long time with the combination btrfs + systemd + journald Classification: openSUSE Product: openSUSE 12.3 Version: Final Platform: x86-64 OS/Version: Other Status: NEW Severity: Normal Priority: P5 - None Component: Other AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: gnyers@suse.com QAContact: qa-bugs@suse.de CC: jeffm@suse.com, mge@suse.com Found By: --- Blocker: --- Consequence is that boot and shutdown of the system can take several minutes. Demo 1:: # systemd-analyze blame 106983ms ntp.service 16382ms NetworkManager.service 16185ms smpppd.service [...] Demo2:: # sync && echo 1 > /proc/sys/vm/drop_caches # time systemctl status ntp ntp.service - LSB: Network time protocol daemon (ntpd) Loaded: loaded (/etc/init.d/ntp) Active: active (running) since Wed, 2013-09-04 09:10:54 CEST; 8h ago Process: 1299 ExecStart=/etc/init.d/ntp start (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/ntp.service └ 1557 /usr/sbin/ntpd -p /var/run/ntp/ntpd.pid -g -u ntp:ntp -i /var/lib/ntp -c /etc/ntp.conf Sep 04 09:10:54 lupus.demo.lan ntpd[1557]: proto: precision = 0.396 usec Sep 04 09:10:54 lupus.demo.lan ntpd[1557]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16 Sep 04 09:10:54 lupus.demo.lan ntpd[1557]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123 Sep 04 09:10:54 lupus.demo.lan ntp[1299]: Starting network time protocol daemon (NTPD)..done Sep 04 09:10:54 lupus.demo.lan ntpd[1557]: Listen and drop on 1 v6wildcard :: UDP 123 Sep 04 09:10:54 lupus.demo.lan ntpd[1557]: Listen normally on 2 lo 127.0.01 UDP 123 Sep 04 09:10:54 lupus.demo.lan ntpd[1557]: Listen normally on 3 lo ::1 UDP 123 Sep 04 09:10:54 lupus.demo.lan ntpd[1557]: peers refreshed Sep 04 09:10:54 lupus.demo.lan ntpd[1557]: Listening on routing socket on fd #20 for interface updates Sep 04 09:10:54 lupus.demo.lan systemd[1]: Started LSB: Network time protocol daemon (ntpd). real 1m32.013s user 0m0.003s sys 0m0.396s # Above results are very similar for any other service. Config info: ------------ Please note that above results are being produced despite the capping of Journald's size to 200M:: # grep -v '^#' /etc/systemd/journald.conf [Journal] Compress=yes SystemMaxUse=200M When removing all Journald files (`rm -fr /var/log/journald/`) service status return is immediate, in other words, emptying the Journald DB makes the problem go away. The btrfs subvol layout:: # btrfs sub list / ID 256 top level 5 path tmp ID 258 top level 5 path opt ID 259 top level 5 path srv ID 260 top level 5 path var/crash ID 261 top level 5 path var/spool ID 262 top level 5 path var/log ID 263 top level 5 path var/tmp ID 264 top level 5 path home ID 268 top level 5 path .snapshots [...] -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c
FeiXiang Zhang
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c1
Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c3
Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c4
--- Comment #4 from Frederic Crozat
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c
Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c7
--- Comment #7 from Gábor Nyers
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c8
--- Comment #8 from Frederic Crozat
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c9
--- Comment #9 from Gábor Nyers
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c10
--- Comment #10 from Bernhard Wiedemann
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c
Alberto Planas Dominguez
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c11
--- Comment #11 from Gábor Nyers
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c12
--- Comment #12 from Gábor Nyers
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c13
--- Comment #13 from Frederic Crozat
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c16
--- Comment #16 from Gábor Nyers
A file with the 'C' attribute set will not be subject to copy-on- write updates. This flag is only supported on file systems which perform copy-on-write. (Note: For btrfs, the 'C' flag should be set on new or empty files. If it is set on a file which already has data blocks, it is undefined when the blocks assigned to the file will be fully stable. If the 'C' flag is set on a directory, it will have no effect on the directory, but new files created in that directory will the No_COW attribute.)
Would be interesting to know the reason... incomplete implementation or a more fundamental problem? Anyway, I tried and can confirm. Also, a way to shortcut the need to flag files individually with the "C" attribute would be to do this on the directory. All newly created files will inherit it. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c
Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c17
David Sterba
A file with the 'C' attribute set will not be subject to copy-on- write updates. This flag is only supported on file systems which perform copy-on-write. (Note: For btrfs, the 'C' flag should be set on new or empty files. If it is set on a file which already has data blocks, it is undefined when the blocks assigned to the file will be fully stable. If the 'C' flag is set on a directory, it will have no effect on the directory, but new files created in that directory will the No_COW attribute.)
Would be interesting to know the reason... incomplete implementation or a more fundamental problem? Anyway, I tried and can confirm.
It's a compromise to keep the implementation sane with the same guarantees. Making any existing file nocow is possible in theory, but the transition would need to be fully crash safe and keeping the intermediate status would need incompatible changes.
Also, a way to shortcut the need to flag files individually with the "C" attribute would be to do this on the directory. All newly created files will inherit it.
This should work for the journal files. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c18
--- Comment #18 from Gábor Nyers
(In reply to comment #16) [...]
It's a compromise to keep the implementation sane with the same guarantees. Making any existing file nocow is possible in theory, but the transition would need to be fully crash safe and keeping the intermediate status would need incompatible changes.
A reasonable compromise.
Also, a way to shortcut the need to flag files individually with the "C" attribute would be to do this on the directory. All newly created files will inherit it.
This should work for the journal files.
Well, that doesn't seems to be the case always. This is a brief quote from Jeff from a related mail conversation:
Setting nocow only changes the CoW policy for future overwrites once that file is the sole reference holder of an extent. [...]
This seems to suggest that the NoCoW attribute's current implementation can work in theory, but not very useful for any practical purposes. After all how to make sure that only the a NoCoW'd file(s) will be holding an extent? Care to confirm/deny? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c19
--- Comment #19 from Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c20
--- Comment #20 from Gábor Nyers
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c21
--- Comment #21 from Frederic Crozat
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c22
--- Comment #22 from Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c23
Matthias Eckermann
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c24
Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c25
--- Comment #25 from Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c26
--- Comment #26 from Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c28
--- Comment #28 from Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c29
--- Comment #29 from Gábor Nyers
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c30
--- Comment #30 from David Sterba
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c31
--- Comment #31 from David Sterba
Regarding the solution with the NoCoW flag, please see the quote from JeffM (comment #18), as I describe it there, testing NoCoW on my 12.3 system have not worked as expected.
I don't understand what's the concern in comment 18. Newly created files can be set NOCOW or inherit that flag from the directory, that's been mentioned already and that's what the proposed patch will ensure. Please be more specific what did not work for you on 12.3. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c32
--- Comment #32 from Gábor Nyers
I don't understand what's the concern in comment 18. Newly created files can be set NOCOW or inherit that flag from the directory, that's been mentioned already and that's what the proposed patch will ensure.
Please be more specific what did not work for you on 12.3.
Sure, let's give it another try. The point in comment #18 is that (at least in my case) setting the NoCoW attribute did not had an effect on the fragmentation and as I suspect (based on some tests with cloning a file and overwriting 1 clone) the CoW behavior of Btrfs. To be specific: I've had the '+C' attribute on the /var/log/journal/*/* directories and files for months now and the journal fragments like crazy: # cd /var/log/journal/3e9b548e87a6e6e3afed740e000003c5/ # filefrag * | sort -nk 2 | tail -10 user-1000.journal: 72 extents found user-1000@d7a2dfbcebfc44dca3cdc8f77e78260e-0000000000054e42-0004ecc95eda51ee.journal: 187 extents found user-1000@d7a2dfbcebfc44dca3cdc8f77e78260e-000000000005e101-0004ed1c2aa242d6.journal: 204 extents found user-1000@d7a2dfbcebfc44dca3cdc8f77e78260e-00000000000597c6-0004ed020f1b176a.journal: 223 extents found user-1000@d7a2dfbcebfc44dca3cdc8f77e78260e-000000000006273c-0004ed304ec0db9c.journal: 223 extents found system.journal: 560 extents found system@d7a2dfbcebfc44dca3cdc8f77e78260e-0000000000062618-0004ed30101d2b2c.journal: 2975 extents found system@d7a2dfbcebfc44dca3cdc8f77e78260e-0000000000054e13-0004ecc94e42a2ab.journal: 3015 extents found system@d7a2dfbcebfc44dca3cdc8f77e78260e-000000000005e0b2-0004ed1c0f9d9a8e.journal: 3077 extents found system@d7a2dfbcebfc44dca3cdc8f77e78260e-0000000000059773-0004ed01fec7bb9d.journal: 3096 extents found This level of fragmentation is just after a few days of having done a `btrfs file defrag /var/log/journal/3e9b548e87a6e6e3afed740e000003c5/* So I'd say that there is some empirical evidence suggesting that changing Btrfs NoCoW behavior is not trivial. I've raised this issue to Jeff in a private mail and I've quoted above statement from his response. Maybe he could elaborate more on this. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c33
--- Comment #33 from Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c
Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c35
Vojtech Pavlik
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c36
--- Comment #36 from Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c38
--- Comment #38 from Arvin Schnell
On openSUSE we have snapper by default. Snapper makes sure everything is snapshotted regularly and that means that any journald files will have their extents referenced by the snapshots, unless they're excluded through subvolumes.
For that reason YaST creates several subvolumes, e.g. /var/log. So snapper has nothing to do with this bugreport. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=838475
https://bugzilla.novell.com/show_bug.cgi?id=838475#c
Dr. Werner Fink
http://bugzilla.novell.com/show_bug.cgi?id=838475
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=838475
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=838475
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=838475
Swamp Workflow Management
http://bugzilla.novell.com/show_bug.cgi?id=838475
http://bugzilla.novell.com/show_bug.cgi?id=838475#c41
--- Comment #41 from Swamp Workflow Management
participants (1)
-
bugzilla_noreply@novell.com