[opensuse-factory] limited write cycles on filesystem media
We know popular modern media has limited write expectancy. The numbers are high and climbing as technology improves, but are nevertheless finite. Why are we habitually wasting them? Examples: Every installation, and upgrade, of netcfg, though often as few as 5 bytes, is writing a 4096 byte hostname/HOSTNAME sector (or possibly more) to the media, though no change to it/them has been made. During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup. Many files in /etc/sysconfig get written during updating, yet remain unchanged from their backups. This wastefulness is not common to other distros I have familiarity with. Why is it happening in openSUSE? I did create a FATE for this. Today it reached 30 months of age. It has a score of 4 and no comments. https://features.opensuse.org/313803 -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Jan 19, 2015 at 11:53 PM, Felix Miata <mrmazda@earthlink.net> wrote:
Do you do zypper dup often enough to worry about this? SSD media, which I assume you're worrying about, buffer and batch writes internally so unless some of those writes explicitly fsync, writing 5 bytes in 100 files or 5000 bytes at once shouldn't matter much. Basically, count how many bytes (blocks?) in total are written to disk while updating, and unless you can significantly decrease that number, which files get touched matter less. I'd worry more (way more) about logging. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Claudio Freire composed on 2015-01-20 00:29 (UTC-0300):
Felix Miata wrote:
During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup.
Many files in /etc/sysconfig get written during updating, yet remain unchanged from their backups.
This wastefulness is not common to other distros I have familiarity with. Why is it happening in openSUSE?
Do you do zypper dup often enough to worry about this?
There's more than one issue here. Did you read https://features.opensuse.org/313803 ever? Maybe it is trivial write volume, but there's the principle involved of wastefulness that applies not just in writing at update time, but also backup/restore processes.
I don't understand why you wrote this. I'm not at all concerned about efficiency here except as writing any sector at all is unnecessary. The file was not changed, so why the write?
Again I don't get it. Writes are slower I/O than reads. In the case of file not changed, why any write at all?
I'd worry more (way more) about logging.
A bigger can of worms to be sure, but little problems can batch up enough to break the camel's back. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Jan 20, 2015 at 12:46 AM, Felix Miata <mrmazda@earthlink.net> wrote:
It's irrelevant. That doesn't reference write cycles, and the post was about write cycles. I addressed the post (write cycles), and not the fate itself. In fact, it's not even clear doing what the fate indicates would improve write cycle efficiency, it may even make it worse (say, if the update script merely left the new file in place, and updated the timestamps on the old files repeatedly causing more metadata flushes).
Because batched writes get written in one flash sector (usually 2MB), and can contain more than one filesystem block. Writes, even random writes, on a block device backed by flash usually results in one single write cycle on one block. That's wear leveling at work.
If you write one block at all, the remaining 511 will usually be for free. So, why make a huge effort to avoid writing a few blocks if it's not going to report any benefit?
It should all start from the lowest hanging fruit. It makes little sense to go for the highest first. For another example, new installations should use relatime by default. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Claudio Freire composed on 2015-01-20 00:58 (UTC-0300):
There's still communication failure here. My interest is in why there is any write at all when file content before and after update process is identical. Maybe fstab makes the better example, since the stub of it in some rpm is obviously not going to work as-is on an already installed and configured installation. It too is a small file, so RAM should be sufficient for update process to work whatever magic it attempts.
Because batched writes get written in one flash sector (usually 2MB), and can contain more than one filesystem block.
I don't understand why this would be relevant to my question. One single sector file could be the difference between 2 batches and 1 batch.
Again I don't get it. Writes are slower I/O than reads. In the case of file not changed, why any write at all?
If you write one block at all, the remaining 511 will usually be for free.
So, why make a huge effort to avoid writing a few blocks if it's not going to report any benefit?
Huge effort? A few here, a few there. Maybe it doesn't seem so with electrons, but it does consume energy to make them move. Waste not, want not.
It should all start from the lowest hanging fruit.
It makes little sense to go for the highest first.
I only ever thought about $SUBJECT as a result of remembering the FATE. Here is a possibility of hitting multiple targets hit with one shot, instituting policy that induces less waste.
For another example, new installations should use relatime by default.
How that is better than noatime, which I started using long before ever hearing about relatime, I never figured out. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2015-01-20 05:35, Felix Miata wrote:
Claudio Freire composed on 2015-01-20 00:58 (UTC-0300):
He refers to the programming effort needed to change this.
If it is a programming practice, you can not influence programmers much. And you'd have to find exactly why this is happening, case by case. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On Tue, Jan 20, 2015 at 1:35 AM, Felix Miata <mrmazda@earthlink.net> wrote:
I imagine the rewriting will happen anyway unless a program is carefully crafted not to write to disk. I don't know the culprit, but I've programmed enough to get a rough idea of how such an update might work. diff, for example, needs file inputs. So you need to create a file in order to compare against the original. The file cannot be created in /tmp, that would be a security hole (/tmp can be written by anyone). If you created an /etc/fstab.new and diffed it, you may hope the kernel will cache the new file in memory and absorb the write followed by the unlink, but it's not entirely clear whether something like that would work when coded into the update procedure. Updating fstab won't be the only thing in that procedure, and the write cache could very well get flushed before you unlink. So, it *is* tricky coming up with a solution to that very specific problem without introducing either security issues or more unintended writes. On Tue, Jan 20, 2015 at 7:18 AM, Stanislav Baiduzhyi <baiduzhyi.devel@gmail.com> wrote:
I did report that for Firefox, it didn't get much attention. It's a performance killer too, since they call fsync every second as well on a timestamp file. Lots of the most prominent filesystems can't handle single-file fsyncs very efficiently. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Claudio Freire <klaussfreire@gmail.com> writes:
diff, for example, needs file inputs. So you need to create a file in order to compare against the original.
diff can work with pipes, FWIW. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 01/20/2015 05:17 PM, Andreas Schwab wrote:
and the shell syntax <(...) may help, too. E.g. compare /etc/SuSE-release on localhost vs. that on a server: $ diff -u /etc/SuSE-release <( ssh $SERVER 'cat /etc/SuSE-release' ) --- /etc/SuSE-release 2013-06-03 13:15:06.000000000 +0200 +++ /dev/fd/63 2015-01-20 17:31:17.280034243 +0100 @@ -1,3 +1,3 @@ -SUSE Linux Enterprise Server 11 (x86_64) -VERSION = 11 +SUSE Linux Enterprise Server 10 (i586) +VERSION = 10 PATCHLEVEL = 3 Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Jan 20, 2015 at 1:33 PM, Bernhard Voelker <mail@bernhard-voelker.de> wrote:
Exactly my point. Not a trivial thing to write in a safe manner, covering all corner cases (ie: you have to generate the comparison target twice, one for diff, one for creating the file, and make sure both times it's exactly the same, and that they don't generate temporaries). -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 19/01/15 a las 23:53, Felix Miata escribió:
Sorry, I do not believe this is something people should spend time on, something else in your computer will fail before the SSD runs out of write cycles. What you are describing are just insignificant implementation details. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Cristian Rodríguez composed on 2015-01-20 01:08 (UTC-0300):
We don't know that. SSD hasn't been in widespread use very long yet. The bad caps problem should be gone by now, leaving storage media the historically least reliable of PC components.
What you are describing are just insignificant implementation details.
So you say. Different distros have different strengths and weaknesses. Unnecessary disk writes I count as a unique openSUSE weakness that puzzles me that even exists. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 20.01.2015 um 05:48 schrieb Felix Miata:
If that was openSUSE's only weakness I consider that a great win! Greetings, Stephan -- Ma muaß weiterkämpfen, kämpfen bis zum Umfalln, a wenn die ganze Welt an Arsch offen hat, oder grad deswegn. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Stephan Kulow composed on 2015-01-20 08:19 (UTC+0100):
Felix Miata composed:
If that was openSUSE's only weakness I consider that a great win!
To be clear, I used the word "unique" because writing of unchanged files is an objectionable characteristic I have found exclusively on openSUSE. It is a problem exclusive to openSUSE, not openSUSE's sole problem. Certainly it has more than one, because it also has an also unique spelling case problem. :-p I consider openSUSE's Teutonic heritage and contribution weight YA asset keeping openSUSE above other distros in desirability. It's one of the reasons why I ever chose to try SuSE in the first place. I'm sure if openSUSE didn't have coolo doing what coolo does that openSUSE would have many more weaknesses than it has now. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hello Felix and all, On 2015-01-19 T 21:53 -0500 Felix Miata wrote:
On 2015-01-20 T 04:20 -0500 Felix Miata wrote:
... and probably that "Teutonic heritage" is one of the reasons for your impression of "wasting disk cycles": over the last years in product management here at SUSE I personally was surprised about many details, or I even doubted the validity of implementation details. Well. However, I also have learned that the approach of "better safe than sorry" often pays off in stability or reliability. Rewriting an unchanged file might be the result of this strategy. Nevertheless, I understand your concerns, as soon as we are not talking "average" HDD/SDD devices, but flash devices with a low rewrite count. That said, my suggestion is, to not approach your concern with an "all you can eat" FATE feature request, but document your examples in bugzilla and submit fixes to the individual packages you see responsible for that "waste" of disk cycles. That way you may learn, why things have been done the way they are, you may agree that it is the more safe approach, or you will have contributed to an improvement. Just my € 0.02 So long - MgE -- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: F.Imendörffer,J.Smithard,J.Guild,D.Upmanyu,G.Norton, HRB 21284(AG Nürnberg)
Matthias G. Eckermann composed on 2015-01-20 10:47 (UTC+0100):
"Well"??? Until that word, I was following.
Laudable plan. Oh that each day had more hours and each year more days. I'm afraid the foreseeable future and possibly much longer portends an opposite change in contribution level from these aging bones and what controls them. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2015-01-20 T 05:56 -0500 Felix Miata wrote:
The "Well" was referring to my role: as a product manager, I am not even entitled to review the "validity of implementation details", ... I sometime cannot resist and do nevertheless, ...;-)
My point is that someone _has_ to do the work and analyse where openSUSE behaves as you critize it. Who if not you could point to the single points, which made you open a FATE of removing that critized behaviour? So long - MgE -- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: F.Imendörffer,J.Smithard,J.Guild,D.Upmanyu,G.Norton, HRB 21284(AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2015-01-20 10:47, Matthias G. Eckermann wrote:
I can point at one case, known for a long time. When "rpm" updates a package, it replaces configuration files storing the old as .rpmold, or the new as .rpmnew — but often both files have exactly the same content! Admins are supposed to check both versions to determine what has to be migrated from the old configuration so that the system works properly; but checking files that are the same is a waste of our time. The rpm program should verify this itself and do not produce the duplicate, because rpm is in the right position to do this. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlS+PyUACgkQtTMYHG2NR9UxFACgicmn0R5GSzHabcqx8hwnvgrJ DaIAn1g7QGyv6SHT15441rf6FTKtsDEr =mYAN -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Jan 20, 2015 at 2:42 PM, Carlos E. R. <carlos.e.r@opensuse.org> wrote: ...
Actually it does (verify). If old file matches exactly signature in RPM database file is simply replaced with new. One thing where it breaks is modification date - edit file, remove changes - file is changed from RPM PoV. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2015-01-20 12:51, Andrei Borzenkov wrote:
I guessed that. :-) It should change policy and do a diff. As it is, I have to run "rcrpmconfigcheck" and do the diffs myself... so far, manually, which is a lot of time. Why check, if they are the same? Waste, no? Well... I have to weed out those that are the same file in order to find the few that are modified. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlS+Qt0ACgkQtTMYHG2NR9VFagCZAQ5qOMTDsIOBAhDgCi4ioAwi pdwAn3Ol7T4Bho3T0jW/frqs959HLg4+ =JVln -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 20.01.2015 um 05:08 schrieb Cristian Rodríguez:
Not only. Because for me it is often quite interesting when the last "real" change to a file was made -> ctime. So if there are useless and unnecessary ctime updates, this information is lost. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Jan 20, 2015 at 4:07 PM, Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
You probably mean mtime unless you are interested in the last chmod or chown.
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 20.01.2015 um 14:31 schrieb Andrei Borzenkov:
the time "ls -l" shows me :-) Anyway, I hope the point was clear: unnecessary changes are bad, not (only) because of write cycles or performance, but because of auditing. So we should avoid them if possible. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 20/01/15 a las 10:07, Stefan Seyfried escribió:
Here we agree, if the file contents are the same then the mtime has to be preserved.. that said I am running tumbleweed and it appears that only fstab was touch'ed in the last update...of course comparing the files will cause even more IO.. :-) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Monday 19 January 2015 21:53:06 Felix Miata wrote:
I do not want to raise another flame thread, but just out of curiosity... If you see that as a problem, what kind of use-case are you talking about? I've just opened 'iotop -o -d 0.5' one one monitor, and using firefox on second monitor. Every page click results in 3-4 seconds of io operations, some special effects, any page refresh. My KMail is using mysqld, and considering number of mailing lists it results in a bunch of IO operations every now and then. And those operations happen all day long, for most part of the week. Yet you're talking about system updates... This could mean that your use-case is so far away from mine, I'm really curious how you're using your machines... -- Regards, Stas -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Stanislav Baiduzhyi composed on 2015-01-20 11:18 (UTC+0100):
I've a nature curious how things work, or not. With software I never became anything like efficient at producing solutions to problems found, doing better at conveying observations to those who might implement improvement. Most of my machines are tools of experimentation and reproducion. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 20.01.2015 um 03:53 schrieb Felix Miata:
not here: seife@susi:~> ls -l /etc/HOSTNAME lrwxrwxrwx 1 root root 8 19. Dez 10:02 /etc/HOSTNAME -> hostname seife@susi:~> ls -l /etc/hostname -rw-r--r-- 1 root root 5 17. Feb 2014 /etc/hostname seife@susi:~> rpm -q --last netcfg netcfg-11.5-29.1.noarch Mo 22 Dez 2014 20:57:39 CET
During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup.
not for me: seife@susi:~> ls -l /etc/fstab -rw-r--r-- 1 root root 1353 23. Nov 23:10 /etc/fstab seife@susi:~> l /etc/*-release -rw-r--r-- 1 root root 288 8. Jan 13:11 /etc/os-release -rw-r--r-- 1 root root 165 8. Jan 13:11 /etc/SuSE-release (last update apparently on 2015-01-08) So it's probably easiest if you find out who is touching which file (might be a postinstall script of a package) and file a but against that. systemtap is one possibility to get the required information. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2015-01-20 14:03, Stefan Seyfried wrote:
Mine is dated 2013. (It is not a symlink, and 'hostname' does not exist. Maybe 13.2 changed this)
About same here, Nov 8.
cer@Telcontar:~> l /etc/*-release - -rw-r--r-- 1 root root 153 Oct 23 2013 /etc/SuSE-release - -rw-r--r-- 1 root root 110 Oct 2 2013 /etc/lsb-release - -rw-r--r-- 1 root root 264 Oct 23 2013 /etc/os-release cer@Telcontar:~>
I agree. I'm unfamiliar with that one, but apparmour is another. Do you know of a link that explains "systemtap"? - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlS+X60ACgkQtTMYHG2NR9XregCfSPNxLLrYeYgqEnHrsYgKgsGa xuwAnifjUmN3dxTSxoFfhKX2vcEEvJ1e =/65x -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
"Carlos E. R." <carlos.e.r@opensuse.org> writes:
Do you know of a link that explains "systemtap"?
The wikipedia page has a lot of information. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 20.01.2015 um 15:01 schrieb Carlos E. R.:
zypper in systemtap sytemtap-docs then go through /usr/share/doc/packages/systemtap/examples/ Beware: as any powerful tool, it is not for the faint of heart :) To actually use it, you also need the debuginfo matching your running kernel installed which needs quite some space: susi:~ # zypper in kernel-desktop-debuginfo Loading repository data... Reading installed packages... Resolving package dependencies... The following NEW package is going to be installed: kernel-desktop-debuginfo 1 new package to install. Overall download size: 342.0 MiB. Already cached: 0 B After the operation, additional 2.4 GiB will be used. Continue? [y/n/? shows all options] (y): n But if I really would want to know who is changing which file in /etc/ during "zypper dup", I would install it, adapt one of the examples in .../examples/io/ to just print out all file accesses under /etc/ with process name and PID and then probably trigger a "pstree -p" in userspace for everyone to find the "originating" process. It's probably doable, but quite some work. And since I cannot reproduce the problem, I cannot debug it. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2015-01-20 16:41, Stefan Seyfried wrote:
Ok, then apparmor is far easier: nothing is required. I can dig out the details from my notes somewhere, or Cristian could tell it: it was actually his idea, years ago ;-) Now I have to do something else. Wait... I found the note: it was filled as apparmour instead of apparmor. +++—-—-—-—-—-—-—-—-—-—-—-—- try this: # auditctl -w /dev/null -p a # auditctl -e 1 and then watch the logs... if auditctl is not found, when you execute it as root, install package "audit" Date: Fri, 06 Mar 2009 12:04:21 -0300 From: Cristian Rodríguez <> —-—-—-—-—-—-—-—-—-—-—-—-++- The above will detect a change to /dev/null. Read the manual, the options are different depending on what type of change or access you want to detect. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlS+gLEACgkQtTMYHG2NR9UpSgCdGecNzQazG9GNdKDrdLhfewFg M5wAoILtWC1DWScXMjGYHxLTV94nB2wy =iSA2 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Jan 19, 2015 at 11:53 PM, Felix Miata <mrmazda@earthlink.net> wrote:
Do you do zypper dup often enough to worry about this? SSD media, which I assume you're worrying about, buffer and batch writes internally so unless some of those writes explicitly fsync, writing 5 bytes in 100 files or 5000 bytes at once shouldn't matter much. Basically, count how many bytes (blocks?) in total are written to disk while updating, and unless you can significantly decrease that number, which files get touched matter less. I'd worry more (way more) about logging. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Claudio Freire composed on 2015-01-20 00:29 (UTC-0300):
Felix Miata wrote:
During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup.
Many files in /etc/sysconfig get written during updating, yet remain unchanged from their backups.
This wastefulness is not common to other distros I have familiarity with. Why is it happening in openSUSE?
Do you do zypper dup often enough to worry about this?
There's more than one issue here. Did you read https://features.opensuse.org/313803 ever? Maybe it is trivial write volume, but there's the principle involved of wastefulness that applies not just in writing at update time, but also backup/restore processes.
I don't understand why you wrote this. I'm not at all concerned about efficiency here except as writing any sector at all is unnecessary. The file was not changed, so why the write?
Again I don't get it. Writes are slower I/O than reads. In the case of file not changed, why any write at all?
I'd worry more (way more) about logging.
A bigger can of worms to be sure, but little problems can batch up enough to break the camel's back. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Jan 20, 2015 at 12:46 AM, Felix Miata <mrmazda@earthlink.net> wrote:
It's irrelevant. That doesn't reference write cycles, and the post was about write cycles. I addressed the post (write cycles), and not the fate itself. In fact, it's not even clear doing what the fate indicates would improve write cycle efficiency, it may even make it worse (say, if the update script merely left the new file in place, and updated the timestamps on the old files repeatedly causing more metadata flushes).
Because batched writes get written in one flash sector (usually 2MB), and can contain more than one filesystem block. Writes, even random writes, on a block device backed by flash usually results in one single write cycle on one block. That's wear leveling at work.
If you write one block at all, the remaining 511 will usually be for free. So, why make a huge effort to avoid writing a few blocks if it's not going to report any benefit?
It should all start from the lowest hanging fruit. It makes little sense to go for the highest first. For another example, new installations should use relatime by default. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Claudio Freire composed on 2015-01-20 00:58 (UTC-0300):
There's still communication failure here. My interest is in why there is any write at all when file content before and after update process is identical. Maybe fstab makes the better example, since the stub of it in some rpm is obviously not going to work as-is on an already installed and configured installation. It too is a small file, so RAM should be sufficient for update process to work whatever magic it attempts.
Because batched writes get written in one flash sector (usually 2MB), and can contain more than one filesystem block.
I don't understand why this would be relevant to my question. One single sector file could be the difference between 2 batches and 1 batch.
Again I don't get it. Writes are slower I/O than reads. In the case of file not changed, why any write at all?
If you write one block at all, the remaining 511 will usually be for free.
So, why make a huge effort to avoid writing a few blocks if it's not going to report any benefit?
Huge effort? A few here, a few there. Maybe it doesn't seem so with electrons, but it does consume energy to make them move. Waste not, want not.
It should all start from the lowest hanging fruit.
It makes little sense to go for the highest first.
I only ever thought about $SUBJECT as a result of remembering the FATE. Here is a possibility of hitting multiple targets hit with one shot, instituting policy that induces less waste.
For another example, new installations should use relatime by default.
How that is better than noatime, which I started using long before ever hearing about relatime, I never figured out. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2015-01-20 05:35, Felix Miata wrote:
Claudio Freire composed on 2015-01-20 00:58 (UTC-0300):
He refers to the programming effort needed to change this.
If it is a programming practice, you can not influence programmers much. And you'd have to find exactly why this is happening, case by case. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On Tue, Jan 20, 2015 at 1:35 AM, Felix Miata <mrmazda@earthlink.net> wrote:
I imagine the rewriting will happen anyway unless a program is carefully crafted not to write to disk. I don't know the culprit, but I've programmed enough to get a rough idea of how such an update might work. diff, for example, needs file inputs. So you need to create a file in order to compare against the original. The file cannot be created in /tmp, that would be a security hole (/tmp can be written by anyone). If you created an /etc/fstab.new and diffed it, you may hope the kernel will cache the new file in memory and absorb the write followed by the unlink, but it's not entirely clear whether something like that would work when coded into the update procedure. Updating fstab won't be the only thing in that procedure, and the write cache could very well get flushed before you unlink. So, it *is* tricky coming up with a solution to that very specific problem without introducing either security issues or more unintended writes. On Tue, Jan 20, 2015 at 7:18 AM, Stanislav Baiduzhyi <baiduzhyi.devel@gmail.com> wrote:
I did report that for Firefox, it didn't get much attention. It's a performance killer too, since they call fsync every second as well on a timestamp file. Lots of the most prominent filesystems can't handle single-file fsyncs very efficiently. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (12)
-
Andreas Schwab -
Andrei Borzenkov -
Bernhard Voelker -
Carlos E. R. -
Carlos E. R. -
Claudio Freire -
Cristian Rodríguez -
Felix Miata -
Matthias G. Eckermann -
Stanislav Baiduzhyi -
Stefan Seyfried -
Stephan Kulow