[opensuse-factory] limited write cycles on filesystem media

newer
[opensuse-factory] kwallet fails...

Felix Miata

20 Jan 2015 20 Jan '15

02:53

We know popular modern media has limited write expectancy. The numbers are high and climbing as technology improves, but are nevertheless finite. Why are we habitually wasting them? Examples: Every installation, and upgrade, of netcfg, though often as few as 5 bytes, is writing a 4096 byte hostname/HOSTNAME sector (or possibly more) to the media, though no change to it/them has been made. During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup. Many files in /etc/sysconfig get written during updating, yet remain unchanged from their backups. This wastefulness is not common to other distros I have familiarity with. Why is it happening in openSUSE? I did create a FATE for this. Today it reached 30 months of age. It has a score of 4 and no comments. https://features.opensuse.org/313803 -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Show replies by date

Claudio Freire

20 Jan 20 Jan

03:29

On Mon, Jan 19, 2015 at 11:53 PM, Felix Miata wrote:

...

Every installation, and upgrade, of netcfg, though often as few as 5 bytes, is writing a 4096 byte hostname/HOSTNAME sector (or possibly more) to the media, though no change to it/them has been made.

During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup.

Many files in /etc/sysconfig get written during updating, yet remain unchanged from their backups.

This wastefulness is not common to other distros I have familiarity with. Why is it happening in openSUSE?

Do you do zypper dup often enough to worry about this? SSD media, which I assume you're worrying about, buffer and batch writes internally so unless some of those writes explicitly fsync, writing 5 bytes in 100 files or 5000 bytes at once shouldn't matter much. Basically, count how many bytes (blocks?) in total are written to disk while updating, and unless you can significantly decrease that number, which files get touched matter less. I'd worry more (way more) about logging. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Felix Miata

03:46

Claudio Freire composed on 2015-01-20 00:29 (UTC-0300):

...

Felix Miata wrote:

...

...
Every installation, and upgrade, of netcfg, though often as few as 5 bytes, is writing a 4096 byte hostname/HOSTNAME sector (or possibly more) to the media, though no change to it/them has been made.

...

...
During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup.

...

...
Many files in /etc/sysconfig get written during updating, yet remain unchanged from their backups.

...

...
This wastefulness is not common to other distros I have familiarity with. Why is it happening in openSUSE?

...

Do you do zypper dup often enough to worry about this?

There's more than one issue here. Did you read https://features.opensuse.org/313803 ever? Maybe it is trivial write volume, but there's the principle involved of wastefulness that applies not just in writing at update time, but also backup/restore processes.

...

SSD media, which I assume you're worrying about, buffer and batch writes internally so unless some of those writes explicitly fsync, writing 5 bytes in 100 files or 5000 bytes at once shouldn't matter much.

I don't understand why you wrote this. I'm not at all concerned about efficiency here except as writing any sector at all is unnecessary. The file was not changed, so why the write?

...

Basically, count how many bytes (blocks?) in total are written to disk while updating, and unless you can significantly decrease that number, which files get touched matter less.

Again I don't get it. Writes are slower I/O than reads. In the case of file not changed, why any write at all?

...

I'd worry more (way more) about logging.

A bigger can of worms to be sure, but little problems can batch up enough to break the camel's back. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Claudio Freire

03:58

On Tue, Jan 20, 2015 at 12:46 AM, Felix Miata wrote:

...

Claudio Freire composed on 2015-01-20 00:29 (UTC-0300):

...
Felix Miata wrote:

...
...
Every installation, and upgrade, of netcfg, though often as few as 5 bytes, is writing a 4096 byte hostname/HOSTNAME sector (or possibly more) to the media, though no change to it/them has been made.

...
...
During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup.

...
...
Many files in /etc/sysconfig get written during updating, yet remain unchanged from their backups.

...
...
This wastefulness is not common to other distros I have familiarity with. Why is it happening in openSUSE?

...
Do you do zypper dup often enough to worry about this?

There's more than one issue here. Did you read https://features.opensuse.org/313803 ever? Maybe it is trivial write volume, but there's the principle involved of wastefulness that applies not just in writing at update time, but also backup/restore processes.

It's irrelevant. That doesn't reference write cycles, and the post was about write cycles. I addressed the post (write cycles), and not the fate itself. In fact, it's not even clear doing what the fate indicates would improve write cycle efficiency, it may even make it worse (say, if the update script merely left the new file in place, and updated the timestamps on the old files repeatedly causing more metadata flushes).

...

...
SSD media, which I assume you're worrying about, buffer and batch writes internally so unless some of those writes explicitly fsync, writing 5 bytes in 100 files or 5000 bytes at once shouldn't matter much.

I don't understand why you wrote this. I'm not at all concerned about efficiency here except as writing any sector at all is unnecessary. The file was not changed, so why the write?

Because batched writes get written in one flash sector (usually 2MB), and can contain more than one filesystem block. Writes, even random writes, on a block device backed by flash usually results in one single write cycle on one block. That's wear leveling at work.

...

...
Basically, count how many bytes (blocks?) in total are written to disk while updating, and unless you can significantly decrease that number, which files get touched matter less.

Again I don't get it. Writes are slower I/O than reads. In the case of file not changed, why any write at all?

If you write one block at all, the remaining 511 will usually be for free. So, why make a huge effort to avoid writing a few blocks if it's not going to report any benefit?

...

...
I'd worry more (way more) about logging.

A bigger can of worms to be sure, but little problems can batch up enough to break the camel's back.

It should all start from the lowest hanging fruit. It makes little sense to go for the highest first. For another example, new installations should use relatime by default. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Felix Miata

04:35

Claudio Freire composed on 2015-01-20 00:58 (UTC-0300):

...

it's not even clear doing what the fate indicates would improve write cycle efficiency, it may even make it worse (say, if the update script merely left the new file in place, and updated the timestamps on the old files repeatedly causing more metadata flushes).

There's still communication failure here. My interest is in why there is any write at all when file content before and after update process is identical. Maybe fstab makes the better example, since the stub of it in some rpm is obviously not going to work as-is on an already installed and configured installation. It too is a small file, so RAM should be sufficient for update process to work whatever magic it attempts.

...

...
I don't understand why you wrote this. I'm not at all concerned about efficiency here except as writing any sector at all is unnecessary. The file was not changed, so why the write?

...

Because batched writes get written in one flash sector (usually 2MB), and can contain more than one filesystem block.

I don't understand why this would be relevant to my question. One single sector file could be the difference between 2 batches and 1 batch.

...

Writes, even random writes, on a block device backed by flash usually results in one single write cycle on one block. That's wear leveling at work.

...

...
...
Basically, count how many bytes (blocks?) in total are written to disk while updating, and unless you can significantly decrease that number, which files get touched matter less.

...

...
Again I don't get it. Writes are slower I/O than reads. In the case of file not changed, why any write at all?

...

If you write one block at all, the remaining 511 will usually be for free.

...

So, why make a huge effort to avoid writing a few blocks if it's not going to report any benefit?

Huge effort? A few here, a few there. Maybe it doesn't seem so with electrons, but it does consume energy to make them move. Waste not, want not.

...

It should all start from the lowest hanging fruit.

...

It makes little sense to go for the highest first.

I only ever thought about $SUBJECT as a result of remembering the FATE. Here is a possibility of hitting multiple targets hit with one shot, instituting policy that induces less waste.

...

For another example, new installations should use relatime by default.

How that is better than noatime, which I started using long before ever hearing about relatime, I never figured out. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Carlos E. R.

11:35

On 2015-01-20 05:35, Felix Miata wrote:

...

Claudio Freire composed on 2015-01-20 00:58 (UTC-0300):

...

...
So, why make a huge effort to avoid writing a few blocks if it's not going to report any benefit?

Huge effort? A few here, a few there. Maybe it doesn't seem so with electrons, but it does consume energy to make them move. Waste not, want not.

He refers to the programming effort needed to change this.

...

I only ever thought about $SUBJECT as a result of remembering the FATE. Here is a possibility of hitting multiple targets hit with one shot, instituting policy that induces less waste.

If it is a programming practice, you can not influence programmers much. And you'd have to find exactly why this is happening, case by case. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

Claudio Freire

14:57

On Tue, Jan 20, 2015 at 1:35 AM, Felix Miata wrote:

...

...
If you write one block at all, the remaining 511 will usually be for free.

...
So, why make a huge effort to avoid writing a few blocks if it's not going to report any benefit?

Huge effort? A few here, a few there. Maybe it doesn't seem so with electrons, but it does consume energy to make them move. Waste not, want not.

I imagine the rewriting will happen anyway unless a program is carefully crafted not to write to disk. I don't know the culprit, but I've programmed enough to get a rough idea of how such an update might work. diff, for example, needs file inputs. So you need to create a file in order to compare against the original. The file cannot be created in /tmp, that would be a security hole (/tmp can be written by anyone). If you created an /etc/fstab.new and diffed it, you may hope the kernel will cache the new file in memory and absorb the write followed by the unlink, but it's not entirely clear whether something like that would work when coded into the update procedure. Updating fstab won't be the only thing in that procedure, and the write cache could very well get flushed before you unlink. So, it *is* tricky coming up with a solution to that very specific problem without introducing either security issues or more unintended writes. On Tue, Jan 20, 2015 at 7:18 AM, Stanislav Baiduzhyi wrote:

...

I do not want to raise another flame thread, but just out of curiosity... If you see that as a problem, what kind of use-case are you talking about?

I've just opened 'iotop -o -d 0.5' one one monitor, and using firefox on second monitor. Every page click results in 3-4 seconds of io operations, some special effects, any page refresh.

I did report that for Firefox, it didn't get much attention. It's a performance killer too, since they call fsync every second as well on a timestamp file. Lots of the most prominent filesystems can't handle single-file fsyncs very efficiently. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Andreas Schwab

16:17

New subject: [opensuse-factory] Re: limited write cycles on filesystem media

Claudio Freire writes:

...

diff, for example, needs file inputs. So you need to create a file in order to compare against the original.

diff can work with pipes, FWIW. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Bernhard Voelker

16:33

New subject: [opensuse-factory] Re: limited write cycles on filesystem media

On 01/20/2015 05:17 PM, Andreas Schwab wrote:

...

Claudio Freire writes:

...
diff, for example, needs file inputs. So you need to create a file in order to compare against the original.

diff can work with pipes, FWIW.

and the shell syntax <(...) may help, too. E.g. compare /etc/SuSE-release on localhost vs. that on a server: $ diff -u /etc/SuSE-release <( ssh $SERVER 'cat /etc/SuSE-release' ) --- /etc/SuSE-release 2013-06-03 13:15:06.000000000 +0200 +++ /dev/fd/63 2015-01-20 17:31:17.280034243 +0100 @@ -1,3 +1,3 @@ -SUSE Linux Enterprise Server 11 (x86_64) -VERSION = 11 +SUSE Linux Enterprise Server 10 (i586) +VERSION = 10 PATCHLEVEL = 3 Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Claudio Freire

16:38

New subject: [opensuse-factory] Re: limited write cycles on filesystem media

On Tue, Jan 20, 2015 at 1:33 PM, Bernhard Voelker wrote:

...

On 01/20/2015 05:17 PM, Andreas Schwab wrote:

...
Claudio Freire writes:

...
diff, for example, needs file inputs. So you need to create a file in order to compare against the original.

diff can work with pipes, FWIW.

and the shell syntax <(...) may help, too. E.g. compare /etc/SuSE-release on localhost vs. that on a server:

$ diff -u /etc/SuSE-release <( ssh $SERVER 'cat /etc/SuSE-release' ) --- /etc/SuSE-release 2013-06-03 13:15:06.000000000 +0200 +++ /dev/fd/63 2015-01-20 17:31:17.280034243 +0100 @@ -1,3 +1,3 @@ -SUSE Linux Enterprise Server 11 (x86_64) -VERSION = 11 +SUSE Linux Enterprise Server 10 (i586) +VERSION = 10 PATCHLEVEL = 3

Exactly my point. Not a trivial thing to write in a safe manner, covering all corner cases (ie: you have to generate the comparison target twice, one for diff, one for creating the file, and make sure both times it's exactly the same, and that they don't generate temporaries). -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Cristian Rodríguez

04:08

El 19/01/15 a las 23:53, Felix Miata escribió:

...

We know popular modern media has limited write expectancy. The numbers are high and climbing as technology improves, but are nevertheless finite. Why are we habitually wasting them?

Examples:

Every installation, and upgrade, of netcfg, though often as few as 5 bytes, is writing a 4096 byte hostname/HOSTNAME sector (or possibly more) to the media, though no change to it/them has been made.

During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup.

Many files in /etc/sysconfig get written during updating, yet remain unchanged from their backups.

This wastefulness is not common to other distros I have familiarity with. Why is it happening in openSUSE?

I did create a FATE for this. Today it reached 30 months of age. It has a score of 4 and no comments. https://features.opensuse.org/313803

Sorry, I do not believe this is something people should spend time on, something else in your computer will fail before the SSD runs out of write cycles. What you are describing are just insignificant implementation details. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Felix Miata

04:48

Cristian Rodríguez composed on 2015-01-20 01:08 (UTC-0300):

...

Sorry, I do not believe this is something people should spend time on, something else in your computer will fail before the SSD runs out of write cycles.

We don't know that. SSD hasn't been in widespread use very long yet. The bad caps problem should be gone by now, leaving storage media the historically least reliable of PC components.

...

What you are describing are just insignificant implementation details.

So you say. Different distros have different strengths and weaknesses. Unnecessary disk writes I count as a unique openSUSE weakness that puzzles me that even exists. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Stephan Kulow

07:19

Am 20.01.2015 um 05:48 schrieb Felix Miata:

...

So you say. Different distros have different strengths and weaknesses. Unnecessary disk writes I count as a unique openSUSE weakness that puzzles me that even exists.

If that was openSUSE's only weakness I consider that a great win! Greetings, Stephan -- Ma muaß weiterkämpfen, kämpfen bis zum Umfalln, a wenn die ganze Welt an Arsch offen hat, oder grad deswegn. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Felix Miata

09:20

Stephan Kulow composed on 2015-01-20 08:19 (UTC+0100):

...

Felix Miata composed:

...

...
So you say. Different distros have different strengths and weaknesses. Unnecessary disk writes I count as a unique openSUSE weakness that puzzles me that even exists.

...

If that was openSUSE's only weakness I consider that a great win!

To be clear, I used the word "unique" because writing of unchanged files is an objectionable characteristic I have found exclusively on openSUSE. It is a problem exclusive to openSUSE, not openSUSE's sole problem. Certainly it has more than one, because it also has an also unique spelling case problem. :-p I consider openSUSE's Teutonic heritage and contribution weight YA asset keeping openSUSE above other distros in desirability. It's one of the reasons why I ever chose to try SuSE in the first place. I'm sure if openSUSE didn't have coolo doing what coolo does that openSUSE would have many more weaknesses than it has now. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Matthias G. Eckermann

09:47

Hello Felix and all, On 2015-01-19 T 21:53 -0500 Felix Miata wrote:

...

Examples:

Every installation, and upgrade, of netcfg, though often as few as 5 bytes, is writing a 4096 byte hostname/HOSTNAME sector (or possibly more) to the media, though no change to it/them has been made.

During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup.

Many files in /etc/sysconfig get written during updating, yet remain unchanged from their backups.

This wastefulness is not common to other distros I have familiarity with. Why is it happening in openSUSE?

On 2015-01-20 T 04:20 -0500 Felix Miata wrote:

...

To be clear, I used the word "unique" because writing of unchanged files is an objectionable characteristic I have found exclusively on openSUSE. It is a problem exclusive to openSUSE, not openSUSE's sole problem. Certainly it has more than one, because it also has an also unique spelling case problem. :-p

I consider openSUSE's Teutonic heritage and contribution weight YA asset keeping openSUSE above other distros in desirability. It's one of the reasons why I ever chose to try SuSE in the first place.

... and probably that "Teutonic heritage" is one of the reasons for your impression of "wasting disk cycles": over the last years in product management here at SUSE I personally was surprised about many details, or I even doubted the validity of implementation details. Well. However, I also have learned that the approach of "better safe than sorry" often pays off in stability or reliability. Rewriting an unchanged file might be the result of this strategy. Nevertheless, I understand your concerns, as soon as we are not talking "average" HDD/SDD devices, but flash devices with a low rewrite count. That said, my suggestion is, to not approach your concern with an "all you can eat" FATE feature request, but document your examples in bugzilla and submit fixes to the individual packages you see responsible for that "waste" of disk cycles. That way you may learn, why things have been done the way they are, you may agree that it is the more safe approach, or you will have contributed to an improvement. Just my € 0.02 So long - MgE -- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: F.Imendörffer,J.Smithard,J.Guild,D.Upmanyu,G.Norton, HRB 21284(AG Nürnberg)

Felix Miata

10:56

Matthias G. Eckermann composed on 2015-01-20 10:47 (UTC+0100):

...

... and probably that "Teutonic heritage" is one of the reasons for your impression of "wasting disk cycles": over the last years in product management here at SUSE I personally was surprised about many details, or I even doubted the validity of implementation details. Well.

"Well"??? Until that word, I was following.

...

However, I also have learned that the approach of "better safe than sorry" often pays off in stability or reliability. Rewriting an unchanged file might be the result of this strategy.

...

Nevertheless, I understand your concerns, as soon as we are not talking "average" HDD/SDD devices, but flash devices with a low rewrite count.

...

That said, my suggestion is, to not approach your concern with an "all you can eat" FATE feature request, but document your examples in bugzilla and submit fixes to the individual packages you see responsible for that "waste" of disk cycles. That way you may learn, why things have been done the way they are, you may agree that it is the more safe approach, or you will have contributed to an improvement.

Laudable plan. Oh that each day had more hours and each year more days. I'm afraid the foreseeable future and possibly much longer portends an opposite change in contribution level from these aging bones and what controls them. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Matthias G. Eckermann

16:54

On 2015-01-20 T 05:56 -0500 Felix Miata wrote:

...

Matthias G. Eckermann composed on 2015-01-20 10:47 (UTC+0100):

...
... and probably that "Teutonic heritage" is one of the reasons for your impression of "wasting disk cycles": over the last years in product management here at SUSE I personally was surprised about many details, or I even doubted the validity of implementation details. Well.

"Well"??? Until that word, I was following.

The "Well" was referring to my role: as a product manager, I am not even entitled to review the "validity of implementation details", ... I sometime cannot resist and do nevertheless, ...;-)

...

...
However, I also have learned that the approach of "better safe than sorry" often pays off in stability or reliability. Rewriting an unchanged file might be the result of this strategy.

...
Nevertheless, I understand your concerns, as soon as we are not talking "average" HDD/SDD devices, but flash devices with a low rewrite count.

...
That said, my suggestion is, to not approach your concern with an "all you can eat" FATE feature request, but document your examples in bugzilla and submit fixes to the individual packages you see responsible for that "waste" of disk cycles. That way you may learn, why things have been done the way they are, you may agree that it is the more safe approach, or you will have contributed to an improvement.

Laudable plan. Oh that each day had more hours and each year more days. I'm afraid the foreseeable future and possibly much longer portends an opposite change in contribution level from these aging bones and what controls them.

My point is that someone _has_ to do the work and analyse where openSUSE behaves as you critize it. Who if not you could point to the single points, which made you open a FATE of removing that critized behaviour? So long - MgE -- Matthias G. Eckermann Senior Product Manager SUSE® Linux Enterprise SUSE LINUX GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: F.Imendörffer,J.Smithard,J.Guild,D.Upmanyu,G.Norton, HRB 21284(AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Carlos E. R.

11:42

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2015-01-20 10:47, Matthias G. Eckermann wrote:

...

That said, my suggestion is, to not approach your concern with an "all you can eat" FATE feature request, but document your examples in bugzilla and submit fixes to the individual packages you see responsible for that "waste" of disk cycles. That way you may learn, why things have been done the way they are, you may agree that it is the more safe approach, or you will have contributed to an improvement.

Just my € 0.02

I can point at one case, known for a long time. When "rpm" updates a package, it replaces configuration files storing the old as .rpmold, or the new as .rpmnew — but often both files have exactly the same content! Admins are supposed to check both versions to determine what has to be migrated from the old configuration so that the system works properly; but checking files that are the same is a waste of our time. The rpm program should verify this itself and do not produce the duplicate, because rpm is in the right position to do this. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlS+PyUACgkQtTMYHG2NR9UxFACgicmn0R5GSzHabcqx8hwnvgrJ DaIAn1g7QGyv6SHT15441rf6FTKtsDEr =mYAN -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Andrei Borzenkov

11:51

On Tue, Jan 20, 2015 at 2:42 PM, Carlos E. R. wrote: ...

...

When "rpm" updates a package, it replaces configuration files storing the old as .rpmold, or the new as .rpmnew — but often both files have exactly the same content!

Admins are supposed to check both versions to determine what has to be migrated from the old configuration so that the system works properly; but checking files that are the same is a waste of our time.

The rpm program should verify this itself and do not produce the duplicate, because rpm is in the right position to do this.

Actually it does (verify). If old file matches exactly signature in RPM database file is simply replaced with new. One thing where it breaks is modification date - edit file, remove changes - file is changed from RPM PoV. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Carlos E. R.

11:58

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2015-01-20 12:51, Andrei Borzenkov wrote:

...

On Tue, Jan 20, 2015 at 2:42 PM, Carlos E. R. <> wrote: ...

...
When "rpm" updates a package, it replaces configuration files storing the old as .rpmold, or the new as .rpmnew — but often both files have exactly the same content!

Admins are supposed to check both versions to determine what has to be migrated from the old configuration so that the system works properly; but checking files that are the same is a waste of our time.

The rpm program should verify this itself and do not produce the duplicate, because rpm is in the right position to do this.

Actually it does (verify). If old file matches exactly signature in RPM database file is simply replaced with new. One thing where it breaks is modification date - edit file, remove changes - file is changed from RPM PoV.

I guessed that. :-) It should change policy and do a diff. As it is, I have to run "rcrpmconfigcheck" and do the diffs myself... so far, manually, which is a lot of time. Why check, if they are the same? Waste, no? Well... I have to weed out those that are the same file in order to find the few that are modified. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlS+Qt0ACgkQtTMYHG2NR9VFagCZAQ5qOMTDsIOBAhDgCi4ioAwi pdwAn3Ol7T4Bho3T0jW/frqs959HLg4+ =JVln -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Stefan Seyfried

13:07

Am 20.01.2015 um 05:08 schrieb Cristian Rodríguez:

...

Sorry, I do not believe this is something people should spend time on, something else in your computer will fail before the SSD runs out of write cycles.

What you are describing are just insignificant implementation details.

Not only. Because for me it is often quite interesting when the last "real" change to a file was made -> ctime. So if there are useless and unnecessary ctime updates, this information is lost. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Andrei Borzenkov

13:31

On Tue, Jan 20, 2015 at 4:07 PM, Stefan Seyfried wrote:

...

Am 20.01.2015 um 05:08 schrieb Cristian Rodríguez:

...
Sorry, I do not believe this is something people should spend time on, something else in your computer will fail before the SSD runs out of write cycles.

What you are describing are just insignificant implementation details.

Not only. Because for me it is often quite interesting when the last "real" change to a file was made -> ctime.

You probably mean mtime unless you are interested in the last chmod or chown.

...

So if there are useless and unnecessary ctime updates, this information is lost. -- Stefan Seyfried

"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Stefan Seyfried

15:33

Am 20.01.2015 um 14:31 schrieb Andrei Borzenkov:

...

On Tue, Jan 20, 2015 at 4:07 PM, Stefan Seyfried wrote:

...
Am 20.01.2015 um 05:08 schrieb Cristian Rodríguez:

...
Sorry, I do not believe this is something people should spend time on, something else in your computer will fail before the SSD runs out of write cycles.

What you are describing are just insignificant implementation details.

Not only. Because for me it is often quite interesting when the last "real" change to a file was made -> ctime.

You probably mean mtime unless you are interested in the last chmod or chown.

the time "ls -l" shows me :-) Anyway, I hope the point was clear: unnecessary changes are bad, not (only) because of write cycles or performance, but because of auditing. So we should avoid them if possible. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Cristian Rodríguez

15:09

El 20/01/15 a las 10:07, Stefan Seyfried escribió:

...

Am 20.01.2015 um 05:08 schrieb Cristian Rodríguez:

...
Sorry, I do not believe this is something people should spend time on, something else in your computer will fail before the SSD runs out of write cycles.

What you are describing are just insignificant implementation details.

Not only. Because for me it is often quite interesting when the last "real" change to a file was made -> ctime. So if there are useless and unnecessary ctime updates, this information is lost.

Here we agree, if the file contents are the same then the mtime has to be preserved.. that said I am running tumbleweed and it appears that only fstab was touch'ed in the last update...of course comparing the files will cause even more IO.. :-) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Stanislav Baiduzhyi

10:18

On Monday 19 January 2015 21:53:06 Felix Miata wrote:

...

We know popular modern media has limited write expectancy. The numbers are high and climbing as technology improves, but are nevertheless finite. Why are we habitually wasting them?

Examples:

Every installation, and upgrade, of netcfg, though often as few as 5 bytes, is writing a 4096 byte hostname/HOSTNAME sector (or possibly more) to the media, though no change to it/them has been made.

During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup.

Many files in /etc/sysconfig get written during updating, yet remain unchanged from their backups.

This wastefulness is not common to other distros I have familiarity with. Why is it happening in openSUSE?

I did create a FATE for this. Today it reached 30 months of age. It has a score of 4 and no comments. https://features.opensuse.org/313803

I do not want to raise another flame thread, but just out of curiosity... If you see that as a problem, what kind of use-case are you talking about? I've just opened 'iotop -o -d 0.5' one one monitor, and using firefox on second monitor. Every page click results in 3-4 seconds of io operations, some special effects, any page refresh. My KMail is using mysqld, and considering number of mailing lists it results in a bunch of IO operations every now and then. And those operations happen all day long, for most part of the week. Yet you're talking about system updates... This could mean that your use-case is so far away from mine, I'm really curious how you're using your machines... -- Regards, Stas -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Felix Miata

11:43

Stanislav Baiduzhyi composed on 2015-01-20 11:18 (UTC+0100):

...

I do not want to raise another flame thread, but just out of curiosity... If you see that as a problem, what kind of use-case are you talking about? ... I'm really curious how you're using your machines...

I've a nature curious how things work, or not. With software I never became anything like efficient at producing solutions to problems found, doing better at conveying observations to those who might implement improvement. Most of my machines are tools of experimentation and reproducion. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Stefan Seyfried

13:03

Am 20.01.2015 um 03:53 schrieb Felix Miata:

...

Every installation, and upgrade, of netcfg, though often as few as 5 bytes, is writing a 4096 byte hostname/HOSTNAME sector (or possibly more) to the media, though no change to it/them has been made.

not here: seife@susi:~> ls -l /etc/HOSTNAME lrwxrwxrwx 1 root root 8 19. Dez 10:02 /etc/HOSTNAME -> hostname seife@susi:~> ls -l /etc/hostname -rw-r--r-- 1 root root 5 17. Feb 2014 /etc/hostname seife@susi:~> rpm -q --last netcfg netcfg-11.5-29.1.noarch Mo 22 Dez 2014 20:57:39 CET

...

During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup.

not for me: seife@susi:~> ls -l /etc/fstab -rw-r--r-- 1 root root 1353 23. Nov 23:10 /etc/fstab seife@susi:~> l /etc/*-release -rw-r--r-- 1 root root 288 8. Jan 13:11 /etc/os-release -rw-r--r-- 1 root root 165 8. Jan 13:11 /etc/SuSE-release (last update apparently on 2015-01-08) So it's probably easiest if you find out who is touching which file (might be a postinstall script of a package) and file a but against that. systemtap is one possibility to get the required information. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Carlos E. R.

14:01

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2015-01-20 14:03, Stefan Seyfried wrote:

...

Am 20.01.2015 um 03:53 schrieb Felix Miata:

...
Every installation, and upgrade, of netcfg, though often as few as 5 bytes, is writing a 4096 byte hostname/HOSTNAME sector (or possibly more) to the media, though no change to it/them has been made.

not here: seife@susi:~> ls -l /etc/HOSTNAME lrwxrwxrwx 1 root root 8 19. Dez 10:02 /etc/HOSTNAME -> hostname seife@susi:~> ls -l /etc/hostname

Mine is dated 2013. (It is not a symlink, and 'hostname' does not exist. Maybe 13.2 changed this)

...

...
During every zypper dup, something is writing /etc/fstab, yet, its content matches the backup.

not for me:

seife@susi:~> ls -l /etc/fstab -rw-r--r-- 1 root root 1353 23. Nov 23:10 /etc/fstab

About same here, Nov 8.

...

seife@susi:~> l /etc/*-release -rw-r--r-- 1 root root 288 8. Jan 13:11 /etc/os-release -rw-r--r-- 1 root root 165 8. Jan 13:11 /etc/SuSE-release

(last update apparently on 2015-01-08)

cer@Telcontar:~> l /etc/*-release - -rw-r--r-- 1 root root 153 Oct 23 2013 /etc/SuSE-release - -rw-r--r-- 1 root root 110 Oct 2 2013 /etc/lsb-release - -rw-r--r-- 1 root root 264 Oct 23 2013 /etc/os-release cer@Telcontar:~>

...

So it's probably easiest if you find out who is touching which file (might be a postinstall script of a package) and file a but against that.

systemtap is one possibility to get the required information.

I agree. I'm unfamiliar with that one, but apparmour is another. Do you know of a link that explains "systemtap"? - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlS+X60ACgkQtTMYHG2NR9XregCfSPNxLLrYeYgqEnHrsYgKgsGa xuwAnifjUmN3dxTSxoFfhKX2vcEEvJ1e =/65x -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Andreas Schwab

14:59

New subject: [opensuse-factory] Re: limited write cycles on filesystem media

"Carlos E. R." writes:

...

Do you know of a link that explains "systemtap"?

The wikipedia page has a lot of information. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Stefan Seyfried

15:41

Am 20.01.2015 um 15:01 schrieb Carlos E. R.:

...

On 2015-01-20 14:03, Stefan Seyfried wrote:

...
systemtap is one possibility to get the required information.

I agree.

I'm unfamiliar with that one, but apparmour is another. Do you know of a link that explains "systemtap"?

zypper in systemtap sytemtap-docs then go through /usr/share/doc/packages/systemtap/examples/ Beware: as any powerful tool, it is not for the faint of heart :) To actually use it, you also need the debuginfo matching your running kernel installed which needs quite some space: susi:~ # zypper in kernel-desktop-debuginfo Loading repository data... Reading installed packages... Resolving package dependencies... The following NEW package is going to be installed: kernel-desktop-debuginfo 1 new package to install. Overall download size: 342.0 MiB. Already cached: 0 B After the operation, additional 2.4 GiB will be used. Continue? [y/n/? shows all options] (y): n But if I really would want to know who is changing which file in /etc/ during "zypper dup", I would install it, adapt one of the examples in .../examples/io/ to just print out all file accesses under /etc/ with process name and PID and then probably trigger a "pstree -p" in userspace for everyone to find the "originating" process. It's probably doable, but quite some work. And since I cannot reproduce the problem, I cannot debug it. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Carlos E. R.

16:22

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2015-01-20 16:41, Stefan Seyfried wrote:

...

Am 20.01.2015 um 15:01 schrieb Carlos E. R.:

...
On 2015-01-20 14:03, Stefan Seyfried wrote:

...
systemtap is one possibility to get the required information.

I agree.

I'm unfamiliar with that one, but apparmour is another. Do you know of a link that explains "systemtap"?

zypper in systemtap sytemtap-docs then go through /usr/share/doc/packages/systemtap/examples/

Beware: as any powerful tool, it is not for the faint of heart :) To actually use it, you also need the debuginfo matching your running kernel installed which needs quite some space:

Ok, then apparmor is far easier: nothing is required. I can dig out the details from my notes somewhere, or Cristian could tell it: it was actually his idea, years ago ;-) Now I have to do something else. Wait... I found the note: it was filled as apparmour instead of apparmor. +++—-—-—-—-—-—-—-—-—-—-—-—- try this: # auditctl -w /dev/null -p a # auditctl -e 1 and then watch the logs... if auditctl is not found, when you execute it as root, install package "audit" Date: Fri, 06 Mar 2009 12:04:21 -0300 From: Cristian Rodríguez <> —-—-—-—-—-—-—-—-—-—-—-—-++- The above will detect a change to /dev/null. Read the manual, the options are different depending on what type of change or access you want to detect. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlS+gLEACgkQtTMYHG2NR9UpSgCdGecNzQazG9GNdKDrdLhfewFg M5wAoILtWC1DWScXMjGYHxLTV94nB2wy =iSA2 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

3391

Age (days ago)

3391

Last active (days ago)

List overview

Download

30 comments

12 participants

participants (12)

Andreas Schwab
Andrei Borzenkov
Bernhard Voelker
Carlos E. R.
Carlos E. R.
Claudio Freire
Cristian Rodríguez
Felix Miata
Matthias G. Eckermann
Stanislav Baiduzhyi
Stefan Seyfried
Stephan Kulow

[opensuse-factory] limited write cycles on filesystem media

tags

participants (12)