Securely formatting BTRFS
Hi everybody, does anyone know how to securely format a btrfs-partition[1]? I have an unused SSD with a former root partition that I have to return and I want to make sure that all data is erased before giving it back. Thank you in advance! Simon [1] Without physically destroying the SSD obviously
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
Hi everybody,
does anyone know how to securely format a btrfs-partition[1]? I have an unused SSD with a former root partition that I have to return and I want to make sure that all data is erased before giving it back.
Thank you in advance! Simon
[1] Without physically destroying the SSD obviously
Why not: cat /dev/random > /dev/yourssdhoweveritskonnected after you get a error that its full, make a new btrfs partition (gparted). (take a look before overwriting how the partition look like) simoN -- www.becherer.de
On 2023-10-25 15:41, Simon Becherer wrote:
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
Hi everybody,
Why did you post the question twice?
does anyone know how to securely format a btrfs-partition[1]? I have an unused SSD with a former root partition that I have to return and I want to make sure that all data is erased before giving it back.
Thank you in advance! Simon
[1] Without physically destroying the SSD obviously
Why not:
cat /dev/random > /dev/yourssdhoweveritskonnected after you get a error that its full,
This can be very slow. If you are interested, I have a tool to do it very fast with less random data. -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.5 (Laicolasse))
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
Hi everybody,
does anyone know how to securely format a btrfs-partition[1]? I have an unused SSD with a former root partition that I have to return and I want to make sure that all data is erased before giving it back.
The SSD controller will map arbitrary regions to arbitrary blocks of the disk, so just overwriting every block isn't working as you'd expect - usually, the controller will send writes to new, unused blocks and leave the old data alone to spread wear evenly. If you overwrite the whole disk with 0 bytes, the controller might even leave all data alone and redirect all "writes" to the same (empty) block. There are several ways to fix this: - Your BIOS might contain a function to securely erase SSDs. Try this if you bought the PC with the SSD already installed. - The manufacturer of the SSD might provide a utility for this purpose. - Send the ATA 'secure erase' command to the device. hdparm(1) can do this. - Use a tool like shred(1). This will overwrite most blocks with random data; but a bit of data will be left behind because the SSD has more capacity that you can access so it can replace bad blocks. How much that is depends on the specific device. - In the future, encrypt the SSD using hardware encryption BEFORE using it the first time. When you change or reset the hardware encryption key, there is (most likely) no way to restore the data. If hardware encryption isn't an option, at least use file system or LVM partition encryption. As long as no one can guess your key, the data is safe. See also: - https://www.thomas-krenn.com/en/wiki/Perform_a_SSD_Secure_Erase Regards, -- Aaron "Optimizer" Digulla a.k.a. Philmann Dark "It's not the universe that's limited, it's our imagination. Follow me and I'll show you something beyond the limits." http://blog.pdark.de/
Am 25.10.23 um 16:42 schrieb Aaron Digulla:
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
Hi everybody,
does anyone know how to securely format a btrfs-partition[1]? I have an unused SSD with a former root partition that I have to return and I want to make sure that all data is erased before giving it back.
The SSD controller will map arbitrary regions to arbitrary blocks of the disk, so just overwriting every block isn't working as you'd expect - usually, the controller will send writes to new, unused blocks and leave the old data alone to spread wear evenly.
Only for my understanding: (i have no deeper idear how ssd's work) i know that ssd's have !some! size more to map bad parts. but: did a ssd have several times the size that is printed on the cover (and shown if i use partimage/gparted?) this would be the only reason why overwriting all with random data would not work. - or if the ssd is intelligent and see that this is random data, and if i read it it will generate !!the same!! random data. if writing direct with cut or dd to the blocks, also writing to a huge file is (with my at the moment knowledge) a option top overwrite all. i am interested because i do not understand your explanation why this did not work (ok, maybe some small parts of the reserve parts of the ssd will with this not be overwritten) but doing the overwriting step more than one time should also in theory delete this by "spread wear evenly" simoN -- www.becherer.de
Am 25.10.23 um 17:42 schrieb Simon Becherer:
Am 25.10.23 um 16:42 schrieb Aaron Digulla:
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
Hi everybody,
does anyone know how to securely format a btrfs-partition[1]? I have an unused SSD with a former root partition that I have to return and I want to make sure that all data is erased before giving it back.
The SSD controller will map arbitrary regions to arbitrary blocks of the disk, so just overwriting every block isn't working as you'd expect - usually, the controller will send writes to new, unused blocks and leave the old data alone to spread wear evenly.
Only for my understanding: (i have no deeper idear how ssd's work) i know that ssd's have !some! size more to map bad parts. but: did a ssd have several times the size that is printed on the cover (and shown if i use partimage/gparted?)
this would be the only reason why overwriting all with random data would not work. - or if the ssd is intelligent and see that this is random data, and if i read it it will generate !!the same!! random data. if writing direct with cut or dd to the blocks, also writing to a huge file is (with my at the moment knowledge) a option top overwrite all.
i am interested because i do not understand your explanation why this did not work (ok, maybe some small parts of the reserve parts of the ssd will with this not be overwritten) but doing the overwriting step more than one time should also in theory delete this by "spread wear evenly"
simoN
well i do still not understand it. have read a lot inside the net, here is another nice explanation (in german): https://wiki.ubuntuusers.de/SSD/Secure-Erase/ simoN
-- www.becherer.de
On 2023-10-25 17:59, Simon Becherer wrote:
Am 25.10.23 um 17:42 schrieb Simon Becherer:
Am 25.10.23 um 16:42 schrieb Aaron Digulla:
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
Hi everybody,
does anyone know how to securely format a btrfs-partition[1]? I have an unused SSD with a former root partition that I have to return and I want to make sure that all data is erased before giving it back.
The SSD controller will map arbitrary regions to arbitrary blocks of the disk, so just overwriting every block isn't working as you'd expect - usually, the controller will send writes to new, unused blocks and leave the old data alone to spread wear evenly.
Only for my understanding: (i have no deeper idear how ssd's work) i know that ssd's have !some! size more to map bad parts. but: did a ssd have several times the size that is printed on the cover (and shown if i use partimage/gparted?)
this would be the only reason why overwriting all with random data would not work. - or if the ssd is intelligent and see that this is random data, and if i read it it will generate !!the same!! random data.
No :-) It might do that when sending all zeroes. Or perhaps something similar. But generating the same random data, no, that is not possible. It would then be bad quality random data.
if writing direct with cut or dd to the blocks,
also writing to a huge file is (with my at the moment knowledge) a option top overwrite all.
i am interested because i do not understand your explanation why this did not work (ok, maybe some small parts of the reserve parts of the ssd will with this not be overwritten) but doing the overwriting step more than one time should also in theory delete this by "spread wear evenly"
-- Cheers / Saludos, Carlos E. R. (from openSUSE 15.5 (Laicolasse))
Am 25.10.23 um 17:42 schrieb Simon Becherer:
Am 25.10.23 um 16:42 schrieb Aaron Digulla:
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
Hi everybody,
does anyone know how to securely format a btrfs-partition[1]? I have an unused SSD with a former root partition that I have to return and I want to make sure that all data is erased before giving it back.
The SSD controller will map arbitrary regions to arbitrary blocks of the disk, so just overwriting every block isn't working as you'd expect - usually, the controller will send writes to new, unused blocks and leave the old data alone to spread wear evenly.
Only for my understanding: (i have no deeper idear how ssd's work) i know that ssd's have !some! size more to map bad parts. but: did a ssd have several times the size that is printed on the cover (and shown if i use partimage/gparted?)
Sorry, I didn't phrase that well. The SSD will have a small percentage (say 1-5%, I don't know for sure) of blocks as spares. Unlike normal hard disks, those spares are used all the time. In a traditional hard disk, when a block starts to fail, the controller will map a spare. So they are kept only for emergencies. With SSDs, the controller remembers how often each block was written to. When you write new data, it will find a free block with the least usage and write the data there and remember "I saved the data for block 15 in the real memory address 0x... + increment the usage counter; the old real memory is now in the free pool". That way, every memory block will get roughly the same number of writes, even if you write block 15 a million times. So when you write random data to the whole disk, the disk will look as it was erased but behind the scenes, the most used blocks will have been replaced by least used ones and some of the old data is still there. Usually, this is no problem unless you have strict security related rules. For example, there is a danger that small documents might fit into a single block and survive. That means somewhere in the storage hardware, you can still find some of your data. Note that there is no way to read this data using normal ATA commands. Only the controller knows where those are. But for high security scenarios, this is not acceptable because attackers will open the case and read the chips directly and then see everything. So for most people, overwriting the whole disk with random data should work well enough. The drawback is that this is slow and adds a lot of wear to the cells since the controller can't optimize this "erase" operation - it can't tell what you're trying to achieve. That is why the "secure erase" command exists. It allows to SSD controller to do this fast and secure way, whatever that means for your specific hardware. This should also clean any spare blocks, so nothing is left behind. Regards, -- Aaron "Optimizer" Digulla a.k.a. Philmann Dark "It's not the universe that's limited, it's our imagination. Follow me and I'll show you something beyond the limits." http://blog.pdark.de/
On 2023-10-25 20:47, Aaron Digulla wrote:
Am 25.10.23 um 17:42 schrieb Simon Becherer:
Am 25.10.23 um 16:42 schrieb Aaron Digulla:
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
...
Sorry, I didn't phrase that well.
The SSD will have a small percentage (say 1-5%, I don't know for sure) of blocks as spares. Unlike normal hard disks, those spares are used all the time. In a traditional hard disk, when a block starts to fail, the controller will map a spare. So they are kept only for emergencies.
With SSDs, the controller remembers how often each block was written to. When you write new data, it will find a free block with the least usage and write the data there and remember "I saved the data for block 15 in the real memory address 0x... + increment the usage counter; the old real memory is now in the free pool". That way, every memory block will get roughly the same number of writes, even if you write block 15 a million times.
Do you know where that metadata is stored? In the disk itself would cause a lot of wear. Some type of permanent RAM? ... -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.5 (Laicolasse))
On Wed, Oct 25, 2023 at 9:58 PM Carlos E. R. <robin.listas@telefonica.net> wrote:
On 2023-10-25 20:47, Aaron Digulla wrote:
Am 25.10.23 um 17:42 schrieb Simon Becherer:
Am 25.10.23 um 16:42 schrieb Aaron Digulla:
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
...
Sorry, I didn't phrase that well.
The SSD will have a small percentage (say 1-5%, I don't know for sure) of blocks as spares. Unlike normal hard disks, those spares are used all the time. In a traditional hard disk, when a block starts to fail, the controller will map a spare. So they are kept only for emergencies.
With SSDs, the controller remembers how often each block was written to. When you write new data, it will find a free block with the least usage and write the data there and remember "I saved the data for block 15 in the real memory address 0x... + increment the usage counter; the old real memory is now in the free pool". That way, every memory block will get roughly the same number of writes, even if you write block 15 a million times.
Do you know where that metadata is stored?
And it matters exactly how?
In the disk itself would cause a lot of wear.
The total amount of TBW (terabytes written) promised by the vendor accounts for any internal overhead for bookkeeping, translation layer, write levelling, garbage collection, write amplification etc. You never get to see this extra overhead anyway nor do you know what is the physical capacity of your SSD device, because you never use raw flash, but the whole device together with its controller and algorithms to manage raw flash.
Some type of permanent RAM?
Am 26.10.23 um 08:42 schrieb Andrei Borzenkov:
On Wed, Oct 25, 2023 at 9:58 PM Carlos E. R. <robin.listas@telefonica.net> wrote:
On 2023-10-25 20:47, Aaron Digulla wrote:
Am 25.10.23 um 17:42 schrieb Simon Becherer:
Am 25.10.23 um 16:42 schrieb Aaron Digulla:
With SSDs, the controller remembers how often each block was written to. When you write new data, it will find a free block with the least usage and write the data there and remember "I saved the data for block 15 in the real memory address 0x... + increment the usage counter; the old real memory is now in the free pool". That way, every memory block will get roughly the same number of writes, even if you write block 15 a million times.
Do you know where that metadata is stored?
And it matters exactly how?
because maybe he did not like to die as a stupid person and is interested in how it works. i am also, i do like to know things, often regardless of the actual situation. i do not like "black boxes" so this question is in my opinion what physical hardware is used for this part of storing data who will change very often. simoN -- www.becherer.de
On 2023-10-26 08:42, Andrei Borzenkov wrote:
On Wed, Oct 25, 2023 at 9:58 PM Carlos E. R. <robin.listas@telefonica.net> wrote:
On 2023-10-25 20:47, Aaron Digulla wrote:
Am 25.10.23 um 17:42 schrieb Simon Becherer:
Am 25.10.23 um 16:42 schrieb Aaron Digulla:
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
...
Sorry, I didn't phrase that well.
The SSD will have a small percentage (say 1-5%, I don't know for sure) of blocks as spares. Unlike normal hard disks, those spares are used all the time. In a traditional hard disk, when a block starts to fail, the controller will map a spare. So they are kept only for emergencies.
With SSDs, the controller remembers how often each block was written to. When you write new data, it will find a free block with the least usage and write the data there and remember "I saved the data for block 15 in the real memory address 0x... + increment the usage counter; the old real memory is now in the free pool". That way, every memory block will get roughly the same number of writes, even if you write block 15 a million times.
Do you know where that metadata is stored?
And it matters exactly how?
I just would like to know :-)
In the disk itself would cause a lot of wear.
The total amount of TBW (terabytes written) promised by the vendor accounts for any internal overhead for bookkeeping, translation layer, write levelling, garbage collection, write amplification etc. You never get to see this extra overhead anyway nor do you know what is the physical capacity of your SSD device, because you never use raw flash, but the whole device together with its controller and algorithms to manage raw flash.
The remapping table would be just one number per sector, but one full rewrite of the table each time a sector in the disk changes (or maybe not, it is several sectors in size). I thought that would be too much wear. And the table itself would be subject to remapping? Then how can the firmware find it? -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.5 (Laicolasse))
On Thu, Oct 26, 2023 at 1:47 PM Carlos E. R. <robin.listas@telefonica.net> wrote:
The remapping table would be just one number per sector, but one full rewrite of the table each time a sector in the disk changes (or maybe not, it is several sectors in size). I thought that would be too much wear. And the table itself would be subject to remapping? Then how can the firmware find it?
It could use self-identifying flash format and scan it on power-on to build the translation table. Translation table itself would be kept in RAM to speed up access. This would mean overhead of one block per erase page but no extra writes.
On 2023-10-26 14:47, Andrei Borzenkov wrote:
On Thu, Oct 26, 2023 at 1:47 PM Carlos E. R. <robin.listas@telefonica.net> wrote:
The remapping table would be just one number per sector, but one full rewrite of the table each time a sector in the disk changes (or maybe not, it is several sectors in size). I thought that would be too much wear. And the table itself would be subject to remapping? Then how can the firmware find it?
It could use self-identifying flash format and scan it on power-on to build the translation table. Translation table itself would be kept in RAM to speed up access. This would mean overhead of one block per erase page but no extra writes.
That's what I thought initially, ram. But at some point, it has to be written to some permanent storage. I suppose periodically. Can the media keep enough charge in capacitors to back up the last changes when power fails suddenly? For instance, running "smartctl -a /dev/nvme0n1", one of the lines I noticed says: Power Cycles: 398 Power On Hours: 2,321 Unsafe Shutdowns: 47 So "hard" poweroff can be dangerous, cause some kind of corruption, perhaps. -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.5 (Laicolasse))
Am 25.10.23 um 20:47 schrieb Aaron Digulla:
Am 25.10.23 um 17:42 schrieb Simon Becherer:
Am 25.10.23 um 16:42 schrieb Aaron Digulla:
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
Hi everybody,
does anyone know how to securely format a btrfs-partition[1]? I have an unused SSD with a former root partition that I have to return and I want to make sure that all data is erased before giving it back.
The SSD controller will map arbitrary regions to arbitrary blocks of the disk, so just overwriting every block isn't working as you'd expect - usually, the controller will send writes to new, unused blocks and leave the old data alone to spread wear evenly.
Only for my understanding: (i have no deeper idear how ssd's work) i know that ssd's have !some! size more to map bad parts. but: did a ssd have several times the size that is printed on the cover (and shown if i use partimage/gparted?)
Sorry, I didn't phrase that well.
The SSD will have a small percentage (say 1-5%, I don't know for sure) of blocks as spares. Unlike normal hard disks, those spares are used all the time. In a traditional hard disk, when a block starts to fail, the controller will map a spare. So they are kept only for emergencies.
With SSDs, the controller remembers how often each block was written to. When you write new data, it will find a free block with the least usage and write the data there and remember "I saved the data for block 15 in the real memory address 0x... + increment the usage counter; the old real memory is now in the free pool". That way, every memory block will get roughly the same number of writes, even if you write block 15 a million times.
So when you write random data to the whole disk, the disk will look as it was erased but behind the scenes, the most used blocks will have been replaced by least used ones and some of the old data is still there. Usually, this is no problem unless you have strict security related rules. For example, there is a danger that small documents might fit into a single block and survive. That means somewhere in the storage hardware, you can still find some of your data.
Note that there is no way to read this data using normal ATA commands. Only the controller knows where those are. But for high security scenarios, this is not acceptable because attackers will open the case and read the chips directly and then see everything.
So for most people, overwriting the whole disk with random data should work well enough. The drawback is that this is slow and adds a lot of wear to the cells since the controller can't optimize this "erase" operation - it can't tell what you're trying to achieve.
That is why the "secure erase" command exists. It allows to SSD controller to do this fast and secure way, whatever that means for your specific hardware. This should also clean any spare blocks, so nothing is left behind.
Regards,
Thanks, thats a good explanation. simoN -- www.becherer.de
On 2023-10-25 16:42, Aaron Digulla wrote:
Am 25.10.23 um 15:36 schrieb Simon Heimbach:
Hi everybody,
does anyone know how to securely format a btrfs-partition[1]? I have an unused SSD with a former root partition that I have to return and I want to make sure that all data is erased before giving it back.
The SSD controller will map arbitrary regions to arbitrary blocks of the disk, so just overwriting every block isn't working as you'd expect - usually, the controller will send writes to new, unused blocks and leave the old data alone to spread wear evenly.
If you overwrite the whole disk with 0 bytes, the controller might even leave all data alone and redirect all "writes" to the same (empty) block.
There are several ways to fix this:
- Your BIOS might contain a function to securely erase SSDs. Try this if you bought the PC with the SSD already installed.
- The manufacturer of the SSD might provide a utility for this purpose.
- Send the ATA 'secure erase' command to the device. hdparm(1) can do this.
- Use a tool like shred(1). This will overwrite most blocks with random data; but a bit of data will be left behind because the SSD has more capacity that you can access so it can replace bad blocks. How much that is depends on the specific device.
Don't do this on an SSD, it kill lives (shred writes several times). On Flash media it is enough writing once.
- In the future, encrypt the SSD using hardware encryption BEFORE using it the first time. When you change or reset the hardware encryption key, there is (most likely) no way to restore the data. If hardware encryption isn't an option, at least use file system or LVM partition encryption. As long as no one can guess your key, the data is safe.
Have you used hardware encryption in Linux? Or know anyone that has? There are practicalities as the BIOS asking for the password on boot or not, and being able to read or not the disk on a different computer (if the computer breaks). The BIOS can add some arbitrary text of its own the your password, so that the disk doesn't work outside.
See also:
- https://www.thomas-krenn.com/en/wiki/Perform_a_SSD_Secure_Erase
-- Cheers / Saludos, Carlos E. R. (from openSUSE 15.5 (Laicolasse))
participants (5)
-
Aaron Digulla -
Andrei Borzenkov -
Carlos E. R. -
Simon Becherer -
Simon Heimbach