On 10/10/22 19:39, Robert Webb wrote:
* Patrick Shanahan <paka@opensuse.org> [10-10-22 17:48]:
* Carlos E. R. <robin.listas@telefonica.net> [10-10-22 17:25]:
On 2022-10-10 22:43, Patrick Shanahan wrote:
* Lew Wolfgang <> [10-10-22 15:30]:
On 10/10/22 12:07, David C. Rankin wrote: > On 10/7/22 18:01, Lew Wolfgang wrote: ...
> There is a coreutils utility called 'split' that allows you to slice > large files into smaller files by lines or bytes and it also has a > --filter=command option to filter through a shell script which would > give you the ability to umount/mount new JBOD devices between parts of > the original being written. Yes, split is great and would work, but we're not going to need it this time. and once more, dar will accomplish all that for you very quickly, especially using the gui AFAIK, dar doesn't store full independent files on the disk, that can be just read or copied directly from the (backup) disk without using dar for recovery.
AFAIU He wants the files saved as the same files, complete, not split, not changed. DAR will store files rather than compress files. files stored are identical copies, complete and not split unless told to do so. but "dar" is necessary to access the files. Mergerfs [1], a union, fuse filesystem, provides some of the needed features. It merges an assortment of filesystems (your archive disks)
On Mon, 10 Oct 2022 18:12:14 -0400, Patrick Shanahan wrote: that can be mounted and unmounted "at will". When a filesystem is not part of the union, it can be used normally, without running mergerfs. You would read and write whole files, not pieces.
It does not prompt when a filesystem fills up, but it can be configured to switch to writing to the next in priority filesystem when the free space is too small (your 7GB max filesize) by setting the 'minfreespace' option.
So, for your process of archiving to a pile of disks, if you somehow detect when writing has transitioned to another disk, you could unmount the first one, prompt for the next, join it to the union, and then wait for another disk to get filled up by your writing process. This depends on configuring a write policy for mergerfs that, instead of trying to even out the percent full-ness of the component disks, gives priority to a disk that already has more on it.
[1] https://github.com/trapexit/mergerfs -- Robert Webb
Thanks for the pointer, Robert! I've never used anything like this, but I did dabble with zfs once. Does mergerfs allow each of the drives in a pool to be mounted individually on another host? That's one of our requirements, that the drives not be aggregated or raided together. Our requirement isn't for archiving, but for data dissemination to possibly non-Linux systems not administrated by us. Regards, Lew