[opensuse] LTFS [was Re: Large Data Backups?]
On Tue, Oct 31, 2017 at 12:36 AM, Lew Wolfgang
On 10/30/2017 01:30 PM, Greg Freemyer wrote:
Are you familiar with LTFS which LTO has supported since 2010 or so:
https://en.wikipedia.org/wiki/Linear_Tape_File_System#Nature
Thanks for the pointer, Greg. I don't recall hearing about LTFS
I've never used, but I know a world class tape expert. He is using it for data archiving, but not daily backups. If you have 10 thousand plus tapes you want to retire from your enterprise, watch this video: http://www.sullivanstrickler.com/invenire.html One thing he does is migrate the data from legacy tapes to new generation LTO. LTFS is part of his solution. I did some reading/questioning.
You use your library to make a full copy of the filesystem, then as files change, you write them to the tape via LTFS.
Unfortunately, that's all I know about LTFS. Questions you need to answer:
- Does openSUSE (or your OS of choice) have support
Looks like its available on SLES.
It also looks easy to compile youself.
- Does it provide point in time recovery. Or are you stuck with only the latest version of files.
I think you're stuck with the latest versions.
Agreed, but surprised.
- What happens when a file in primary storage is deleted?
The blocks are marked "unavailable" and the space not reclaimed.
Deletion indeed seems to make the file inaccessible. Strange to me for what I assumed was part of a backup solution.
I've been using rdiff-backup for years. It works well for on-line backups. It maintains a complete current image of the source filesystem, and calculates past point-restores as diffs. I run it on a daily basis and couldn't do without it now. I can restore with one day granularity going back for as many days as I've allowed. I usually trim it at 30-days. but it could archive for years. But I don't know how to use it with tape.
[I used to use rdiff-backup years ago. Replaced it with spideroak.] That seems to be the point of LTFS. It allows solutions like rdiff-backup to write to tape without modification. It presents a filesystem interface to rdiff-backup and translates that into tape commands on the backend. LTFS seems to work best with large files. rdiff-backup makes no effort to consolidate small files, but if your files being backed up, it should perform well. Next question: With a large dataset will a single LTFS filesystem span a tapeset? Are libraries supported?
Regards, Lew
I may have to buy an older generation LTO (5? 6?) and library myself if the tape library aspect works. I have 50 TB of large files I archive to a NAS. The NAS is awkward at times. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/31/2017 04:20 AM, Greg Freemyer wrote:
That seems to be the point of LTFS. It allows solutions like rdiff-backup to write to tape without modification. It presents a filesystem interface to rdiff-backup and translates that into tape commands on the backend.
Hm... Good point, I didn't consider that.
LTFS seems to work best with large files. rdiff-backup makes no effort to consolidate small files, but if your files being backed up, it should perform well.
Yes, lots of large files. Spanning tapes with LTFS would be the big question. A quick google shows that it may be possible. I'll have to dig deeper. Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 31/10/17 15:31, Lew Wolfgang wrote:
On 10/31/2017 04:20 AM, Greg Freemyer wrote:
That seems to be the point of LTFS. It allows solutions like rdiff-backup to write to tape without modification. It presents a filesystem interface to rdiff-backup and translates that into tape commands on the backend.
Hm... Good point, I didn't consider that.
LTFS seems to work best with large files. rdiff-backup makes no effort to consolidate small files, but if your files being backed up, it should perform well.
Yes, lots of large files.
Spanning tapes with LTFS would be the big question. A quick google shows that it may be possible. I'll have to dig deeper.
If LTFS allows a raid-like solution over multiple tapes :-) it'll be similar to technology that was available long ago with tape libraries. Okay, it needed fancy robotic self-loading tapes, but I'm sure I worked with it on a Perkin-Elmer (or was it UniVac) mainframe back in the 80s. Files that weren't accessed for a while were shuffled off to tape, and when you tried to access them they were retrieved. It seems to me you're going to find LTFS a bit tricky to work into your backup solution unless you're planning just to dump incrementals to tape, in which case you're not really using LTFS as designed. But it certainly looks interesting technology. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, Oct 31, 2017 at 11:45 AM, Wols Lists
On 31/10/17 15:31, Lew Wolfgang wrote:
On 10/31/2017 04:20 AM, Greg Freemyer wrote: <snip>
LTFS seems to work best with large files. rdiff-backup makes no effort to consolidate small files, but if your files being backed up, it should perform well.
Yes, lots of large files.
Spanning tapes with LTFS would be the big question. A quick google shows that it may be possible. I'll have to dig deeper.
If LTFS allows a raid-like solution over multiple tapes :-) it'll be similar to technology that was available long ago with tape libraries.
Okay, it needed fancy robotic self-loading tapes, but I'm sure I worked with it on a Perkin-Elmer (or was it UniVac) mainframe back in the 80s. Files that weren't accessed for a while were shuffled off to tape, and when you tried to access them they were retrieved.
It seems to me you're going to find LTFS a bit tricky to work into your backup solution unless you're planning just to dump incrementals to tape, in which case you're not really using LTFS as designed. But it certainly looks interesting technology.
Wol, I'm just coming up to speed myself on LTFS. But I know rdiff-backup. rdiff-backup on day one makes a copy of every file in the folder structure being backed up. You can use any file explorer to browse the created duplicate tree and find a copy of the files as of day one. Then every time you invoke it (daily?) it compares the primary instance of the file to the copy in the rdiff-backup set and saves the set of differences. Most files won't change daily, so those files don't get anything written to the secondary media. If there is a new file or a file changes, then either a copy of the file is made, or a changeset file is written. Once you've had it running for say 30 days, you can tell rdiff-backup to do a restore to any of the various point-in-time copies you made of the dataset. By design, rdiff-backup uses a filesystem to hold the secondary copy, possibly on a low cost NAS. Let's guess Lew would need 400TB of capacity to hold his backup including a month's worth of daily incremental changesets. That's 40 10TB drives without redundancy. Or about $16K worth of disks. Easily $20K when you include a couple chasis's to hold it all. It seems at this point that by using LTFS, that secondary storage can be replaced with a LTO tape library. A tape library with 1 tape drive and enough tape capacity to hold it all should be in the same price range (or so I assume). The real question is if Lew would rather have a tape backup or a NAS backup. I personally like the ability to take a tape set offsite occasionally. Or send it to Iron Mountain. It is that offsite ability that would lead me to choosing a tape solution if they are both near the same price. Unfortunately, a single LTO-8 tape isn't big enough to hold Lew's 300TB of data. He will need to be able to concatenate multiple tapes into one logical tape. I don't know if that last requirement can be met by LTFS or not. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, 31 Oct 2017 15:45:35 +0000
Wols Lists
If LTFS allows a raid-like solution over multiple tapes :-) it'll be similar to technology that was available long ago with tape libraries.
Okay, it needed fancy robotic self-loading tapes, but I'm sure I worked with it on a Perkin-Elmer (or was it UniVac) mainframe back in the 80s. Files that weren't accessed for a while were shuffled off to tape, and when you tried to access them they were retrieved.
GEORGE 3 could do that sort of thing in the early 70s. No need for fancy tape machines - it just printed instructions on the console to tell the operator to load whichever tape it required on whichever drive it could afford to make free. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Wols Lists wrote:
On 31/10/17 15:31, Lew Wolfgang wrote:
On 10/31/2017 04:20 AM, Greg Freemyer wrote:
That seems to be the point of LTFS. It allows solutions like rdiff-backup to write to tape without modification. It presents a filesystem interface to rdiff-backup and translates that into tape commands on the backend.
Hm... Good point, I didn't consider that.
LTFS seems to work best with large files. rdiff-backup makes no effort to consolidate small files, but if your files being backed up, it should perform well.
Yes, lots of large files.
Spanning tapes with LTFS would be the big question. A quick google shows that it may be possible. I'll have to dig deeper.
If LTFS allows a raid-like solution over multiple tapes :-) it'll be similar to technology that was available long ago with tape libraries.
Okay, it needed fancy robotic self-loading tapes, but I'm sure I worked with it on a Perkin-Elmer (or was it UniVac) mainframe back in the 80s. Files that weren't accessed for a while were shuffled off to tape, and when you tried to access them they were retrieved.
On IBM, that was called HSM - Hierarchical Strorage Management. -- Per Jessen, Zürich (4.0°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (5)
-
Dave Howorth -
Greg Freemyer -
Lew Wolfgang -
Per Jessen -
Wols Lists