On Mon, Oct 30, 2017 at 3:35 PM, Lew Wolfgang <wolfgang@sweet-haven.com> wrote:
Hi Folks,
BLUF: How to backup large data stores?
Say I've got 300-TB of on-line data, most if it is fairly static, while some of it changes frequently on a daily basis. I do daily incremental backups on the dynamic areas to disk filesystems on separate computers, with a monthly full disk image being stored off-line at an off-site location. But I'm worried about the bulk of the data that isn't being backed up.
Most of the data consists of lots of binary files stored on multiple hardware RAID-6 arrays. The arrays have hot-spare disks, and I've got spares on the shelf. But as the graybeards know, reliable RAID isn't backup!
One technology being considered is LTO tape. LTO-8 is due out any day now that claims to store 12-TB native. A drive or two with a tape library would possibly fill the requirement.
Are you familiar with LTFS which LTO has supported since 2010 or so: https://en.wikipedia.org/wiki/Linear_Tape_File_System#Nature You use your library to make a full copy of the filesystem, then as files change, you write them to the tape via LTFS. Unfortunately, that's all I know about LTFS. Questions you need to answer: - Does openSUSE (or your OS of choice) have support - Does it provide point in time recovery. Or are you stuck with only the latest version of files. - What happens when a file in primary storage is deleted?
Does anyone have any thoughts/advice? What do clouds like Google and AWS do for backups?
I don't think they do. They are highly redundant. Google does resiliency testing by turning off a a rack of computers at a time to see if any data is lost. That is an intentional level of testing, or so I understand. They do unintentional testing at the DC level. What happens when a full DC has an outage? I can't recall how often that happens, but I think it does happen from time to time. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org