Re: [opensuse] HP Ultrium LTO Compression
the tar -t command only lists the files. Not the sizes.
but appending --totals shows the total at the end of the tar output
4352256 (4.1GiB)
du -s /srv/media/testfiles
4250656 4.1G
So now the files are bigger? Obviously not working still
mounting the tape device results in:
mount: /dev/st3 is not a block device
and it does not mount.
using mt status: Tape block size 0 bytes. Density code 0x44
The density code of the lto2 0x42 and lto1 0x40
does this change if the drive is compressed?
output of stinit -v -v:
Trying to open database '/etc/stinit.def'.
Open succeeded.
stinit, processing tape 0
Mode 1, name '/dev/nst0'
Mode 2, name '/dev/nst0l'
Mode 3, name '/dev/nst0m'
Mode 4, name '/dev/nst0a'
The manufacturer is 'HP', product is 'C7438A', and revision 'V312'.
Mode 1 definition: compression=1
stinit, processing tape 1
Mode 1, name '/dev/nst1'
Mode 2, name '/dev/nst1l'
Mode 3, name '/dev/nst1m'
Mode 4, name '/dev/nst1a'
The manufacturer is 'HP', product is 'Ultrium 1-SCSI', and revision 'P61D'.
Mode 1 definition: compression=1
stinit, processing tape 2
Mode 1, name '/dev/nst2'
Mode 2, name '/dev/nst2l'
Mode 3, name '/dev/nst2m'
Mode 4, name '/dev/nst2a'
The manufacturer is 'HP', product is 'Ultrium 2-SCSI', and revision 'S65D'.
Mode 1 definition: compression=1
stinit, processing tape 3
Mode 1, name '/dev/nst3'
Mode 2, name '/dev/nst3l'
Mode 3, name '/dev/nst3m'
Mode 4, name '/dev/nst3a'
The manufacturer is 'HP', product is 'Ultrium 3-SCSI', and revision 'D22D'.
Mode 1 definition: compression=1
stinit, processing tape 4
Initialized 4 tape devices.
On 26 April 2016 at 23:59, Ritchie Fraser
On Tuesday 26 Apr 2016 21:43:52 you wrote:
Hi Ritchie
Thanks, so I have made a /etc/stinit.def (Below). and I ran stinit as
root which said Initialized 4 tape devices.
So now if I run tar -V test-tape -cvWf /dev/st3 /srv/some/data then
should compression work?
Tar options being
-V label
-c create
-v verbose
-W verify
-f file
Is there quicker a way to test if compression is on other that waiting
several hours for a backup?
Backup a smaller directory of known size.
Then list tape contents using tar
-t or --list option
(this may just list the files in the current archive file (volume).
Alternatively mount the tape in your file system
mount /dev/st0 /mnt/tape
Then just
ls -lh
Compare the size of the archive file to the size of the directory
Also, do you know how I can tell tar to split the backup once the tape
is full and ask for another tape?
Use -M or --multi-volume options
( https://www.gnu.org/software/tar/manualhtml_section/tar_79.html )
Thanks
Paul
cat /etc/stinit.def
#DAT72
manufaturer=HP model = "C7438A" {
mode1 compression=1
}
#LTO1
manufaturer=HP model = "Ultrium 1-SCSI" {
mode1 compression=1
}
#LTO2
manufaturer=HP model = "Ultrium 2-SCSI" {
mode1 compression=1
}
#LTO3
manufaturer=HP model = "Ultrium 3-SCSI" {
mode1 compression=1
}
On 22 April 2016 at 23:10, Ritchie Fraser
wrote: My reply to the [opensuse] list was bounced :-(...
So I've forwarded it (below)
R
---------- Forwarded Message ----------
Subject: Re: [opensuse] HP Ultrium LTO Compression
Date: Friday 22 Apr 2016, 21:29:21
From: Ritchie Fraser
To: opensuse@opensuse.org
On Friday 22 Apr 2016 13:33:23 Per Jessen wrote:
Greg Freemyer wrote:
I don't recall if you have to add a call stinit during bootup, or if
it is automatic.
I've never had to do it.
Hi Paul,
I remember looking up info for the same issue at a previous company I
worked for... They were using Bacula and the first two links below are
for Bacula documentation...
Ensuring that the Tape Modes Are Properly Set - Linux Only
http://www.bacula.org/5.2.x-manuals/en/problems/problems/Testing_Your_Tape
_Drive.html#SECTION00435000000000000000 [1]
Tape Hardware Compression and Blocking Size
http://www.bacula.org/5.2.x-manuals/en/problems/problems/Testing_Your_Tape
_Drive.html#SECTION00436000000000000000 [2]
Backup tape compression
http://serverfault.com/questions/424936/backup-tape-compression [3]
Open Source Backup - Hardware compression
https://wiki.zmanda.com/index.php/Hardware_compression#Tape_drive_v.s._Lin
ux_kernel_driver_compression_settings [4]
--
Kind Regards
Ritchie
--------
[1]
http://www.bacula.org/5.2.x-manuals/en/problems/problems/Testing_Your_Tape
_Drive.html#SECTION00435000000000000000
[2]
http://www.bacula.org/5.2.x-manuals/en/problems/problems/Testing_Your_Tape
_Drive.html#SECTION00436000000000000000
[3] http://serverfault.com/questions/424936/backup-tape-compression
[4]
https://wiki.zmanda.com/index.php/Hardware_compression#Tape_drive_v.s._Lin
ux_kernel_driver_compression_settings
-----------------------------------------
--
Kind Regards
Ritchie
Ritchie Fraser BSc (Hons)
Home: +44 (1303) 891767
Mobile: +44 (7867) 976708
--
Kind Regards
Ritchie
Ritchie Fraser BSc (Hons)
Home: +44 (1303) 891767
Mobile: +44 (7867) 976708
Web: http://www.ritchiefraser.co.uk -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Apr 27, 2016 at 1:08 PM, Paul Groves
the tar -t command only lists the files. Not the sizes. but appending --totals shows the total at the end of the tar output
4352256 (4.1GiB)
du -s /srv/media/testfiles
4250656 4.1G
So now the files are bigger? Obviously not working still
Not sure what you are doing, but hardware compression is transparent to tar. It has no idea how much data is on the tape or how much physical space it taking up. The --totals just shows how much data was send to the tape drive, not how much space it took on the tape media. Try this and see if it reports a difference. Setup a compression mode and a non-compression mode as the first 2 modes: Does just calling status show a difference between the compressed and uncompressed mode: mt -f /dev/nst0 status mt -f /dev/nst0l status If not, maybe the block location after writing your data to tape will help. # test mode 1 mt -f /dev/st0 rew tar -cf /dev/nst0 /srv/media/testfiles # note the use of the no-rewind device mt -f /dev/nst0 status # test mode 2 mt -f /dev/st0l rew tar -cf /dev/nst0l /srv/media/testfiles # note the use of the no-rewind device mt -f /dev/nst0l status Does the block number vary? If neither of those do the trick, then I think you will have to fill a tape up and see how far you get before you get an End of Tape message. fyi: For keeping high-speed tape drives running, I really like the "mbuffer" tool. It is part of the opensuse distro: https://build.opensuse.org/package/show/utilities/mbuffer Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-04-27 22:00, Greg Freemyer wrote:
On Wed, Apr 27, 2016 at 1:08 PM, Paul Groves <> wrote:
Not sure what you are doing, but hardware compression is transparent to tar. It has no idea how much data is on the tape or how much physical space it taking up.
The --totals just shows how much data was send to the tape drive, not how much space it took on the tape media.
There must be a way to count how many centimetres of tape length have been used :-? - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlchG9wACgkQja8UbcUWM1zI0wD/VLzcK/i6Nt0S3AOHzZ+8dZ2t l6yn4jCVkm8XrD2cD4IA/2ZyFfqlyIhBMndh5Y5Yj5aB5sCHcEnOShDah6gsnHJZ =Fj6P -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Not sure what you are doing, but hardware compression is transparent to tar. It has no idea how much data is on the tape or how much physical space it taking up.
Suspected as much. Thanks Greg.
Does anyone know how can I check the real size of the data on the tape?
Also, I have tried running a backup until the end of tape message:
I am using an LTO3 tape which is 400GB native (up to 800GB) so I would
expect somewhere between these values.
The directory contains many un-compressed media files so should be
compressible, and it is only 368GB in size.
It stops at around about 250-300GB mark by my estimation, which was my
original problem. It isn;t even using anywhere near it's native size.
I regularly fit 300 - 370GB on an LTO2 tape at work with Symantec
backup exec! I shouldn't need an LTO3 here (depending on the
compression rate then I might go a few GB over).
I have to say, this is not a well documented process. Hopefully
someone has used this successfully before?
P.S. Same issue with LTO4, LTO5 and LTO6 drives at work so this is
obviously a common problem. All the drives are HP Ultrium
Storageworks.
On 27 April 2016 at 21:06, Carlos E. R.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2016-04-27 22:00, Greg Freemyer wrote:
On Wed, Apr 27, 2016 at 1:08 PM, Paul Groves <> wrote:
Not sure what you are doing, but hardware compression is transparent to tar. It has no idea how much data is on the tape or how much physical space it taking up.
The --totals just shows how much data was send to the tape drive, not how much space it took on the tape media.
There must be a way to count how many centimetres of tape length have been used :-?
- -- Cheers / Saludos,
Carlos E. R.
(from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux)
iF4EAREIAAYFAlchG9wACgkQja8UbcUWM1zI0wD/VLzcK/i6Nt0S3AOHzZ+8dZ2t l6yn4jCVkm8XrD2cD4IA/2ZyFfqlyIhBMndh5Y5Yj5aB5sCHcEnOShDah6gsnHJZ =Fj6P -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Apr 28, 2016 at 12:02 PM, Paul Groves
Not sure what you are doing, but hardware compression is transparent to tar. It has no idea how much data is on the tape or how much physical space it taking up.
Suspected as much. Thanks Greg.
Does anyone know how can I check the real size of the data on the tape?
Also, I have tried running a backup until the end of tape message:
I am using an LTO3 tape which is 400GB native (up to 800GB) so I would expect somewhere between these values. The directory contains many un-compressed media files so should be compressible, and it is only 368GB in size. It stops at around about 250-300GB mark by my estimation, which was my original problem. It isn;t even using anywhere near it's native size.
I regularly fit 300 - 370GB on an LTO2 tape at work with Symantec backup exec! I shouldn't need an LTO3 here (depending on the compression rate then I might go a few GB over).
I have to say, this is not a well documented process. Hopefully someone has used this successfully before?
What is the block size your backup software using to write data to the tape? The block size will effect not only the rate at which data can/will be written to tape but also the amount of data which can be written to a single tape. In a past life I used Amanda to backup systems to LTO2 and LTO3 libraries and with the default 32k block size saw horrible performance, nowhere near what the devices could provide. After much trial and error determined that setting a block size of 128M or 256m, it escapes me, was needed for the drives to perform at their maximum rate. This certainly limited the amount of data I would write to tape, but didn't care too much since these were attached to like 12 tape libraries.
P.S. Same issue with LTO4, LTO5 and LTO6 drives at work so this is obviously a common problem. All the drives are HP Ultrium Storageworks.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
28.04.2016 19:02, Paul Groves пишет:
Not sure what you are doing, but hardware compression is transparent to tar. It has no idea how much data is on the tape or how much physical space it taking up.
Suspected as much. Thanks Greg.
Does anyone know how can I check the real size of the data on the tape?
Also, I have tried running a backup until the end of tape message:
I am using an LTO3 tape which is 400GB native (up to 800GB) so I would expect somewhere between these values. The directory contains many un-compressed media files so should be compressible, and it is only 368GB in size. It stops at around about 250-300GB mark by my estimation, which was my original problem. It isn;t even using anywhere near it's native size.
Amount of data that fits on tape depends on too many factors. For well compressible data and when data could be streamed I have often got much better results than 2:1. Minimal speed that is required to keep LTO3 in streaming mode is around 27MB/s. If your application cannot keep up with it, your tape consists more of gaps than actual data.
Another question.
tar -M is great, it splits across two tapes
How to I tell it to start writing to the secind tape which is in a
different drive?
e.g. when my lto3 is full write the remaining data to the lto1 drive
On 29 April 2016 at 04:36, Andrei Borzenkov
28.04.2016 19:02, Paul Groves пишет:
Not sure what you are doing, but hardware compression is transparent to tar. It has no idea how much data is on the tape or how much physical space it taking up.
Suspected as much. Thanks Greg.
Does anyone know how can I check the real size of the data on the tape?
Also, I have tried running a backup until the end of tape message:
I am using an LTO3 tape which is 400GB native (up to 800GB) so I would expect somewhere between these values. The directory contains many un-compressed media files so should be compressible, and it is only 368GB in size. It stops at around about 250-300GB mark by my estimation, which was my original problem. It isn;t even using anywhere near it's native size.
Amount of data that fits on tape depends on too many factors. For well compressible data and when data could be streamed I have often got much better results than 2:1.
Minimal speed that is required to keep LTO3 in streaming mode is around 27MB/s. If your application cannot keep up with it, your tape consists more of gaps than actual data.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Did you try to read manual? E.g. by giving different device name when prompted for media change. Отправлено с iPhone
29 апр. 2016 г., в 21:33, Paul Groves
написал(а): Another question.
tar -M is great, it splits across two tapes
How to I tell it to start writing to the secind tape which is in a different drive?
e.g. when my lto3 is full write the remaining data to the lto1 drive
On 29 April 2016 at 04:36, Andrei Borzenkov
wrote: 28.04.2016 19:02, Paul Groves пишет: Not sure what you are doing, but hardware compression is transparent to tar. It has no idea how much data is on the tape or how much physical space it taking up.
Suspected as much. Thanks Greg.
Does anyone know how can I check the real size of the data on the tape?
Also, I have tried running a backup until the end of tape message:
I am using an LTO3 tape which is 400GB native (up to 800GB) so I would expect somewhere between these values. The directory contains many un-compressed media files so should be compressible, and it is only 368GB in size. It stops at around about 250-300GB mark by my estimation, which was my original problem. It isn;t even using anywhere near it's native size.
Amount of data that fits on tape depends on too many factors. For well compressible data and when data could be streamed I have often got much better results than 2:1.
Minimal speed that is required to keep LTO3 in streaming mode is around 27MB/s. If your application cannot keep up with it, your tape consists more of gaps than actual data. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Apr 28, 2016 at 11:36 PM, Andrei Borzenkov
Minimal speed that is required to keep LTO3 in streaming mode is around 27MB/s. If your application cannot keep up with it, your tape consists more of gaps than actual data.
fact? old wive's tale? I'm aware of the speed inefficiency with not keeping the drive buffers full, but not with the introduction of gaps. All I know about is the tape having to rewind over and over again (shoe-shining). Greg -- Greg Freemyer www.IntelligentAvatar.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Отправлено с iPhone
29 апр. 2016 г., в 22:32, Greg Freemyer
написал(а): On Thu, Apr 28, 2016 at 11:36 PM, Andrei Borzenkov
wrote: Minimal speed that is required to keep LTO3 in streaming mode is around 27MB/s. If your application cannot keep up with it, your tape consists more of gaps than actual data. fact?
old wive's tale?
I won't eat my hat, but LTO is written in fixed size chunks (DS - data sets) separated by variable length gaps (DSS - data set separator). Length of DSS between DS in streaming mode is significantly less than length of DSS between appended data. Also if there is not enough data to fill DS, part of it is written unused. I have seen statements that no back hitching happens for short time interruption. That sounds plausible, given all negative effects of it. Finally tape drive cannot wait indefinitely for new data; at some point it needs to commit what it got so far. Depending on pattern it may result in large amount of very small appends. In my experience tapes used for large number of short term sessions were filled significantly faster than tapes with small number of large streams.
I'm aware of the speed inefficiency with not keeping the drive buffers full, but not with the introduction of gaps. All I know about is the tape having to rewind over and over again (shoe-shining).
Greg -- Greg Freemyer www.IntelligentAvatar.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (5)
-
Andrei Borzenkov
-
Carlos E. R.
-
Darin Perusich
-
Greg Freemyer
-
Paul Groves