| Author |
Message |
Charles Morrall
Guest
|
 Posted:
Fri Dec 10, 2004 1:21 pm Post subject:
Software vs hardware compression in backups |
|
|
Backing up to disk is growing in popularity, and I for one certainly
recommend it in most cases. Another theory of mine is that it's more
favourable to use software compression when configuring the backup job
backing up to disk, and to configure the subsequent staging to tape with no
compression. I base this theory on the assumption that the compression
algorithms are well known, and essentially it's the same process regardless
if the algorithm is implemented in hardware in the tape drive or software in
the backup application. I doubt there's a an ASIC in the tape drive
dedicated to handle compression though. Also, it's probably safe to assume
that processing and memory capacity of a backup server is vastly larger that
of a tape drive even considering the server has to do many other things I've
come to the conclusion that using compression in software has a better
chance of compressing the data stream going to disk than a tape drive has,
when it has to try to stream the tape at the same time. With LTO-3 around
the corner with a native speed of 80MB/s, I doubt you could make the drive
stream with compression anyway in most cases.
Therefore it makes more sense to me to compress the incoming data stream to
disk, and let the tape drive stream the already compressed data using no
compression.
Your comments are, as always, much appreciated.
/charles |
|
| Back to top |
|
 |
Yura Pismerov
Guest
|
| Posted:
Fri Dec 10, 2004 7:14 pm Post subject:
Re: Software vs hardware compression in backups |
|
|
I think it really depends on your particular setup - speed of the network, backup clients, tape drives, capacity of your backup
system, etc.
It also depends on your data. From my personal experience, running software (client side) compression was always better for some
reason - smaller backup frames and less tape media used.
But I should admit that it was not speediest network and tape drives in my case (Legato Networker in mixed gigabit/100MB network and
(4) AIT-3 drives).
Just my $0.02
Charles Morrall wrote:
| Quote: | Backing up to disk is growing in popularity, and I for one certainly
recommend it in most cases. Another theory of mine is that it's more
favourable to use software compression when configuring the backup job
backing up to disk, and to configure the subsequent staging to tape with no
compression. I base this theory on the assumption that the compression
algorithms are well known, and essentially it's the same process regardless
if the algorithm is implemented in hardware in the tape drive or software in
the backup application. I doubt there's a an ASIC in the tape drive
dedicated to handle compression though. Also, it's probably safe to assume
that processing and memory capacity of a backup server is vastly larger that
of a tape drive even considering the server has to do many other things I've
come to the conclusion that using compression in software has a better
chance of compressing the data stream going to disk than a tape drive has,
when it has to try to stream the tape at the same time. With LTO-3 around
the corner with a native speed of 80MB/s, I doubt you could make the drive
stream with compression anyway in most cases.
Therefore it makes more sense to me to compress the incoming data stream to
disk, and let the tape drive stream the already compressed data using no
compression.
Your comments are, as always, much appreciated.
/charles
|
|
|
| Back to top |
|
|
Rob Turk
Guest
|
| Posted:
Fri Dec 10, 2004 11:23 pm Post subject:
Re: Software vs hardware compression in backups |
|
|
"Charles Morrall" <charles.morrall@telia.com> wrote in message
news:h0dud.10631$d5.94554@newsb.telia.net...
| Quote: | Backing up to disk is growing in popularity, and I for one certainly
recommend it in most cases. Another theory of mine is that it's more
favourable to use software compression when configuring the backup job
backing up to disk, and to configure the subsequent staging to tape with
no compression.
|
Host based compression is usually more efficient because there's more buffer
memory for the host to play with and to anticipate data patterns then in the
tape drive itself. Also, when compression happens at the client before being
moved to the backup server and tape drive then it helps to eliminate
bottleneck elsewhere in the chain.
| Quote: | I base this theory on the assumption that the compression algorithms are
well known, and essentially it's the same process regardless if the
algorithm is implemented in hardware in the tape drive or software in the
backup application. I doubt there's a an ASIC in the tape drive dedicated
to handle compression though.
|
That's a wrong assumption. ALDC/IDRC/LZW compression chips have been
available for years and are in common use by tape drive manufacturers. I
don't know of any tape drive doing it's compression in firmware, it's always
a dedicated ASIC or part of a larger buffer management chip.
| Quote: | Also, it's probably safe to assume that processing and memory capacity of
a backup server is vastly larger that of a tape drive even considering the
server has to do many other things I've come to the conclusion that using
compression in software has a better chance of compressing the data stream
going to disk than a tape drive has, when it has to try to stream the tape
at the same time.
|
Agreed.
| Quote: | With LTO-3 around the corner with a native speed of 80MB/s, I doubt you
could make the drive stream with compression anyway in most cases.
|
Yup. You'll need some really beefy hardware to keep them suckers happy.
| Quote: | Therefore it makes more sense to me to compress the incoming data stream
to disk, and let the tape drive stream the already compressed data using
no compression.
Your comments are, as always, much appreciated.
/charles
|
Sounds like a good strategy. Added benefit is that an intermediate staging
disk will allow very fast restores for those "Whoops, I didn't mean to
delete that" incidents of the average PHB...
The only drawback is that you have to make absolutely sure that each next
version of your backup software is backward compatible with the compression
algorithm your software and disaster recovery systems use today.
Rob |
|
| Back to top |
|
|
Charles Morrall
Guest
|
| Posted:
Sat Dec 11, 2004 12:38 am Post subject:
Re: Software vs hardware compression in backups |
|
|
"Rob Turk" <_wipe_me_r.turk@chello.nl> skrev i meddelandet
news:41b9e992$0$145$e4fe514c@dreader9.news.xs4all.nl...
| Quote: | "Charles Morrall" <charles.morrall@telia.com> wrote in message
news:h0dud.10631$d5.94554@newsb.telia.net...
I base this theory on the assumption that the compression algorithms are
well known, and essentially it's the same process regardless if the
algorithm is implemented in hardware in the tape drive or software in the
backup application. I doubt there's a an ASIC in the tape drive dedicated
to handle compression though.
That's a wrong assumption. ALDC/IDRC/LZW compression chips have been
available for years and are in common use by tape drive manufacturers. I
don't know of any tape drive doing it's compression in firmware, it's
always a dedicated ASIC or part of a larger buffer management chip.
|
I see. Thank's for setting that straight.
| Quote: | Also, it's probably safe to assume that processing and memory capacity of
a backup server is vastly larger that of a tape drive even considering
the server has to do many other things I've come to the conclusion that
using compression in software has a better chance of compressing the data
stream going to disk than a tape drive has, when it has to try to stream
the tape at the same time.
Agreed.
Which is sort of was what I was getting at, a compression chip can't |
outperform the spare CPU cycles and perhaps a GB of RAM of even a modest
server.
| Quote: |
Therefore it makes more sense to me to compress the incoming data stream
to disk, and let the tape drive stream the already compressed data using
no compression.
Sounds like a good strategy. Added benefit is that an intermediate staging
disk will allow very fast restores for those "Whoops, I didn't mean to
delete that" incidents of the average PHB...
The only drawback is that you have to make absolutely sure that each next
version of your backup software is backward compatible with the
compression algorithm your software and disaster recovery systems use
today.
Rob
|
Didn't think of that actually. However, as long as I do incremental upgrades
of the same software package I should be OK, and in case I'm switching
software I'll just make sure I destage the entire disk device to tape. Also,
there's no guarantee that the tape format (compressed or not) is compatible
anyway so it's not really a new problem, just a fresh angle of an old
problem. :)
Thanks Rob!
/charles |
|
| Back to top |
|
|
Paul Rubin
Guest
|
| Posted:
Sat Dec 11, 2004 12:58 am Post subject:
Re: Software vs hardware compression in backups |
|
|
"Rob Turk" <_wipe_me_r.turk@chello.nl> writes:
| Quote: | The only drawback is that you have to make absolutely sure that each next
version of your backup software is backward compatible with the compression
algorithm your software and disaster recovery systems use today.
|
I used to backup with a modified version of gnu tar. I set my backup
script to do the actual backup in one tape file, then automatically
dump the source code of the modified tar to a second tape file. So
every backup tape had a source code copy of the backup program in its
own tape file where it could be easily recovered by just about anything. |
|
| Back to top |
|
|
Maxim S. Shatskih
Guest
|
| Posted:
Sat Dec 11, 2004 3:04 pm Post subject:
Re: Software vs hardware compression in backups |
|
|
Good compression algorithms like BWT or (especially) PPM have very bad
worst-case speed on uncompressible data, which means using them in streaming
environment a bit problematic. Yes, the host CPU is fast, but the tape drive is
too, and compression speed of the CPU can be slower then the drive speed.
Surely the tape drive's logic uses faster and worse algorithms like LZ or LZH.
They also do not require much memory (BWT requires starting from 1MB up).
Also - loading the server's CPU by compression can be unaffordable for some
scenarios.
--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@storagecraft.com
http://www.storagecraft.com
"Charles Morrall" <charles.morrall@telia.com> wrote in message
news:h0dud.10631$d5.94554@newsb.telia.net...
| Quote: | Backing up to disk is growing in popularity, and I for one certainly
recommend it in most cases. Another theory of mine is that it's more
favourable to use software compression when configuring the backup job
backing up to disk, and to configure the subsequent staging to tape with no
compression. I base this theory on the assumption that the compression
algorithms are well known, and essentially it's the same process regardless
if the algorithm is implemented in hardware in the tape drive or software in
the backup application. I doubt there's a an ASIC in the tape drive
dedicated to handle compression though. Also, it's probably safe to assume
that processing and memory capacity of a backup server is vastly larger that
of a tape drive even considering the server has to do many other things I've
come to the conclusion that using compression in software has a better
chance of compressing the data stream going to disk than a tape drive has,
when it has to try to stream the tape at the same time. With LTO-3 around
the corner with a native speed of 80MB/s, I doubt you could make the drive
stream with compression anyway in most cases.
Therefore it makes more sense to me to compress the incoming data stream to
disk, and let the tape drive stream the already compressed data using no
compression.
Your comments are, as always, much appreciated.
/charles
|
|
|
| Back to top |
|
|
Guest
|
| Posted:
Sat Dec 11, 2004 8:03 pm Post subject:
Re: Software vs hardware compression in backups |
|
|
"Charles Morrall" <charles.morrall@telia.com> writes:
| Quote: | Backing up to disk is growing in popularity, and I for one certainly
recommend it in most cases. Another theory of mine is that it's more
favourable to use software compression when configuring the backup
job backing up to disk, and to configure the subsequent staging to
tape with no compression.
|
see below
| Quote: | I base this theory on the assumption that the compression algorithms
are well known, and essentially it's the same process regardless if
the algorithm is implemented in hardware in the tape drive or
software in the backup application. I doubt there's a an ASIC in the
tape drive dedicated to handle compression though.
|
This is in fact wrong for at least DLTs and 8mm drives. They do use
compression chips in the data path to do the compression.
I would recomend you do NOT compress you backups unless you can be
totally sure that it will not bite in unexpected ways. ie, DB files
that have large amounts of tempory data for there are multi TB of
blocks that are no longer zeros, and way less compressable! Needing
the space for both the compressed and the original data, and the
software! when recovering to a back up system...
--
Paul Repacholi 1 Crescent Rd.,
+61 (08) 9257-1001 Kalamunda.
West Australia 6076
comp.os.vms,- The Older, Grumpier Slashdot
Raw, Cooked or Well-done, it's all half baked.
EPIC, The Architecture of the future, always has been, always will be. |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in