| Author |
Message |
Chevalier des bois
Guest
|
 Posted:
Thu Dec 15, 2005 12:25 am Post subject:
Re: Indefinite Storage |
|
|
Anton Rang a écrit :
a Faraday cage (to be protected from the cell phones emissions ...).
You may change the location of your office : data may be lost
The air conditioner may fail for 5 minutes in 30 years (which is an
amazing mtbf isn't it ?) : data may be lost.
....
IS THAT BE SURE OF RETRIEVING THE DATA ?
Tape is a backup media, not an archiving one.
| Quote: |
There are certainly plenty of low-cost tape formats which aren't so
stable, but tape vendors know what they're doing...and more to the
point, there are a lot of sites with experience in maintaining tape
archives for many years.
It's true : the optical technology was not existing 15 years ago that |
was the only way.
The question is : do these people own their old tapes, or do they try to
restore the data ?
From 5 year now, all the state agencies I know, who have to justify the
integrity of data for 40 or 50 years for legal purpose, do use MO
because they are controlled from time to time and obliged to restore
regularly.
| Quote: |
If you do need to maintain the data for more than 20-30 years, you'd
better plan to migrate it to new technologies anyway. I suspect that
MO disks written today may be readable in 100 years ...
if you could
find either a drive, or an appropriate microscope. ;-) Remember the
phase-change disks in the mid-1980s? Seen a phase-change drive lately?
-- Anton
(Disclaimer: I work for Sun, now a tape vendor. I don't speak for them.) |
|
|
| Back to top |
|
 |
Curious George
Guest
|
| Posted:
Thu Dec 15, 2005 1:16 am Post subject:
Re: Indefinite Storage |
|
|
On 14 Dec 2005 10:43:43 -0600, Anton Rang <rang@visi.com> wrote:
| Quote: | Surely you're not proposing a response to the OP as "dump all the
possibilities on management's lap and walk away believing no one will
ask or expect anything further of you."
Not to put words into his mouth, but I'd suggest a response more along
the lines of "suggest to management that we work with a consulting
firm with experience in both the legal and technical aspects of
records management." It's certainly possible to implement all of this
yourself, but is it a wise investment of time and resources?
Anton
|
Others have offered some suggestions but jlsue has criticized them
generally/nonspecifically. If we take him at his word we are
basically back at square one, not knowing which suggestions weren't
"misinformation" only hoping an outside source will provide salvation
- a very vulnerable position to be in IMHO.
Bringing in outside expertise to work out the details is absolutely
fine - actually wise. But it doesn't have to be a DIY project to
begin kicking around or ruling out some basic ideas/technologies. If
nothing more it's a good starting point when shopping for firms.
"What about doing X?"
"We do Y because it addresses the following compliance issues... X
will cause problems with ..."
But what really caught my eye more was:
"The first key thing to remember is that record retention is NOT a
technical (i.e., IT) problem. It is a business and legal problem."
It sounded to me like the common corporate mentality of "cover your
ass / only worry where the shit falls." For electronic records,
record retention & management is a shared problem & responsibility.
If there's no one in IT coordinating with the execs or legal or
accounting or outside consulting - only reliance on outsiders trying
to sell stuff - a company has big problems. Technology is supposed to
serve business needs & bad decisions, distractions from core values,
or unrealistic expectations of nontechnical ppl or can be devastating.
I'm sure jisue is a successful and respected professional. It's just
some things that caught my eye. Maybe I misread him - but that's why
I posted (giving opportunity for reply) rather than simply shrug it
off. |
|
| Back to top |
|
|
Anton Rang
Guest
|
| Posted:
Fri Dec 16, 2005 12:31 am Post subject:
Re: Indefinite Storage |
|
|
Chevalier des bois <fla.fla@wanadoo.fr> writes:
| Quote: | More than 10 years at 17°C +/- 2° + 30% Humidity +/- 5% and that all
in a Faraday cage (to be protected from the cell phones emissions ...).
|
Um, cell phones? Do you understand the difference between electric
and magnetic fields? Do you understand the concept of coercivity?
| Quote: | IS THAT BE SURE OF RETRIEVING THE DATA ?
Tape is a backup media, not an archiving one.
|
Tape has been an archival media for many years, and continues to be.
| Quote: | There are certainly plenty of low-cost tape formats which aren't so
stable, but tape vendors know what they're doing...and more to the
point, there are a lot of sites with experience in maintaining tape
archives for many years.
It's true : the optical technology was not existing 15 years ago that
was the only way.
|
Optical technology is far too expensive to store large amounts of data.
A single tape can hold 100-500 GB of data. How much can you fit on an
optical disk these days? How about magneto-optical, which has better
longevity?
| Quote: | The question is : do these people own their old tapes, or do they try
to restore the data ?
|
Generally these are not backups, they are offline data. So some data
is retrieved periodically. You are probably familiar with a lot of
these sites: NCAR, NASA, CERN, the national weather services, the
commercial satellite imaging companies, the major oil companies, the
national intelligence and defense industries ... all of these have the
need to store multiple terabytes (or even petabytes) of data for many
years. All of them are very concerned about not losing data, because
it is often either irreplaceable or too expensive to recreate. All of
them use very large quantities of magnetic tape.
| Quote: | From 5 year now, all the state agencies I know, who have to justify
the integrity of data for 40 or 50 years for legal purpose, do use MO
because they are controlled from time to time and obliged to restore
regularly.
|
It's reasonable to use MO if you've got fairly small amounts of data.
It has definite advantages if you have the need to retrieve small
files at random as well, since seek times are so much faster than
tape. (The sites I listed above generally are dealing with large data
sets, so random access speed is not so important for them -- though
there are tapes optimized for relatively fast random access, too.)
-- Anton
(P.S. Just to indicate that I'm not too biased on this, at home I
have both tape and MO technologies, and the product I work on supports
archives on both tape and MO. ;-) |
|
| Back to top |
|
|
Maxim S. Shatskih
Guest
|
| Posted:
Fri Dec 16, 2005 1:16 am Post subject:
Re: Indefinite Storage |
|
|
| Quote: | Um, cell phones? Do you understand the difference between electric
and magnetic fields? Do you understand the concept of coercivity?
|
Any wireless equipment uses electro-magnetic waves, which have both electric
and magnetic components.
--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@storagecraft.com
http://www.storagecraft.com |
|
| Back to top |
|
|
Chevalier des bois
Guest
|
| Posted:
Fri Dec 16, 2005 9:16 am Post subject:
Re: Indefinite Storage |
|
|
Anton Rang a écrit :
| Quote: | Chevalier des bois <fla.fla@wanadoo.fr> writes:
More than 10 years at 17°C +/- 2° + 30% Humidity +/- 5% and that all
in a Faraday cage (to be protected from the cell phones emissions ...).
Um, cell phones? Do you understand the difference between electric
and magnetic fields? Do you understand the concept of coercivity?
Oops coercivity is still very dependent on temperature (As far as I |
remember my Physics) ...
So it is not applicable to long term real life.
| Quote: |
IS THAT BE SURE OF RETRIEVING THE DATA ?
Tape is a backup media, not an archiving one.
Tape has been an archival media for many years, and continues to be.
Long term archival ... uhmmm ... |
| Quote: |
There are certainly plenty of low-cost tape formats which aren't so
stable, but tape vendors know what they're doing...and more to the
point, there are a lot of sites with experience in maintaining tape
archives for many years.
It's true : the optical technology was not existing 15 years ago that
was the only way.
Optical technology is far too expensive to store large amounts of data.
A single tape can hold 100-500 GB of data. How much can you fit on an
optical disk these days? How about magneto-optical, which has better
longevity?
The question is : do these people own their old tapes, or do they try
to restore the data ?
Generally these are not backups, they are offline data. So some data
is retrieved periodically. You are probably familiar with a lot of
these sites: NCAR, NASA, CERN, the national weather services, the
commercial satellite imaging companies, the major oil companies, the
national intelligence and defense industries ... all of these have the
need to store multiple terabytes (or even petabytes) of data for many
years. All of them are very concerned about not losing data, because
it is often either irreplaceable or too expensive to recreate. All of
them use very large quantities of magnetic tape.
All of these use almost every technology from almost any vendors so |
these can be claimed to use whatever techno.
Data are multi-replicated and their concern is far from a less than a
To, which we are talking about now.
For the most of these companies the huge part of their data is raw
scientific or image data. For these type of data there is usualy no
legal issue.
I know lots of sites in the defense industry that use MO to store the
test data of various devices which are tested uniquely to provide legal
tracebility for periods of over 20 years.
| Quote: |
From 5 year now, all the state agencies I know, who have to justify
the integrity of data for 40 or 50 years for legal purpose, do use MO
because they are controlled from time to time and obliged to restore
regularly.
It's reasonable to use MO if you've got fairly small amounts of data.
Although it WAS much more adapted. |
Nowadays BlueRay technology allows a per media capacity in the range
of tape (100Go+). The Robotics are at least as large as tape ones and
much more affordable for a small to medium size company.
No mechanics in the media, very high CEM compatibility standards.
| Quote: | It has definite advantages if you have the need to retrieve small
files at random as well, since seek times are so much faster than
tape. (The sites I listed above generally are dealing with large data
sets, so random access speed is not so important for them -- though
there are tapes optimized for relatively fast random access, too.)
-- Anton
(P.S. Just to indicate that I'm not too biased on this, at home I
have both tape and MO technologies, and the product I work on supports
archives on both tape and MO. ;-)
The most common error is to use a general purpose backup software |
(Veritas, Legato, Atempo, CommVault ...) to do archiving (Although they
all claim they do it well).
There are lot of alternatives to them specialized in archiving.
What is sure is long term archival is not a simple thing that can be
adressed with a tiny budget.
Lots of mis-information as it is in most the storage market.
For the most of the technologies which could be chosen (At least the
magnetic ones), the constraints in term of long term 'blockhaus'
environment are important (constant t°, constant humidity, very high cem
standards...) and such infrastructure is the main part of the real cost
(redundant high end clims integrated, redundant current alimentation
coming from at least tho independant electricity suppliers for the
climatizers,...
And be sure the company or the office will not have to move in the
following 30 years.
Govmnt Agencies can be sure of that.
Elsewhere the MO medium is the only way.
No or poor cem concern, no mecanical concerns, poor t° over time
sensibility, poor humidity over time concerns. |
|
| Back to top |
|
|
Xecaquan
Guest
|
| Posted:
Fri Dec 16, 2005 5:16 pm Post subject:
Re: Indefinite Storage |
|
|
Anton,
I work for a government department that is now legally required to
store email permanently, due to a recent federal case. The 300-500GB
is what our mail boxes normally size is for a full backup, but I am
required to retain the incrementals to catch as much daily mail as I
can. Right now, I am backing up to SDLT600 tapes, so each week right
now uses 4 tapes. This is for the incrementals and the full, and this
costs me around $300 each week.
The reason I started this thread is the life span of the robotic
library I use to backup is far less for many reason than the tapes
themselves. Sure I could purchase a small loader and store that with
the tapes, but I would also need to store copies of drives, os,
inventory/scanning and backup software every so many years as well. So
I want to ask here if there was any new media I wasn't aware of that
might be a little easier to manage for the long haul.
Anton Rang wrote:
| Quote: | Curious George <cg@email.net> writes:
while an interesting overview of the legal landscape, what would you
suggest the OP present as possible solutions to upper management? The
OP, after all asked:
"I am looking for a product that allows for permanent storage of about
300-500GB, without taking a ton of media and time. Scalability to at
least 1 Terra Bytes..."
Well, I'm not in the storage practice though I am in the industry.
But I'd certainly suggest that a company generating 500+ GB of *new
and unique* email per week is exceptional! This sounds like the
suggestion is to basically copy each user's mail file every week.
That's a very inefficient way to store the data.
I (not speaking for my employer, of course) would suggest
investigating email management software ala Lotus Domino, or a mail
server which can otherwise detect and manage duplicates ... and has
the ability to spool out only new data for archiving.
I suspect all of the major IT consulting firms have dealt with this
problem before. Sun, for instance, has implemented their "Infinite
Mailbox" solution for several clients. This integrates Lotus Domino
with SAM-FS to archive email to tape.
Surely you're not proposing a response to the OP as "dump all the
possibilities on management's lap and walk away believing no one will
ask or expect anything further of you."
Not to put words into his mouth, but I'd suggest a response more along
the lines of "suggest to management that we work with a consulting
firm with experience in both the legal and technical aspects of
records management." It's certainly possible to implement all of this
yourself, but is it a wise investment of time and resources?
Anton |
|
|
| Back to top |
|
|
jlsue
Guest
|
| Posted:
Fri Dec 16, 2005 5:16 pm Post subject:
Re: Indefinite Storage |
|
|
On 16 Dec 2005 04:20:07 -0800, "Xecaquan" <Junk@lucidearth.com> wrote:
| Quote: | Anton,
I work for a government department that is now legally required to
store email permanently, due to a recent federal case. The 300-500GB
is what our mail boxes normally size is for a full backup, but I am
required to retain the incrementals to catch as much daily mail as I
can. Right now, I am backing up to SDLT600 tapes, so each week right
now uses 4 tapes. This is for the incrementals and the full, and this
costs me around $300 each week.
The reason I started this thread is the life span of the robotic
library I use to backup is far less for many reason than the tapes
themselves. Sure I could purchase a small loader and store that with
the tapes, but I would also need to store copies of drives, os,
inventory/scanning and backup software every so many years as well. So
I want to ask here if there was any new media I wasn't aware of that
might be a little easier to manage for the long haul.
|
Ah, now there is a really sticky problem. Inedefinite storage of
tapes is going to be a real heartache... but working for the govt,
budget for 'finding' relevent information later on is always available
(somehow).
On the other hand, seeing as how tax dollars are probably involved in
this "reference" activity later on, I'd recommend an archival system
which can work in 'compliance' mode (i.e., everything going in or out
of the email system is sent to the archive). In this case, your
exchange (or domino) email system is configured to log everything, and
the archival system takes each log entry and inserts it into the
archive - preferably indexing the full text in the email messages,
metadata, and of course any attachments.
The indexing is critical, because if you really want to find that
stuff later on - which is ostensibly the reason you're keeping it -
then being able to search the archive for relevant messages is going
to be a big win.
Additionally, a solution that has built-in, low-impact "migration" to
new technology over time will be a huge help for your organization as
well. I.e., 10 years from now, your archive tapes created today will
probably not be readable via any technology you have in-house.
Converting those tapes to newer technology is going to be a costly
exercise.
With a more optimal archival process in place, you no longer need to
keep your backups for anything other than DR, which means only for
around 3-5 backup cycles. You can see a pretty good payback from a
reduction in tape purchases with this solution as well.
Now, I don't mean to push product - really, my pay is not based on any
product sales - but there are only a couple that do this very well.
Yes, it just so happens that I work for HP, and we have one of those
products.
--- jls
The preceding message was personal opinion only.
I do not speak in any authorized capacity for anyone,
and certainly not my employer. |
|
| Back to top |
|
|
Paul Rubin
Guest
|
| Posted:
Sat Dec 17, 2005 1:15 am Post subject:
Re: Indefinite Storage |
|
|
"Xecaquan" <Junk@lucidearth.com> writes:
| Quote: | I work for a government department that is now legally required to
store email permanently, due to a recent federal case. The 300-500GB
is what our mail boxes normally size is for a full backup, but I am
required to retain the incrementals to catch as much daily mail as I
can. Right now, I am backing up to SDLT600 tapes, so each week right
now uses 4 tapes. This is for the incrementals and the full, and this
costs me around $300 each week.
|
I'm a little confused. You get 300-500GB of new email every week? Or
you get a few GB (or less) per week, but you do a full backup every
week anyway? Why not do multi-level incrementals? Level 3 every
day, level 2 weekly, level 1 monthly, level 0 (full backup) quarterly?
I thought that was pretty normal. |
|
| Back to top |
|
|
Gregory Layman
Guest
|
| Posted:
Mon Dec 19, 2005 9:15 am Post subject:
Re: Indefinite Storage |
|
|
Chevalier des bois wrote:
| Quote: | Rob Turk a écrit :
"Chevalier des bois" <fla.fla@wanadoo.fr> wrote in message
news:4399b971$0$18342$8fcfb975@news.wanadoo.fr...
The problem does not deal with backup software but with tape.
Disk is not the subject here, no serious magnetic retention times are
specified by vendors so you will be obliged to rewrite the date from
time to time.
What is the specified magnetic retention time for tape ?
Chances are that in 5 years the data on your precious tapes won't be
there anymore, so 20-30 years ...
For media that are specified for such time frame (To 40 years max),
look at dvdroms that are specified for such time frame and a dvd
library.
As far as I know it is the only way.
Archiving the os and the backup software is not a bad idea though.
Do you have any data to substantiate that tapes will not retain their
data? I have plenty of tapes that are perfectly readable after 15
years. Not sure what to do with my data from back then on my old
EXB-8200 2GB tapes, but they can be read just fine. Proper care and a
decent tape technology will work quite well. (that does not include
the dinky floppy-tape/Travan/Onstream kind of drives).
My experience with CD/DVD is to the contrary. Good mass-produced audio
CD's last a long time, but the writable variant fails without
exception after 2-5 years. They all develop bad area's. This is due to
the substrate that is being used for writable CD/DVD media. It's not
stable over time. The bad thing is that newer CD's appear to break
down faster than the first-generation 2x and 4x writeable CD's. I
guess that's because it must be cheap to produce...
Rob
Five years max specified from tape vendors, after, it's luck.
No regular CD/DVD but DVDROM.
This is the only choice for these who have legal retentention of more
than ten years.
All other solutions are unsure and of course no professionnal.
I think what Rob is referring to is high end tape drives. Specific to |
the question was retaining data for 15-30 years. There probably isn't
any technology today that will still exist 15-30 years from now. I
store medical data (for indefinite periods) on StorageTek 9940 tape
media. The manufactuers' stated life span is 15-30 years. If you use
9940B tape drives you can write 200 GB per tape and in a StorageTek L700
library you can house 678 tapes (therefore you have 135 TB capacity
which you can grow to as needed by adding additional tapes). This will
also interface with Vertias Netbackup.
There is no doubt that you will have to migrate this data from time to
time to the latest technology, but that should be expected. The
building blocks to insure data integrity after 30 years is to use only
high-end equipment (which is going to cost high-end money). You will
get what you pay for on tape technology, so don't settle for a couple
thousand dollar tape drive. |
|
| Back to top |
|
|
rcyoung
Guest
|
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in