[Link Posting] Why the Future of Data Storage is (Still) Magnetic Tape

[Rich]

####################################################################
# ATTENTION: This post is a reference to a website. The poster of #
# this Usenet article is not the author of the referenced website. #
####################################################################

<URL:https://spectrum.ieee.org/computing/hardware/why-the-future-of-data
-storage-is-still-magnetic-tape>

It should come as no surprise that recent advances in big-data analytics
and artificial intelligence have created strong incentives for
enterprises to amass information about every measurable aspect of their
businesses. And financial regulations now require organizations to keep
records for much longer periods than they had to in the past. So
companies and institutions of all stripes are holding onto more and
more.
Studies show that the amount of data being recorded is increasing at 30
to 40 percent per year. At the same time, the capacity of modern hard
drives, which are used to store most of this, is increasing at less than
half that rate. Fortunately, much of this information doesn't need to be
accessed instantly. And for such things, magnetic tape is the perfect
solution.
Seriously? Tape? The very idea may evoke images of reels rotating
fitfully next to a bulky mainframe in an old movie like Desk Set or Dr.
Strangelove. So, a quick reality check: Tape has never gone away!
Indeed, much of the world's data is still kept on tape, including data
for basic science, such as particle physics and radio astronomy, human
heritage and national archives, major motion pictures, banking,
insurance, oil exploration, and more. There is even a cadre of people
(including me, trained in materials science, engineering, or physics)
whose job it is to keep improving tape storage.
Tape has been around for a long while, yes, but the technology hasn't
been frozen in time. Quite the contrary. Like the hard disk and the
transistor, magnetic tape has advanced enormously over the decades.
...

[Supermarine Spitfire]

Post by Rich
####################################################################
# ATTENTION: This post is a reference to a website. The poster of #
# this Usenet article is not the author of the referenced website. #
####################################################################
<URL:https://spectrum.ieee.org/computing/hardware/why-the-future-of-data
-storage-is-still-magnetic-tape>

[snip]

Post by Rich

Tape has been around for a long while, yes, but the technology hasn't
been frozen in time. Quite the contrary. Like the hard disk and the
transistor, magnetic tape has advanced enormously over the decades.
...

I used to work in tape duplication and conversion in a previous job,
going back to the 1990s and 2000s. The formats I dealt with spanned the
decades and storage capacities, from old tape reels through DC (Data
Cartridge), DAT and Exabyte formats, up to LTO, plus some proprietary
IBM and DEC formats. I was developing software to work directly with the
drive hardware and tape formatting, and occasionally doing data
conversion too, between ASCII and EBCDIC character encodings or
extracting files from TAR or CPIO archives. The bulk of that work dried
up over time as CDs and DVDs became the preferred medium for application
and data supply, but we were still doing some tape work up until 2012.

[Huge]

Post by Supermarine Spitfire

Post by Rich
####################################################################
# ATTENTION: This post is a reference to a website. The poster of #
# this Usenet article is not the author of the referenced website. #
####################################################################
<URL:https://spectrum.ieee.org/computing/hardware/why-the-future-of-data
-storage-is-still-magnetic-tape>

[snip]

Post by Rich

Tape has been around for a long while, yes, but the technology hasn't
been frozen in time. Quite the contrary. Like the hard disk and the
transistor, magnetic tape has advanced enormously over the decades.
...

I used to work in tape duplication and conversion in a previous job,
going back to the 1990s and 2000s. The formats I dealt with spanned the
decades and storage capacities, from old tape reels through DC (Data
Cartridge), DAT and Exabyte formats, up to LTO, plus some proprietary
IBM and DEC formats. I was developing software to work directly with the
drive hardware and tape formatting, and occasionally doing data
conversion too, between ASCII and EBCDIC character encodings or
extracting files from TAR or CPIO archives. The bulk of that work dried
up over time as CDs and DVDs became the preferred medium for application
and data supply, but we were still doing some tape work up until 2012.

In theory, everything (several petabytes) at $(MEGABANK) was backed up to
tape, but I don't recall anything ever having to be recovered from it for
real. In theory, recovery was tested every six months, but I'd be prepared
to wager that all people did was tick the box without doing the test. When
everything's kept on multiply redundant NAS, you have to be very unlucky
to lose anything.

(And before anyone pipes up, yes, I'm well aware that NAS != backup.)

[Huge]

Post by Rich
####################################################################
# ATTENTION: This post is a reference to a website. The poster of #
# this Usenet article is not the author of the referenced website. #
####################################################################
<URL:https://spectrum.ieee.org/computing/hardware/why-the-future-of-data
-storage-is-still-magnetic-tape>

It always pissed me off that Exabyte tapes didn't contain any such thing ...

[Rich]

Post by Supermarine Spitfire

Post by Rich
####################################################################
# ATTENTION: This post is a reference to a website. The poster of #
# this Usenet article is not the author of the referenced website. #
####################################################################
<URL:https://spectrum.ieee.org/computing/hardware/why-the-future-of-data
-storage-is-still-magnetic-tape>

[snip]

Post by Rich

Tape has been around for a long while, yes, but the technology hasn't
been frozen in time. Quite the contrary. Like the hard disk and the
transistor, magnetic tape has advanced enormously over the decades.

... In theory, recovery was tested every six months, but I'd be prepared
to wager that all people did was tick the box without doing the test.

If anything beyond ticking the box was performed, they likely mounted a
tape cartridge, then only performed effectively this:

dd if=/dev/st0 of=/dev/null bs=64k

And waited to see if the logs showed any read errors.

[Huge]

Post by Rich

Post by Supermarine Spitfire

Post by Rich
####################################################################
# ATTENTION: This post is a reference to a website. The poster of #
# this Usenet article is not the author of the referenced website. #
####################################################################
<URL:https://spectrum.ieee.org/computing/hardware/why-the-future-of-data
-storage-is-still-magnetic-tape>

[snip]

Post by Rich

Tape has been around for a long while, yes, but the technology hasn't
been frozen in time. Quite the contrary. Like the hard disk and the
transistor, magnetic tape has advanced enormously over the decades.

... In theory, recovery was tested every six months, but I'd be prepared
to wager that all people did was tick the box without doing the test.

If anything beyond ticking the box was performed, they likely mounted a
dd if=/dev/st0 of=/dev/null bs=64k
And waited to see if the logs showed any read errors.

In theory, they were supposed to do a full DR, but I never cared if they
did it or not. It always faintly amused me what would happen in a full
"data centre burned to the ground" scenario when people were trying to
recover ~2000 systems at one go.

[Rich]

Post by Huge

Post by Rich

Post by Supermarine Spitfire

Post by Rich
####################################################################
# ATTENTION: This post is a reference to a website. The poster of #
# this Usenet article is not the author of the referenced website. #
####################################################################
<URL:https://spectrum.ieee.org/computing/hardware/why-the-future-of-data
-storage-is-still-magnetic-tape>

[snip]

Post by Rich

Tape has been around for a long while, yes, but the technology hasn't
been frozen in time. Quite the contrary. Like the hard disk and the
transistor, magnetic tape has advanced enormously over the decades.

... In theory, recovery was tested every six months, but I'd be prepared
to wager that all people did was tick the box without doing the test.

If anything beyond ticking the box was performed, they likely mounted a
dd if=/dev/st0 of=/dev/null bs=64k
And waited to see if the logs showed any read errors.

In theory, they were supposed to do a full DR,

Yep, that is what the checkbox meant.

Reality was that if one got at least a "read of tape from start to end,
looking for reported read errors" one was lucky.

Actual results was usually that the tapes were treated as erasable
write only media until "something bad happened" at which point smelly
brown stuff started hitting many spinning fans.

Post by Huge
but I never cared if they did it or not.

Nor did anyone else who could ignore the issue because "it is that
other team's responsibility".

Post by Huge
It always faintly amused me what would happen in a full "data centre
burned to the ground" scenario when people were trying to recover
~2000 systems at one go.

They would likely discover that some tapes had been recycled enough
that the oxide had worn to the point that there were numerous read
errors with those.

Other tapes would be discovered to be "full backups" but upon restore
find that critical files were not on the tape (often new configuration
files or new data files for V6 of the software because the backup
procedure was build when V2 of the software was current and never
updated for V3 through V6 changes).

And in the end, they would be hosed.

[Huge]

[45 lines snipped]

Post by Rich

Post by Huge
It always faintly amused me what would happen in a full "data centre
burned to the ground" scenario when people were trying to recover
~2000 systems at one go.

They would likely discover that some tapes had been recycled enough
that the oxide had worn to the point that there were numerous read
errors with those.

They'd be too busy fighting over the limited number of drives to worry
about the tapes. In theory, the systems were arranged in order of
criticality, but when you've still got 200 Category 1 (restore ASAP)
systems after triage, the meetings are going to be ... interesting.

And I'd bet there are interactions where System A depends on B depends
on C depends on A which mean it's impossible to get one on the air without
all the others.

[Bob Eager]

Post by Huge
[45 lines snipped]

Post by Rich

Post by Huge
It always faintly amused me what would happen in a full "data centre
burned to the ground" scenario when people were trying to recover
~2000 systems at one go.

They would likely discover that some tapes had been recycled enough
that the oxide had worn to the point that there were numerous read
errors with those.

They'd be too busy fighting over the limited number of drives to worry
about the tapes. In theory, the systems were arranged in order of
criticality, but when you've still got 200 Category 1 (restore ASAP)
systems after triage, the meetings are going to be ... interesting.
And I'd bet there are interactions where System A depends on B depends
on C depends on A which mean it's impossible to get one on the air
without all the others.

That one is bad enough at home. There is a laminated list taped to the
side of the main rack, listing the order in which to power things up.

And THAT runs to 11 items.

[Huge]

Post by Bob Eager

Post by Huge
[45 lines snipped]

Post by Rich

Post by Huge
It always faintly amused me what would happen in a full "data centre
burned to the ground" scenario when people were trying to recover
~2000 systems at one go.

They would likely discover that some tapes had been recycled enough
that the oxide had worn to the point that there were numerous read
errors with those.

They'd be too busy fighting over the limited number of drives to worry
about the tapes. In theory, the systems were arranged in order of
criticality, but when you've still got 200 Category 1 (restore ASAP)
systems after triage, the meetings are going to be ... interesting.
And I'd bet there are interactions where System A depends on B depends
on C depends on A which mean it's impossible to get one on the air
without all the others.

That one is bad enough at home. There is a laminated list taped to the
side of the main rack, listing the order in which to power things up.
And THAT runs to 11 items.

Ye,s but you're slightly unusual by domestic user standards!

[p***@lycos.com]

In my industry, files are huge, so tape works well for shipping data and archiving. For most people with lots of small files, I can see disk is better.
The only problem is that transfer rates haven't kept up:
LTO8 is 120 x the capacity of LTO1, but transfer speed is only 18 x.