Back that thing up- How to handle storage when shooting RAW or 4k: pt 1- The RAID

on September 9, 2014

Now that cameras that shot RAW and 4k are priced for the masses, one of the hidden costs is storage costs.

Luckily, hard drive prices have dropped considerably, and we are seeing larger and larger drives available for next to nothing.

The first reaction to having bigger files is to just get more storage when a hard drive fills up- and that’s when problems start to creep up.

Let’s start by saying that it’s impossible to talk about hard drives without talking about speed. The larger the file, the more important speed becomes when transferring. Offloading a 512g card over USB 3.0 can take an hour and a half to 2 hours. Not planning that exchange in advance is enough to shut down production while everyone just waits for a card to copy. Someone trying to save a few dollars and getting a USB 2.0 cable can turn that process into 7-8 hours.
When considering speed, it is best to think if it as a pipeline. And you will always be limited down stream by your slowest transfer or write speed, so try to eliminate or reduce your bottleneck as much as possible.

Trying to increase bandwidth is also the reason why multiple hard drives are better than single large ones. Each drive has a cable to the computer that limits how muh information can go in and out. Reading is always faster than writing, and having both at the same time is very inefficient.
So let’s discuss RAID very briefly. I will go into raid arrays in a separate post, but the basics is that it just means Redundant Array of Inexpensive (or Independent) Disks.
It just means the hard drives are connected to each other.

RAID 1 means that one drive is a clone of another, so that as you write information on one disk, it writes it to an identical cloned disk in real time. This can slow down your transfer speed slightly, but I have found it to be negligible. It is important to keep in mind that this is nota aBACKUP, but rather- a redundancy. If you have a virus starters corrupt one drive- it has also corrupted the other. If a drive has a me handicap failure, as they all WILL eventually, then you can replace that drive and it’s twin will clone it. Two 2tb drives in RAID will yield 2tb of information on 4tb of storage.

RAID 0 is the opposite. It takes two drives and treats then as one large volume. The advantage of this is that you now have TWO access points to your data and you have doubled your transfer speed, making it MUCH faster. The disadvantage here, of course, is that if one drive fails, you’ve lost the information from both. This is great for places where you don’t care of information is lost- like in a scratch disk.

RAID 5 is a little like both. It requires at least three drives. Information is spread throughout all the drives, using all the inroads and out roads, but one of the disks is reserved for parity, which basically means that if any of the drives fail, you can replace it, and the group will rebuild the information from the existing drives when you replace it. This is a very efficient and safe system- but is also more complicated and possibly expensive with dedicated units attached via Ethernet.

RAID 10 and RAID 01 are basically combinations of both RAID 0 and 1, and are different only by which order they clone disks. Also a very effective method for backup.

The average hard drive should be pretty reliable for 3 years. After that point you start to see rates creep up. Some brands are better than others: sea hate is notoriously bad, while western digital is pretty solid. You can check out the failure rates reported by different brands HERE.

One of THE most important things to remember is to KEEP FREE SPACE ON YOUR drive. When you have less than 20 percent capacity, you run a serious risk of corrupting data, as well as slowing down your performance.

When offloading cards (which is basically a tiny hard drive), best practice dictates to do a card dump at 50 percent capacity. This is because reading and writing on a disk is like arranging the puzzle pieces on one of those games you get at the fair.

You know- the ones that make a picture out of the blocks and only has one empty space? You move the pieces around in order to make your picture using the empty space as best you can. The more free space in your drive, the more empty space you have to move things around (ie write and read off the drive). The drive is constantly reading and writing information to and from the drive in order to optimize the information in order to access it better. If you keep encroaching on the last bits of free space on your drive in order to save a few dollars, you will soon find yourself paying a data recovery specialist a LOT if money to get your precious movie back.
Keep plenty of free space.
It will be the best safeguard you can have.

This all, of course, pertains the most to traditional spinning disk drives. Solid state drives, or SSDs, are a different beast. These things have no moving parts, last a very long time, and are insanely fast. The prices up till now have been expensive, so sizes have been small, but we’ve started to see some very large SSDs hit the market and the smaller ones have gotten cheaper. If you haven’t started using an ssd as your startup drive for your operating system, you are missing out. Keep your media and folder separate, but boot off of an SSD- it will bring speed to your computer you didn’t think was possible. SSDs are also much faster than RAID 0, so if you can afford the space for it, it makes a great scratch disk. How much should you get? Bigger is better.

When working on client projects, I always work with a RAID 1 system locally in my computer. I have an SSD for my operating system. For anything that a especially important, I back it up on the cloud. (RAID is a redundancy- not backup, remember). I used to backup to a separate hard drive in my house until am artist I knew had an apartment fire. An apartment with both hard drives. The cloud avoids that altogether.
Both Dropbox and google drive have insanely cheap monthly options.
1tb of Dropbox is either $9.99 a month or $100 for a year, and I didn’t get paid to say that. I put working client files I. The cloud- that way if my house catches fire or all my conputer stuff gets stolen, my information is still recoverable. If you want to stay cheap, google drive will give you 100gigs for $2 a month.

In the next post we will discuss the separation and organization of drives on order to safeguard your files and work with these massive files.