Back that thing up- How to handle storage when shooting RAW or 4k pt 2- Space Free or Die Hard.

on September 9, 2014

Ok. Now we understand all about hard drives- or at least enough about hard drives- and we are ready to get some.

Not quite.

I highly recommend you take a moment to design your storage system.


Design it.

Don’t just go on connecting crap to your computer.

The best thing to do is think about your needs (Safety? Speed?), how you are going to be using your drives, and how much you are willing to spend.
Remember, we don’t want to touch that last 20% of storage in order to keep our footage safe- see pt 1 HERE.

Take out a blank piece of paper and put your computer in the center draw a picture of your computer in the center.

You should also draw a tiny little hard drive inside your computer- that’s where your operating system is- and it should be an SSD.

Your next drive should be where you keep all your media- movies, music, documents- anything you shouldn’t be putting on your SSD to slow it down.
If you are just starting out, changes are this is where you are putting your client files. You should at leas back up the important stuff in the cloud.

The next disk you should get should be a scratch disk. This will improve your editing performance tremendously. Get an SSD if you can, but it’s not a deal breaker. The more space you have here, the better. This is where you can store all your caches and renders that your programs use to speed up performance. You can use the same drive for Photoshop, your editing program, after effects, etc. if it gets filled up, just empty it. All these files can be rebuilt.

The next set of drives you should be considering would be a RAID 1 array. Just two drives that are clones if each other that you keep client files on.

I like to have a dedicated output drive also, as it speeds up my output render times and I find myself elevating less when I’m near a deadline.

Now- when an outside project comes in, it usually comes in the form of a hard drive. These are the Masters- (or hopefully, a clone of the Masters that the producer has backed up.) when it comes to backups- two is one, and one is none.

I have two options for editing at this point- and online edit (just a normal edit), or an offline edit.
Am online edit means that my editing program “ingests” the footage, and every time it plays it back in the timeline, it points out to the computer where the file is, and it plays back from there. The computer is READING from the master files, and WRITING to the editing file (usually a very small file). Having one pipe for each path speeds this up. As it’s rendering previews for you to watch as playback, it is also WRITING and READING those render caches- this is where you separate scratch disk comes in handy.
If I have a heart CPU and GPU, and have enough space on my RAID array, I can edit the 4k RAW files this way and play them through on my system. This method has the most fidelity to your source material.

However, just as a map that is a full scale replica is not always the most useful, it is not always necessary to have EVERY pixel in its FULL color gamut when you are just setting in and out points for a rough cut, compared to say, the Color or SFX phases.

For larger projects, it is common to do an OFFLINE edit-meaning you don’t edit the files directly, but rather, create lo-res proxy files to work with. You do your edit with the small, easy to handle files, then at the last minute, do the old switcheroo and sub them for the full size version when you export. This has many advantages. One is that you can now edit your “4k” project on a less powerful laptop- which saves you money and also makes the project more portable- you aren’t locked in your editing room. Editing in a coffee shop is actually quite a treat with the right pair of headphones.

It also means that you don’t have to keep referencing your Masters for the edit, which quarantines them from corruption. Every time you read or write information on a disk, you put all the information on that disk at risk. Now each byte of information is a small risk, and even the cumulative risk is small, but if one out of every thousand minutes you recorded was lost, you would have a problem. If you are creating a proxy, keep it separate from your masters and work off the proxies. You can always regenerate those. If the masters are lost, your project is dead.

Now you are done with the edit- with only minutes to spare! Your need this video for a pitch before noon and this is the morning of! Having a render disk to output to can make this process go a little faster. It’s a luxury, but it’s nice to have. Remember we are only as slow as out bottleneck,so the computer is actually generating the information slower than the hard drive is capable of writing it (in most cases).

Here’s my setup for handling 4k and RAW projects:
Media Management

So what’s the take away from this? Think like a plumber.
Hot water is on the left and shot flows downhill.

If you have a lot of it, things go faster if it’s only in one direction. If it’s a little, two directions works. The more pipes you have to relieve the pressure means that the flow doesn’t get clogged up.