Foreword
It’s been a while since I’ve written an article about my workflow with storage in particular. In this article I will not only tell you about how I manage my storage, but also how I work on the road and organise everything so that I can seamlessly transfer it to my home setup without too much hassle. I’ll also talk about my directory structure and backups. I will try to keep everything simple, as I am not a giant tech-head, and I know that that are a lot of readers out there that aren’t either.
The biggest reason for writing this article is the same question that pops up on almost all of my workshops : “Albert, how do you handle storage of all of your photos?” . It’s funny, this question always remains the same throughout the years. Apparently there still isn’t a ‘standard’ way or workflow that everyone uses. And even for myself, some things are not always clear. More on that later. The second reason I am writing this article is that I recently switched storage systems. I have an article on my website in which I worked with my Drobo from about 4 years ago. This is all a bit outdated, so I figured it’s time for a new article.
Full Disclaimer: I asked QNAP to work with me on this in return for a QNAP NAS, which I am currently using for almost a year. The QNAP Nas is just part of this article, while the article is mainly focused on my Lightroom and NAS workflow.
Introduction
First things first: not everyone has the same workflow. The workflow I am using is definitely not the only right one, or the one you should use. But this article can probably help you to give you a general overview of what’s potentially needed when working with large amounts of photographs.
Half a million photos : That’s right. My catalogue is nearing about half a million RAW photos. I actually have no idea if that’s a lot compared to other photographers. I guess it really depends on how much you are shooting, how long you are a photographer and most importantly: How much photos you delete! In my case: That’s zero (yes I’m lazy). I am currently over 40 terabytes of data, and all the photos are in just one catalogue.
Lightroom as a Host
Notice I am using the word catalogue every time. Yes, I indeed mean my Lightroom catalog with it. I use Lightroom as my host and do everything from there. I have been doing is from the very start of my photography career more than 10 years ago and have always been pleased with it. I have not used other software for cataloguing so I can’t say if Lightroom is the best for it. But in my case, it works. Also note that I am using just one catalog for all of my photos. I’ve heard people using different catalogues for more speed. I’m not sure if that really works, but I found that my catalog with 40 terabyte of photos is still quite fast to work with. I do sometimes run ‘File -> Optimise catalogue’ when the catalog runs a bit slower, and that does work to make it fast again.
It’s important to store your Lightroom catalog on the fastest location you have available. For me, that’s simply on my main machine that I am currently using: My Mac Studio M1Ultra. This one has my fastest internal SSD drive, on which I simply store the catalog. Do NOT store your catalog on an external drive, as that will make your entire workflow much slower. It’s fine to store your photos on external drives, more on that later at the Storage part of this article.
To maintain speed and overview, it’s important to build up your catalog in a structured way. For me, that’s as follows:
One folder where all of your photos are in, name it Photography for example
Sub folders with years
Sub folders with months
Sub folders with projects. I used to do dates a few years ago but figured that was not really needed for my use case. Simply naming it the location / name of the project works good enough for me.
Here’s a photo of how that looks in my Lightroom Catalog:
Now what’s important is that you can use Lightroom’s key-wording to easily find back all of your photos over the years. The best is to import your photos through Lightroom and put a few key-words on every import. However, I know some of you are lazy (me included) and don’t put any keywords. But Lightroom is smart enough: Even if you don’t put keywords but give your directories good names, Lightroom will use those when you search for keywords as well. So if I simply search for ‘Amsterdam’ in my entire ‘Photography’ (remember, we put everything in 1 folder) folder, it will find all of the photos from Amsterdam that I took in my entire career as a photographer.
So this is how I structure my Lightroom catalog in a nutshell. The most important thing is to have a good structure, and be consistent. That way you’ll benefit from it in the long run.
Using Lightroom as a host has many advantages. It’s also super easy to sort files based on metadata or ratings. I can easily see how many photos I took with a certain lens over the years. It gives me a good overview of which lenses to bring to what kind of trips.
The way I work in Lightroom after importing the photos.
After I’ve imported every photo session in Lightroom, I go through all of the photos, select which ones I am going to edit and simply put them into a TBP (To be processed) collection. These sometimes pile up as I do not always have the time to immediately post process photos. But having them in the TBP folder sorted by date, I can always easily start working on them fast.
Please note that I don’t do all of my editing in Lightroom, but the end results always come back into Lightroom. I often sidetrack into Photoshop or other software like Luminar Neo. But once I am done there, I save my file and have it pop up back into my Lightroom catalogue, to keep everything organised.
If you are interested to learn postprocessing from me, I have a full editing course available that teaches all my postprocessing techniques.
The next thing I do is use collections for finished projects. I have a collection set for ‘Photo Series’. These are themed specific and not always location bound (although they can be). And then I have bigger collections per location.
A little example from an Antarctica trip:
over 14.000 photos (yes, a lot).
Every day I quickly sorted photos that might be worth to edit. They end up in the TBP folder.
Eventually I edit all the photos, they end up in the Antarctica collection in my Location folders.
However, I have multiple Antarctica series. So some of these end up in the ‘Photo series’ as different series, like ‘Penguins’ or ‘Antarctic Landscapes’
That means that the same photos often can be found on different collections. And that’s okay.
Hope that’s clear. It’s not all that complicated, but again: Structure and workflow is important here. Now you probably wonder how I handle things on the road and make the transition to the office workflow smooth. More on that later in the ‘On the Road’ chapter.
Storage
Let’s step back a little bit and talk about my storage workflow when working with large amount of photos.
A lot of people who will be reading this article are probably be using the ‘Let’s buy a new bigger external hard drive when my other one is close to full’ strategy. They then put the older photos with the older hard drives somewhere in the closet, and almost never look at them again. Trust me, I’ve been there :)
The obvious advantage of having access to all of your old files is obviously what I mentioned above: You’re able to search for photos in your entire catalog of photos with ease. And that’s something you don’t have if you keep buying new hard drives and putting the older ones in the closet. If you want to get slightly more serious about storage, you will have to buy a device that groups multiple drives together. A NAS (network attached storage) or a DAS (Direct attached storage).
You can find plenty of devices online with each their own advantages. Most people will be opting for a NAS to store large amounts of data.
________________________________________
Please note that I was using a DAS before via thunderbolt. You can read about that in my older article on my website, using a Drobo device. I switched to a NAS (that you can actually also connect via Thunderbolt as an extension) after a while because of the following reasons:
A NAS has more functionality: both in terms of configuration and tools
A NAS can be online 24/7 without your workstation having to be online
A NAS is easier to be used by multiple users at the same time.
You can put your NAS anywhere, as its connected via your ethernet network (you can use long cables). A DAS has to be right next to your workstation.
And the biggest reason: my Drobo, the brand that I was using as a DAS, was not anymore supported by newer OS versions as they went (semi) bankrupt. I did always love using the system.
________________________________________
Lots of people with similar setups as mine will want to go for a NAS system. A NAS, Network Attached Storage device, is basically a mini computer that you can access via your network. So you don’t attach it via USB, you attached it via an ethernet cable. It offers lots of functionality, apps and tools that you can play with. Not only that, but modern NAS devices also offer lots of internal (and external) backup options. But the most important thing is that such device has a number of drive bays that group your hard drives together. That means you can basically put a number of hard drives together and your workstation will see it as 1 drive.
In my case I use the QNAP TVS-H1288x, which has 8 normal drive bays, 4 ssd slots and 2 NVME SSD slots. Yes, that’s not an easy name. In my opinion, these products are not so well marketed to ‘the masses’ and focus more on the tech-industry, hence the complicated names. If I may give companies like QNAP some advice: Photographers and content creators often need a lot of storage and are interested in these devices. Simply name a product QNAP Creator Pro and it will immediately catch interest of this whole niche. Just an advice ;)
But more about the QNAP TVS-H1288x. Like I said: it’s basically a mini computer fully specced with a Xeon processor, its own memory and it even has a graphic card. It’s fully functional without the need to connect it to a computer. This has mostly advantages, but can also have disadvantages (think of online vulnerabilities). You simply connect it to your network via an ethernet cable. The QNAP TVS-H1288x has 10gbit functionality for fast data transfers. My Mac Studio supports 10gbit ethernet, hence the choice for a full 10gbit workflow. Keep in mind that you’ll want a full 10gbit workflow to keep all the speeds to the maximum. So you can’t just simply use the router that you received with your internet subscription for that, as most of them do not support 10gbit (they are mostly 1 Gbit). So to get the 10gbit speeds everywhere, I have the QNAP switch. This works super simple: Just plugin your workstation and the QNAP NAS in the fastest 10gbit ports, and plugin your ‘slower’ network devices (like your internet) in the 2.5gbit ports. You do not have to setup anything, just plug them in and it works.
For a full list of specifications of the QNAP I am using, please check https://www.qnap.com/en/product/tvs-h1288x/specs/hardware
So how is my QNAP NAS set up?
Before we move on, it's important to mention that the setup of the QNAP NAS is quite 'general' and can be used for any kind of NAS.
As mentioned before, my QNAP has 8 normal drive bays, 4 ssd slots and 2 NVME ssd slots. NVME SSD do not use the ‘old-fashioned’ SATA connection that limits speeds. They use a PCIe bus interface which is a faster connection to the CPU. In general, NVME can be a bit faster than standards SSDs. In NAS devices, NVME slots are often used for cache storage.
My QNAP setup:
8 drive bays each equipped with a WD Red Plus 10 TB. The WD Red Plus drives are built for NAS devices that are on 24/7, regarding performance and reliability.
4 ssd drive bays equipped with WD Red SA500 1TB SSD
2x NVMe ssd slots equipped with WD Red SN700 NVMe 1 TB SSD
Full disclosure: Western Digital collaborated with me on this setup and sent me a few hard drives as part of WD celebrating their 10th anniversary of the WD Red Series. Congratulations! I have also been using WD Red drives in the past (and have bought them myself)
So does that mean I have close to 100 terabyte of disk space? In theory yes, but it doesn’t really work that way. It’s all about how you configure your NAS device. NAS devices are very flexible that way, and offer different ways on how to set up your storage. It’s mostly about how much backup you want. Do you want your data back-up if 1 drive fails? Or what if 2 fail at the same time? Or 3? The level of backup determines your maximum disk space. This is called a RAID setup. RAID setups are not just for backups, but also for speed. My setup is based on RAID 5, as QNAP’s own website says:
‘Data and parity information are striped across all disks.The capacity of one disk is lost for parity. This means that if any one disk fails, it can be replaced and the data on it can be restored.Striping means read speeds are increased with each additional disk.Recommended for a good balance between data protection and speed.’
I could also choose RAID 6, which has an extra layer of security meaning 2 drives can fail at the same time. But that rarely happens (Never happened to me in the past, but 1 drive failures are not out of the ordinary) , so I opted for RAID 5, as I also have other back-ups (more on that later) incase really bad things happen. For a full list of raid setups, see https://www.qnap.com/en/how-to/faq/article/raid-level-comparison-table
So in the end, I have about 60 TB of storage left on the 80 TB that is in the 8 drive bays. But I get a ‘safe’ feeling, and much faster speeds in return.
So what about the 4 ssd drives? They make 4 tb together, but I also put them in RAID with an effective storage of 2.4tb. This part of the storage I use for more recent projects, as It’s faster storage. And the 2x NVMe SSDs are assigned to ‘cache’. Cache is used by QNAP for files that you use a lot to make things faster. Don’t worry too much about the technical aspect of it: In the QNAP interface you just assign the NVME SSDs to cache, and the device will do the rest.
As for the normal hard drives and the SSDs: the QNAP interface lets you create ‘storage’ pools with a directly assigned to them. This means I can create a directly on my QNAP with all of my photography on the 60 TB part, and another directly called ‘recent projects’ assigned to the 4 TB of ssd storage. You can basically pick which part of storage you want to use for which directory on the drive.
All of the above is done through QNAP’s own operating system that you simply connect to via the network. Here you can see the control panels, dozens of apps and more. You arrange all your drives through the ‘Storage and Snapshots’ . It may be a bit overwhelming at first, but you quickly get the hang of it after using it a couple of times.
Quick Summary of Drive setup
Hard to follow? Here’s a quick summary of the above:
I have 60 TB of storage on my 80 TB drives because of the RAID5 setup for backup and speed. This is the storage I use for all of my data.
I have 2.4 TB of storage on my 4 TB ssd drive part, because of the same as above. I use this for my more recent projects. Once the project is done, I simply move it to the 60TB folder.
My 2x NVMe SSD’s are used for Cache, which you simply assign on the QNAP interface and QNAP does the rest.
Now all of the above is based on my QNAP NAS, but these are general setups that can be applied to other NAS devices as well. So feel free to pick the NAS that fits your requirements.
So do you really need such a ‘professional’ device? That’s completely up to you. You might not need so much data as me, in which case you can buy a much cheaper (and smaller) alternative. These systems come in all kinds of configurations and sizes. But my advice is: Make sure you’re ready for the future. Better to invest a bit more money so that you can use it for many years, instead of having to replace it after just a few years.
So how does the QNAP NAS work in my photography workflow?
I’ll spare you all the technical details of setting up a the NAS. Like I mentioned: It’s like a mini computer so that means it’s all a bit ‘techy’. When I used to be a teenager I loved everything about computers and was super interesting in building them. Setting up the NAS kind of reminded me of those times. But it’s not that difficult. Simply follow the manual or watch a setup video on YouTube. My setup was slightly more ‘complicated’ because I opted for some extra extensions that I added to the QNAP device: the QXP-T32P (here we go again with the names), which is an extension card that allows the device to accept Thunderbolt 4 connections for higher data speed transfers. 10gbit ethernet is fast, but Thunderbolt 4 is faster! To install this, I had to open the QNAP device and plug-in the extension card.
But yeah, let’s get a bit more practical again after all the rather complicated stuff: This is how it looks like on my system:
The NAS is called DROSQNAP and is attached to the network. I simply attached the NAS via the switch (see above) to my workstation, where it is instantly detected. Root folders are simply ‘Projects’ and ‘New Projects’. The ‘Projects’ folder is also the root folder of my catalogue in Lightroom, which you could see in the beginning of this article. Basically, Lightroom does not threat network storage any different than normal folders. You don’t even see that it’s network storage.
Every time I import photos from my SD (or CF) cards into Lightroom, I immediately import them onto the NAS. This works very simple and again, no different than how you would normally import photos. This goes as fast as you would expect on your normal workstation.
Also, editing and scrubbing through photos goes fast in the Lightroom Develop module. Granted, it’s not as fast as directly from your SSD, but I wouldn’t expect it to be. Still, the 10gbit connection easily handles editing directly from the NAS. Of course, the raid system really helps with the speed. Also saving big files with many layers in Photoshop is not much slower than saving it to your SSD. The speed is totally acceptable. I am even editing YouTube videos directly from my NAS.
And if you want to use the 'real' speed, you can always attach it to the thunderbolt port. The speed is then on another level, and copying hundreds of gigabytes happens in a matter of minutes. Connecting through thunderbolt is as easy as putting a thunderbolt cable in between the NAS and your workstation. The workstation will not see the thunderbolt connection as a hard drive. It will see it as an extra network connection. In my case, the normal ethernet connection device is named DROSQNAP, and if you connect it with a thunderbolt cable it’s named DROSQNAP (Thunderbolt). Simply use that one for the super fast speeds. Important to note here is that you really need to use a Thunderbolt 4 cable, and not just a standard usb-c cable. This can be confusing as they look the same. If you use a standard usb-c cable, you might think the connection is not working as it simply doesn’t show up (I had this happen myself).
Here’s a little diagram to show how everything comes together:
On the road workflow
See the little ‘travel laptop’ icon on the right? Let’s talk about that a little bit here. As I mentioned before in the beginning of this article when I briefly talked about my 14k Antarctica photos: It’s important to have a streamlined workflow on the road, and combining it with your office workflow. This makes sure you can seamlessly transfer edits and sorting photos.
On all of my trips I bring a (fast) laptop. I often do quick edits on the road and I find it important that my workstation on the road is as fast as possible, but is portable as well. I always make sure to have plenty of disk space on my mobile workstation for photos and videos. But I also always carry 1 or 2 Sandisk Extreme Portable SSDs with me. They weigh almost nothing, are fast, and have plenty of space (not sponsored, always buying them myself. NOTE that these drives received some bad media attention recently. But for me, they have always worked just fine). I mostly try to store all my files from a trip on the laptop, and back them up via the portable SSDs.
Like on my main workstation, I use Lightroom on my laptop as well. My workflow here is as follow:
Put all the photos in my Lightroom Catalog
Use a similar structure as on my main workstation: month \ location shoot name
Try to sort photos every day (or every few days) with photos that are ‘keepers’ and need processing
Edit photos on the road (whenever I have time)
Put the sorted and finished photos in collections.
For me the sorting is most important here. Sometimes I have time for editing, sometimes I don’t. The results are put into collections. And here is how I make sure the transfer to my home workstation goes seamlessly:
On all the finished photos (in a collection), give them a rating. The amount of stars does not matter. Usually I give ‘finished’ photos 5 stars, and ‘sorted’ photos that still need editing 4 stars. Simply right click all of the photos in a collection, right click -> Set Rating -> give them a star rating. Once you’ve given ratings, select all the files in all the collections, right click -> Metadata -> save metadata to files.
This is very important, and makes sure that all your edits and ratings are saved within an .xmp file for every photo.
Now this is how you transfer your photos:
Simply select the whole folder of photos, and copy it to your workstation. For example for Antarctica, I simply copy my entire Antarctica folder (that has subfolders for every day of shooting) to my workstation , in the month 2020 / March (when it was shot). Please note that I use the thunderbolt 4 extension card on my NAS for big data transfers. Transferring 1 TB takes me not more than 10 minutes to copy.
After copying the files, I go into your Lightroom on my workstation, right click on March and click ‘Synchronise Folder’. Lightroom will now update that folder with the Antarctica photos.
Now go into Library and sort my Attribute (on top) and click the rating. You now sort the photos by rating. Make sure the sorting is set to ‘Rating is Equal to’ and not the standard ‘Rating is greater than or equal to’.
You now see all of the photos that you sorted by rating on your laptop. Now make new collections on your workstation with for example ‘Antarctica finished’ and put the 5 star rating photos in there. The only thing you need to do here is basically make the collections again, but that literally takes you a minute.
This is how I seamlessly transfer your photos from travels to your workstation, without too much hassle. Hope that was helpful :)
If you want to see how exactly I do this in a video, please check my video on YouTube:
Backups and other functions
As can be seen from the above workflow, my QNAP NAS is integrated in my home network, which is also connected to the internet. This allows me to access my QNAP from anywhere in the world. It’s important that the network is secure, so I always keep my QNAP NAS updated with all the new security updates, and make sure I have multiple authentications to login. The advantage of being able to access the QNAP from anywhere is that I can quickly access some files on the road, if the need is there.
I am often away from home for multiple weeks and I occasionally receive emails with license requests. I can then simply take the RAW (and edited) files from my QNAP through its online access, and that’s very convenient.
What’s also convenient is that NAS devices have apps to automatically backup all of your files with lots of options to your choosing. With QNAP, that app is called HBS (Hybrid Backup Sync) 3.0 . This app is super handy to create backup ‘jobs’ for your QNAP NAS device.
HBS3 lets you create multiple ‘sync’ jobs. You can simply choose copy, move or mirror. I mostly use ‘mirror’ to have an exact mirrored copy that always syncs. You can choose to backup through multiple ways, as can be seen on the photo above:
Local NAS : put another (cheaper) NAS on your network that is used as backup
Remote NAS : Use a NAS and put it at another location. I actually really recommend that with bigger storage. Or a Remote FTP server. Simply put a server at your parents and backup to there :)
Cloud storage: HBS3 supports all the major cloud platforms, like Amazon Glacier, Dropbox, Backblaze etc.
What do I use? Good question, as I am currently in a transition phase. Until a month ago, I would have recommended the Dropbox advanced plan, that worked great. It allowed for backups of unlimited storage that you could easily access (like normal dropbox), for an acceptable price of about 70 USD per month. However, until recently, dropbox changed this and only allows for 1 TB per month of backup. So quickly uploading 30-40 tb is simply not possible anymore.
Currently, Amazon s3 glacier deep archive is most cost effective. After that I would recommend Backblaze B2. But currently, the easiest (and cheapest) way is to simply have a backup at another location. Note that I was previously using Backblaze for just 10 USD per month for unlimited storage. This is the absolute best, but unfortunately personal Backblaze only backs up local drives and USB drives. Backing up network storage is not possible, and for that you need Backblaze B2, which is much more expensive.
So in short: I currently use a sync backup at another location.
You can set everything up in HBS 3 with ease and it’s very straight forward.
End
This rather long article really focuses on using a NAS for a photography workflow. A quick summary of what we have discussed:
How to use Lightroom as a Host and how to structure your directories and photos.
How I work in Lightroom and how I import my photos directly on my NAS.
Detailed Storage workflow:
Using a NAS with multiple (different) drives for a quick and organised workflow.
How to connect everything on the ethernet network for a 10gbit workflow.
Using Thunderbolt extension for even faster speeds.
How I organise the photos on my travel laptop, work with metadata and ratings, for a seamless transition onto my main workstation.
How I copy files from my travel laptop to my main workstation via thunderbolt 4.
How to use backup tools like HBS 3 on QNAP to backup all the files on the NAS
It’s important to note that I only scratched the surface here regarding possibilities, apps and tools that you can with a NAS device. If you’re going the NAS route, it’s definitely fun to check out all the apps and possibilities, if you like that kind of stuff. But if you simply want a solid workflow, this may be the one you are looking for. I certainly hope that this article was helpful and as always: If you have any questions, feel free to ask! I always reply!
Thanks for reading.
Albert