Too many external HDD's! How do you keep track?

mogbert

Ars Tribunus Angusticlavius
9,788
I asked an unrelated question the other day and got some really good answers, so I was going to try again.

I have a LOT of HDD's. Some are external, some are internals in a enclosure, and some are just bare internals that I slot into a USB drivebay. Backups, backup of backups, TV shows, 80's cartoons, books, taxes... other stuff. My issue is that I have no idea what I have or where it is. I was looking for an empty HDD the other day, and ran across an old mini external HDD that had a show on it I had been looking for for years.

So I'd like to know if there is some way to organize all this stuff. The closest idea I've been able to come up with was sticky notes on the HDD and something like exporting the DIR to a text file. That seems stupid. I can't be the only one with this problem. Someone has to have already solved it.

So... is there something out there like this? Something that can keep track of everything on these HDDs, which files I have backed up more than twice, how much free space is left of the drive, etc... I'd say I want to get organized, but I already know that is beyond my capabilities.

I'm considering boxing up all my old disc videogames and using the bookshelf to physically keep my HDD's. I will probably also have to get a label maker at some point. It's just that as I move things around, consolidate, delete backups of backups of backups, contents change, so labeling a HDD "Old shows" doesn't help when I need to move that to another HDD because it isn't big enough, or I wind up with Old Shows #1, Old Shows #2, Old Shows #3, and then I'm back to having to look through them for the one I want again.

So if anyone has an idea or could point me in a direction... other than Marie Kondo because deleting things sparks terror for me, not joy.
 

Ardax

Ars Legatus Legionis
19,519
Subscriptor
Remember the cataloging software that used to be a thing back when people would have binders full of zip/jaz/cd/dvds stuffed full of files?

I bet you could find something like that to work today. You'd still have to mark the drive to ID it, but you could keep the file listings on your PC somewhere. Maybe a tool that would automagically index the drives while they were mounted.
 

Chito

Ars Praefectus
3,941
Subscriptor++
I use the tool "Everything" for instant search of my local PCs and NAS. I think it has support for offline volumes, you'll just have to connect each drive and let it get indexed.

https://www.voidtools.com/
It's infinity times better than windows search.

In terms of deduplication, there are some tools I've tried, but never really explored in depth. This is a bigger process if you are indexing hashes, etc. or just filenames, etc. I can't remember the tool I liked the best (not at home right now), but if interested I can look it up.
 

evan_s

Ars Tribunus Angusticlavius
6,347
Subscriptor
Yeah. The best solution is to not have a bunch of drives on the shelf. A QNAP or Synology NAS or a self built storage server is a much better solution. Not only does it solve the issue of being able to actually access the files on the drive it also makes it practical to actually have a proper back up. Having multiple copies of something on three different hard drives you can't identify isn't a back up solution. You can't be sure you'll be able to easily find them when you need them and they could all easily be lost to a single incident at home like a fire or flood.

As just a ballpark how much total capacity do you have across all the easily found HDs? You can consolidate a lot of 80gb 160gb 250gb etc drives into a single 10tb+ drive. If you've got mostly older smaller capacity drives a 2 bay NAS with a couple big drives might be all you need. If you've got newer larger capacity drives in the mix you might need something bigger. It might also make sense to build a NAS and then try to repurpose some of the higher capacity drives into more storage with something like unRaid.
 

mogbert

Ars Tribunus Angusticlavius
9,788
Honestly, I had a NAS and I hated it (QNAP). Two of the offline drives I'm using (both 8TB Toshiba drives made for NAS) are what USED to be in that thing. It promised the world, but was underpowered and way too loud. I think you are underestimating the size and number of the volumes I have. And I don't NEED to have them accessible at all times, I just need to be able to find which one has the show I'm looking for.

Hashes would be nice, but if I can just find duplicates based on filenames, that would be better than nothing. Even just keeping track of how much space is available would let me start gathering similar files together and have them on an appropriately sized drive (80's cartoons take very little room since they were already such low quality (320x240 and mono audio often). Five seasons of just a single more recent show might take 250GB.

I'll look into VoidTools. The words I didn't have before were "offline volumes". It's a much better description than "loose hdd" or "old hdd". That may help me zero in on what I'm looking for. A quick search on "indexing offline volumes" starts returning suggestions! A lot of them are old, but that is something I can look into.
 

evan_s

Ars Tribunus Angusticlavius
6,347
Subscriptor
Honestly, I had a NAS and I hated it (QNAP). Two of the offline drives I'm using (both 8TB Toshiba drives made for NAS) are what USED to be in that thing. It promised the world, but was underpowered and way too loud. I think you are underestimating the size and number of the volumes I have. And I don't NEED to have them accessible at all times, I just need to be able to find which one has the show I'm looking for.

Hashes would be nice, but if I can just find duplicates based on filenames, that would be better than nothing. Even just keeping track of how much space is available would let me start gathering similar files together and have them on an appropriately sized drive (80's cartoons take very little room since they were already such low quality (320x240 and mono audio often). Five seasons of just a single more recent show might take 250GB.

I'll look into VoidTools. The words I didn't have before were "offline volumes". It's a much better description than "loose hdd" or "old hdd". That may help me zero in on what I'm looking for. A quick search on "indexing offline volumes" starts returning suggestions! A lot of them are old, but that is something I can look into.

Assuming it was a 2 drive NAS it probably did have a relatively low power ARM processor and likely a 1gb ethernet both of which can limit performance but neither is an inherent limit for a NAS. You can get QNAP or Synology NASs with Ryzen or Intel processors in them that are pretty powerful. You can also get 2.5gb or even 10gb Ethernet too. If you go DIY the only limit is how much you are willing/able to spend. As far as noise goes you can move it anywhere you want that has networking so the noise won't bother you.

As far as total space goes a NAS can easily be 100's of TBs now a days with enough high capacity drives. An 8 bay nas with 20tb drives is going to be 80-140tbs of storage depending on how you prioritize redundancy/performance. If you go DIY you can easily get dozen+ drives in there in multiple arrays.

It's not going to be a simple or cheap task necessary but it will be much safer for your data if it is important to you.
 

mogbert

Ars Tribunus Angusticlavius
9,788
I'm willing to spend... nothing. Because I'm not looking for a NAS. I don't need the stuff available at any moment, I just need to know which drive has it and I can plug that one in. Seriously. I am not looking for a NAS. The noise alone makes them not worth it. This is in my bedroom and I don't let spinning drives run over night because of the noise. I have access to these drives already, I don't need to complicate things be buying another NAS.

I'm looking for some sort of (looks above) 'offline volume indexing' software, preferably opensource freeware, preferably with dupe filename detection. Hashes, I think, would be overkill I think, and would take too long as some of these drives have entire computers backed up to them from previous PC's.
 

steelghost

Ars Praefectus
5,429
Subscriptor++
I don't know how reliable powered down HDDs are as an archival mechanism, but it doesn't take more than a flipped bits to render many files useless, or at least severely compromised.

All I'm saying is don't be hugely surprised if you go to read one of these drives in a few years and some or all or it is gibberish, or the drive won't respond, or whatever. It's not (quite) the same as keeping books on a shelf. That said, if they don't get too hot or suffer mechanical shock, you'll probably do OK.

If you're putting them all in one place on a shelf, I can recommend these to keep at least the bigger ones from getting smacked about.
 

tiredoldtech

Wise, Aged Ars Veteran
182
Subscriptor++
I don't know how reliable powered down HDDs are as an archival mechanism, but it doesn't take more than a flipped bits to render many files useless, or at least severely compromised.

All I'm saying is don't be hugely surprised if you go to read one of these drives in a few years and some or all or it is gibberish, or the drive won't respond, or whatever. It's not (quite) the same as keeping books on a shelf. That said, if they don't get too hot or suffer mechanical shock, you'll probably do OK.

If you're putting them all in one place on a shelf, I can recommend these to keep at least the bigger ones from getting smacked about.
For those on the other side of the pond (US version of @steelghost UK link):
Inateck 3.5 Inch Hard Drive Case HDD Protective Box with Shockproof Dustproof and Anti-Static Function, Storage Case for 3.5 inch HDDs, 6 Pack, Orange/Blue/Gray (HPFx6)

They do work quite well. At least one prior company used similar units for drive spare storage in the datacenter. They'd buy 10-20 spares at a time and put them in these style units (Grey box with Sharpied drive size/type on the outside) so that if a server drive array had one go bad, they could grab one and run over and put it in right then and there. Those were back the days when there were actual dedicated datacenter staff. I wish I had kept some when they got rid of those and went to hybrid (DC/Cloud) and such.
 

Lord Evermore

Ars Tribunus Militum
2,386
Subscriptor++

mogbert

Ars Tribunus Angusticlavius
9,788
I think Virtual Volumes View might be what I'm looking for. There were a lot of programs that do it, but the vast majority were discontinued 10 or more years ago. VVV is at least still up to date. I'm looking around at the different sites hosting it to try and make sure I get one without added baggage (spyware/adware). Next, I need to buy a label maker (which I'll probably need no matter what solution I went with).

And finally, I'm trying to think up a naming convention to use for those labels. At first I was just going to use something like numbers, maybe even just the serial number of the drive. But then I figured that a naming convention for the labels should be human readable and help me in being able to locate the drive as well, otherwise I'm just sorting through all the drives again looking for the right one. I'm thinking like the first two or three characters being letters representing the manufacture. Then maybe two or three more for the type of drive, like I35 for an internal 3.5" drive or e25 for an external 2.5" drive. Then... well I was thinking of a letter for the color of the drive, but 95% of them are just black, so that won't help. Maybe two characters for the size (1T = 1 terebyte, 5C = 500 gigabytes), then one character to differentiate between the duplicates (1-Z) because there are probably a handful of 64GB flash drives.
 

bkaral

Ars Tribunus Militum
2,803
I agree that NAS drives are convenient, and a great way to motivate one to do/facilitate backups. However, I still consider them a single point of failure, in a sense. True, you have multiple drives, but if the NAS gear itself isn't working, you'll need a way to get access to that data. Worse, if the PSU on it goes bad, technically, it could fry all of the internal drives one shot.

I like to add a surge protector or high quality UPS to the mix, and I disconnect all my external drives' power and data connections at the end of each day.
 
Last edited:

Lord Evermore

Ars Tribunus Militum
2,386
Subscriptor++
I agree that NAS drives are convenient, and a great way to motivate one to do/facilitate backups. However, I still consider them a single point of failure, in a sense. True, you have multiple drives, but if the NAS gear itself isn't working, you'll need a way to get access to that data. Worse, if the PSU on it goes bad, technically, it could fry all of the internal drives one shot.

I like to add a surge protector or high quality UPS to the mix, and I disconnect all my external drives' power and data connections at the end of each day.
The proper solution to a failed NAS is that you have a backup of the whole device, which can be much less robust and performant, and cheap to replace if that fails. A very large but very slow USB mechanical drive, for instance, or cloud-based if you don't mind paying for the storage. (If the power supply fails, it MIGHT take out the backup drive, but probably not. USB ports are usually fused. 3-2-1 rule.) You need to have a second copy of all the data anyway to avoid loss from a failed drive, so it's not "extra" cost compared to using individual drives. The backup data might not be fully up-to-date if the backup only happens once a day, but if you're using individual drives and making two copies of every file you change you've now to to connect two drives, copy the files, then unplug them, so you're adding work and a lot of people might not do it every time. Even if you just read the data occasionally, not writing much, you need to check those drives regularly as well as the backup drives (or one big drive used for backups). If they're all in a NAS, they're monitored and reasonably safe.

If you're using standard RAID with a Linux-based NAS, you can plug the drives into any other Linux machine and mount the array, though obviously that might be cumbersome but manageable if it's just a few drives. If you're running a 12-bay NAS, you're probably spending a lot of money and trying to do it all "right" even if it's expensive, so you have backups. Even if you've only got Windows machines, you could run a VM and mount them there. Some of the "proprietary" storage redundancy implementations in NASes are also compatible with Linux software RAID, and at least one brand (I think it's Synology) has a utility you can install in a computer to access the data even in Windows.

As with everything, there are tradeoffs, pros and cons for both, and the best solution for one person may not be right for another, but just being afraid of a failure isn't a good reason to not use a NAS given the benefits of it for mass network-accessible storage. Disconnecting data and power every day for drives that you use constantly is excessively paranoid, a lot of effort to avoid a miniscule risk, and potentially will accelerate the failure of the drives, and is at the very least wearing out the connectors. If you're going that far, you should also be shutting down your computer and disconnecting network and power cables, as well as unplugging external monitors. And if one of those drives fails, you've still lost the data if you weren't making backups.
 

papadage

Ars Legatus Legionis
43,242
Subscriptor++
I have a NAS with 60TB available storage that I consolidated two very old ones to very early in the year. I back up the important stuff online (everything but non-family video), and have 30TB free.

To back up my non-essential video, I plan to buy a couple of large HDDs early next year and do a quarterly backup. I'll probably store them at my wife's aunt's house across town.

For a non-NAS solution, I would buy larger drives, consolidate them into fewer drives, and keep stuff in my category on them—sci-fi and toon on one, drama and comedies on another, etc. Then, I'd use a Thunderbolt dock to attach to the PC. I would keep the offline ones in padded storage.
 

Lord Evermore

Ars Tribunus Militum
2,386
Subscriptor++
I store mechanical drives and 2.5-inch SSDs on whatever flat surface happens to be nearby and wipe the dust accumulation off when I need to move one to another flat surface to make space, or piled in a drawer with other electronic stuff. (I'm more careful with NVMe SSDs since they're small and just a thin PCB with all components exposed.) Whenever I move, they get piled loose in boxes with other stuff. I've never had any problems with them. I use an old USB3 to SATA base that came from an external Seagate drive where the disk was in a separate enclosure that plugged into the base, and the drive just sits on the desk with the dock hung off the edge so it's not applying pressure to the connector, or on top of or leaning against the PC case under the desk, or lying on the carpet, wherever is convenient.
 

papadage

Ars Legatus Legionis
43,242
Subscriptor++
I store mechanical drives and 2.5-inch SSDs on whatever flat surface happens to be nearby and wipe the dust accumulation off when I need to move one to another flat surface to make space, or piled in a drawer with other electronic stuff. (I'm more careful with NVMe SSDs since they're small and just a thin PCB with all components exposed.) Whenever I move, they get piled loose in boxes with other stuff. I've never had any problems with them. I use an old USB3 to SATA base that came from an external Seagate drive where the disk was in a separate enclosure that plugged into the base, and the drive just sits on the desk with the dock hung off the edge so it's not applying pressure to the connector, or on top of or leaning against the PC case under the desk, or lying on the carpet, wherever is convenient.

This reply just gave me hives.
 

mogbert

Ars Tribunus Angusticlavius
9,788
Picked up a Dymo 100H label maker. It wasn't on sale or anything, but it seems to have been around for a while so I figured it was probably a solid one and I shouldn't have to worry too much about finding the replacement labels. Was a bit... perturbed when I found the price they charge for the spare labels was 50% higher than what they display on their website even when I put on the website that I'm picking it up today in the store, it still apparently gives the price for if you ship it and has no indication they will charge 50% more for trying to pick it up yourself.

Anyway, that isn't here nor there. I'm still moving forward with the plan...
 

Lord Evermore

Ars Tribunus Militum
2,386
Subscriptor++
Virtually every store these days lists "web only" pricing on their site. Sometimes they specifically mention it below the price, sometimes not, they don't give you an option for in-store pricing, and they're not even consistent about it applying to every product. They seriously want you to just stop coming to the store and order everything for delivery (or at least order for curbside pickup) and have a paid subscription because you want to be sure you'll get the lowest price and like the convenience, because they think it somehow saves them money over having your body in the building. (One employee can pick items for many customers at once.) If they could have their way, every store would just work like Amazon.

You can also get third-party labels for most label makers which is a lot cheaper, if you end up going through the tape quickly. At least Dymo seems to be one of the least bad about wasting tape with blank space on each end, but I just read a comment about them putting "DRM" protection on the cartridges like printer makers have been doing, so you can't use anything but their brand.
 

mogbert

Ars Tribunus Angusticlavius
9,788
Virtually every store these days lists "web only" pricing on their site. Sometimes they specifically mention it below the price, sometimes not, they don't give you an option for in-store pricing, and they're not even consistent about it applying to every product. They seriously want you to just stop coming to the store and order everything for delivery (or at least order for curbside pickup) and have a paid subscription because you want to be sure you'll get the lowest price and like the convenience, because they think it somehow saves them money over having your body in the building. (One employee can pick items for many customers at once.) If they could have their way, every store would just work like Amazon.

You can also get third-party labels for most label makers which is a lot cheaper, if you end up going through the tape quickly. At least Dymo seems to be one of the least bad about wasting tape with blank space on each end, but I just read a comment about them putting "DRM" protection on the cartridges like printer makers have been doing, so you can't use anything but their brand.
I really don't see anything fancy on either the maker or the labels that would indicate any sort or DRM on the labels. This is a pretty cheap model. Maybe they would do that for a fancier one.
 

iljitsch

Ars Tribunus Angusticlavius
8,976
Subscriptor++
This is crying out for a NAS
I had a lot of stuff burned to DVD, and consolidating that to a NAS was extremely helpful. I still have ten or so HDDs and for the most part a label maker is good enough to keep track of those.

What also helps is that my last USB HDD is 3 TB and with that big enough as the overflow destination for TV shows that can't fit on the NAS.

Edit: what makes a NAS even better is that you can stream content from it. If you get one, make sure it can run something like Jellyfin or Plex to make that super easy.
 

papadage

Ars Legatus Legionis
43,242
Subscriptor++
That's my other major NAS use case. I have all my music collection, BD rips, purchased videos, photos, and other content on my NAS, and a mini-PC with Plex installed. I can stream all my won stuff to anywhere in the house and on the road and share it selectively with family and friends. Our house is the preferred gathering spot after a family wedding or baptism. I get a copy of the photos and video and set up a viewing day where everyone brings snacks and drinks, and we have a viewing party. After that, I share it with the family members' accounts on Plex.
 

Cool Modine

Ars Tribunus Angusticlavius
8,772
Subscriptor
I agree that NAS drives are convenient, and a great way to motivate one to do/facilitate backups. However, I still consider them a single point of failure, in a sense.
If you have a good backup, you don't have a single point of failure. :)

I think you are underestimating the size and number of the volumes I have.
How many separate drives DO you have? Are they all largeish, i.e. 4TB or more? Or lots of smaller ones?
 

Lord Evermore

Ars Tribunus Militum
2,386
Subscriptor++
If you have a good backup, you don't have a single point of failure.
The enclosure is still a single point of failure for the functionality being provided, a shared network resource, because you have to go to a lot of trouble to make that backup data available, and ALL of the data becomes unavailable so even if it's just one machine accessing it, multiple storage drives would at least allow you to still access the rest of your data if one of them fails. It's just a matter of it being a much more convenient storage and retrieval method, with potential for performing other duties, but unless you have a 10Gb network it will also be slower than USB with decent drives.

Obviously the correct solution is thin client systems (with hot spares available for immediate use) running virtual machines on redundant clustered servers for seamless failover, connected to physically separate redundant 100Gb network equipment and Internet connectivity via dual WANs from different providers on separate firewalls in failover (with your own AS and IP range using BGP), all running on a power system with high-capacity on-line battery backup for the entire building and generator power that kicks in automatically, with one SAN on-site and another connected via VPN at another location in cluster/failover configuration with volumes configured for shared storage and iSCSI drive access for the servers, and the SAN data running over a dedicated network. The SANs should be backed up to offsite tape continuously. Of course, there should also be one janky electrical outlet held together with masking tape which the cleaner uses to power the vacuum at night, which puts out high levels of electromagnetic noise sufficient to corrupt data passing through the bundle of cheap network cables that runs beside it resulting in the corruption of large amounts of data in ways that aren't detectable until an audit is performed or files need to be restored.
 
  • Like
Reactions: bkaral

papadage

Ars Legatus Legionis
43,242
Subscriptor++
That's a lot of overkill for home users.

And aside from that, modern NAS units can be upgraded to 10GBE fairly inexpensively, I did so with mine for about $100 and added a 5-port 10GBE switch for anothet $200. MY PC already has a 5GB port, which is more than fast enough for my needs. My mini-PC for Plex and other fucntions has 2.5GBE built in, which is also more than enough for what it does.

All that together, plus repurposed older drives for backup of everything,, and online backup for the vital stuff, and feel pretty good about the safety of my data and its availability. The HHD backups can always be connected to my PC or mini PC if I really need access if the NAS dies.

Of course, everyone needs to weigh their own options and their tolerance for risk and inconvenience. I've had a NAS fail, but that was a temporary issue since I could just put the drives in a new one of the same brand and just start up again.
 
Last edited:

mogbert

Ars Tribunus Angusticlavius
9,788
How many separate drives DO you have? Are they all largeish, i.e. 4TB or more? Or lots of smaller ones?
Probably around thirty or so. Other than some Western Digital drives from a long time ago, I really haven't had many failures. However, EVERY WD drive I ever bought had failed. They have a good RMA process, but I got tired of using it so often.

Now a problem I have, though I mentioned it earlier, is sometimes I'll take a smaller drive and copy it to a larger drive to consolidate... then forget I've copied the data already or keep the old drive for a back-up of the data and suddenly I now have two drives I have to keep track of. That is why I wanted a way to identify if I have duplicate files (like identical name, size, and created date or something).

I have been wanting to take it easy for a little bit, so I haven't moved forward on this side project after I got the labeler.
 

redleader

Ars Legatus Legionis
35,457
Buying hard drives on brand is foolish when there is actual reliability data for a lot of drive models:

https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-2024/
Go to the size you want, ignore anything with less than half a million drive days of data, and then punch whatever is the lowest annual failure rate model number in Amazon.
 

Lord Evermore

Ars Tribunus Militum
2,386
Subscriptor++
Similarly, I don’t buy Seagate any more for the same reason. My current batch of WDs has held up pretty well.
I haven't bought a Seagate mechanical drive for anything personal for many years (well, more years than since I stopped buying mechanical drives), and stopped buying them for clients to replace out of warranty failures. First experience with failures of my own, then working in IT and seeing the failure rate of the cheap ass models used by Dell in their PCs, AND the failure rate of USB models that we used for Windows Server Backup (for the "price sensitive" clients), and even failure of the models used by Dell in low end servers, then comparing to experience with WD drives over many years, the little bit lower price just isn't worth it. WD of course had their own times of unreliability, but not as bad.

Of course their SSD models may be completely different, but I'm not sure just how much "Seagate" is in the Seagate drives and how much is just a third party building them.
 

Lord Evermore

Ars Tribunus Militum
2,386
Subscriptor++
Buying hard drives on brand is foolish when there is actual reliability data for a lot of drive models:

https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-2024/
Go to the size you want, ignore anything with less than half a million drive days of data, and then punch whatever is the lowest annual failure rate model number in Amazon.
Sure, if you want one of those specific listed models. But even there it's clear that Seagate has a dramatically higher overall failure rate than WDC. At 14TB the Seagate failure rate is 12 times WDC's for one model! If you're just picking a drive based on price and capacity that's available where you're buying, you clearly have a better chance with WDC than Seagate. Even WDC's worst drive is better than all but two of Seagate's. Toshiba and HGST are better, too, with much more consistent failure rates across models which are also better than nearly all Seagate models.
 

Lord Evermore

Ars Tribunus Militum
2,386
Subscriptor++
I probably need to get rid of the old drives that date from 2008 (like my WD 15K RPM Raptors; those things are heavy AF).
Those are iconic. Save them until they become museum-worthy. Impress the grandkids. "This thing only held 100 million low quality songs! Yes, we kept our own songs in our possession! Yes, food and electricity were abundant back then, and we didn't have to hide our music from the secret police!"
 

tiredoldtech

Wise, Aged Ars Veteran
182
Subscriptor++
Those are iconic. Save them until they become museum-worthy. Impress the grandkids. "This thing only held 100 million low quality songs! Yes, we kept our own songs in our possession! Yes, food and electricity were abundant back then, and we didn't have to hide our music from the secret police!"
Yes, we did. The Secret Police were after our music since at least 2000 (remember the Napster vs Metallica & RIAA debacle?).
 

singebob

Ars Scholae Palatinae
766
I hoard redundant drives purely for the reason that every time the office had a clearout of disks for secure disposal I kept forgetting to sneak my disks into the pile. And I think I've finally missed the boat altogether since we're out of all our datacenters and have no spinny rust hardware left - everything that isn't still on tape in archive is somewhere in the cloud. There isn't even a NAS at the office anymore.

I might actually have to pay for it myself, sadge

Also, for the OP - anything wrong with a labelmaker?
 
Last edited:
  • Like
Reactions: papadage