Best Practice in Personal Data Storage

• 

  Jimothy Wongingtons on 08/01/2023 - 20:01

  About 10 years ago I had a seagate hard drive die on me suddenly. We had an automated backup running daily on a headless server but somewhere alone the line, the program doing backup had crashed and just wasn’t running anymore… so we lost like.. nearly 2 year of data .

  Since then I just use Dropbox now (paid sub). Not worth the drama imo.

  • 

    tryingtohelp on 09/01/2023 - 09:33

    Thanks for the response, the paid subs are good for data and pics, anything beyond that gets quite expensive. I'm already managing the photos on icloud.

    • 

      Abaddon on 08/02/2023 - 14:24

      Good plan with iCloud, you can always get your photos back from the next leak :p

• 

  AAC on 08/01/2023 - 20:29

  +1

  It sounds like you've already got an established homelab, have you considered hosting your own cloud storage as well and save on the subscriptions?

  Hardware will inevitably fail, it's just a matter of when. That's why backup solutions are just as important.

  If I were you, I'd look at the following options:
  - RAID configuration
  - If RAID is too expensive, setup dedicated drives for archives
  - Expose your storage externally with Nextcloud / Filebrowser / Seafile.. etc
  - Setup watchdogs / notifications for SMART errors or backup failures (I believe TrueNAS has this natively)

  Look at backup solutions like Kopia / Duplicacy, I don't think you'll have any issues setting them up

  • 

    tryingtohelp on 09/01/2023 - 09:45

    Thanks for the response :-)

    I couldn't agree more in regards to the hardware life, it may fail at any moment.
    Based on your message, and some 2TB old HDDs I had at home that was not being used, I decided to put them together as RAID in my AMD pc for day-to-day computing and data sorting. At least, I'll know if one of them is about to fail and I can move the data quickly. Moving 2TB of data is relatively quick when compared to moving 8TB (this was bloody painful).

    I'll explore NExt Cloud, but remote access isn't one of the objectives here.

    In regards to self-hosting, I still have to learn the way, please provide me with any pointers you may have. I have a rich single uncle in Argentina who has generously offered the Icloud storage (you know what I mean) and this storage is being shared across the family (very convenient). I save on not buying the higher storage tiers on IOS devices.

    • 

      dextertheboygenius on 09/01/2023 - 10:17

      Recently started using Stablebit Drivepool instead of using RAID, pretty happy with the data deduplication feature.

• 

  AustriaBargain on 08/01/2023 - 20:35

  +3

  I keep all stuff on OneDrive. Don't know what I'll do when I hit 1TB.

• 

  slimpickings on 08/01/2023 - 21:08

  +1

  Assuming your main goal here is data redundancy and you don't care about whether a drive failure causes downtime:

  Cheapish options from easiest to hardest

  • If your internet is good I'd pay for backblaze. $10/month for unlimited storage.
  • Buy a really big hard drive, plug it into computer, sync everything over at intervals. Set it as a target for vm snapshots if you want. You can increase the redundancy by buying a second drive for cold storage and regularly swapping them, or hooking one up to a raspberry pi and rsyncing to it from the main drive.
  • Buy a second hand LTO6/LTO7 tape drive and as many tapes as you want. Tapes are cheap per tb and durable, so you have flexibility with volume and backup versions. Much more of a hassle to set up and run.

  Beyond those you're likely to blow past the two grand mark very quickly with the volume of data you're talking about. You keep it cheaper by backing up irreplaceable data only (documents, photos) and ignoring anything easily replaced (movies, full vm images, games).

  • 

    tryingtohelp on 09/01/2023 - 09:55

    Thanks for the response.
    As you described, I'm only intending to save irreplaceable data. I don't care about the movies if they disappear from my computer.

• 

  FoxJump on 08/01/2023 - 21:20

  +1

  I will tell you what I used to do, and what i do now.

  What I used to do: Important files live in Onedrive replicated across several computers at home. Big photos, family videos, ISOs get manually copied into 2x ext hdds. They are meant to be a mirror. When i take pics with my dslr i sort them on my computer and then copy to 1 backup disk, then another backup disks. Photos from all household members get sent to my onedrive camera roll, which i download and merge with the dslr photos before backup. I then copy the last years worth of jpegs into a onedrive archive manually (though it later got scripted)

  What i do now:
  A nas, 2x NVME SSD on raid one. They contain:
  a) a mirror of the onedrive folder, excluding camera roll and 1 yr backup
  b) user-private 'home' folders
  c) incoming photos.
  4x4tb disks in raid 6, they contain:
  - isos
  - family home videos and photos
  - media

  I have several sync jobs. Each day they will,
  a) mirror dropbox, mirror onedrive (mostly)
  b) move onedrive camera roll to the incoming photos/fromonedrive
  c) cp to onedrive the last years worth of jpegs
  d) an ext hdd wakes up and makes a backup from the nas

  Once as week I put an ext hdd into the nas, it backs up stuff to it (except media), and it gets taken off site. Next week, we repeat with another ext disk
  When i take pics with DSLR, i put memcard into card reader, it automatically sucks it empty into the incoming folder.

  I have another nvme on order to act as a spare. my siliconpower nvme ssds died quickly. i now have a wd blue and an old samsung 970 evo and soon the spare.

  Why all this?

  Cos in April 2022 i lost 1 of my external hdds due to a random death. I realised, that for a short period of time, i had no backup. if my copy from the surviving disk to the replacement ext disk failed, i was done for.

  So now i have enough copies.

  • 

    ihbh on 08/01/2023 - 21:56

    taken off site

    relative, friend, work?

    • 

      FoxJump on 08/01/2023 - 23:13

      yes, no, yes

• 

  boomramada on 08/01/2023 - 22:08

  I use three portable HD and duplicate with xcopy leave them in three different places and rotate the copy cycle ever few months.

• 

  ProlapsedHeinous on 09/01/2023 - 02:33

  From the web: "You may have heard of the 3-2-1 backup strategy. It means having at least three copies of your data, two local (on-site) but on different media (read: devices), and at least one copy off-site"

  The two at home is a no brainer, but the third, im thinking you can encrypt and give to a mate or next door neighbour or even put in a fireproof waterproof cool place in your shed or anywhere you can separate that backup from the house burning down or getting robbed.

• 

  ihfree on 09/01/2023 - 17:48

  If you go with an non-online storage solution:

  • mirror important data
  • have an offline backup
  • have an off-site or online backup

  My old solution was using a n40l with a mirrored volume and second volume for media, etc. This was backed up to USB drives.

  There's also a useful utility called vvv for managing external drives which catalogues drive contents.

• 

  tryingtohelp on 11/01/2023 - 09:48

  +1

  Thank you all for responding to this topic :-)
  Greatly appreciated, thanks again. I got enough pointers here, and now it's time to do my homework !!