A Look at an Australian Computing Home Lab

Zorlin on 25/03/2023 - 18:45
Last edited 25/03/2023 - 18:52 by 1 other user

Hey! Thought it might be interesting to talk about my home lab, which I use for experimentation and learning. For those who don't know, home labbing or homelabbing is a hobby where you set up small computer networks at home to learn about computers and kind of experiment.

Everything's at this link - nothing is for sale, nor do I run any sort of ads or subscriptions on my Mirror account.

The tl;dr is 1 pebibyte (roughly 1184TB in total raw storage) of disk, lots and lots of CPU cores, all kinds of other hardware, and a total power bill of less than ~1200W constant draw despite having enough processing power and storage to run a small to medium sized company!

https://mirror.xyz/zorlin.eth/xCVikreJAGJKR9alhkHkKXp0ZB7RJT…

Happy to answer any questions anyone has!

Computing

Comments

  • Zorlin on 25/03/2023 - 18:50
    +1

    Further details:

    Real-world (online and in service) specs:

    • 6 ARM64 cores, with 4GiB of RAM (a lone Kobol Helios64 is the only remaining ARM hardware, after my ODROID HC4s were both retired)
    • 180 AMD64 cores, with 916GiB of RAM
    • Approximately 8TB of NVMe SSDs
    • Over 1 pebibyte of mostly enterprise SATA storage (1184TB total raw capacity, with the bulk of made of 16TB EXOS hard drives from Seagate)
    • 1000Mbps / 1000Mbps unmetered enterprise ethernet fibre from Aussie Broadband, on a 4 hour response, 24/7 SLA
    • 6 ARM64 cores, with 4GiB of RAM (a lone Kobol Helios64 is the only remaining ARM hardware, after my ODROID HC4s were both retired)
    • Standard 1gbps backbone (a cheap TP-Link switch)
  • Zorlin on 25/03/2023 - 18:52

    We use it for Plex/Jellyfin, distributed storage experiments, offloading future workloads from my employer, that sort of thing, I pay for it all out of pocket and the friend who colocates it for me at his place gets fast internet and access to essentially unlimited computing power and storage. Adding in a Bacalhau deployment nicknamed "brackish aquarium" over the next few weeks too so it'll be helping run jobs for people around the world, plus looking at doing Folding@Home and such.

    • dcep on 25/03/2023 - 21:38

      plus looking at doing Folding@Home and such.

      imagine minting btc since 2009

  • Zorlin on 25/03/2023 - 18:54

    Next upgrade planned is adding about 16TB of enterprise SATA SSDs, plus another 128 AMD64 cores (some of the fastest you can buy, AMD 7V13 * 2) and 512GiB of RAM.

    (Also apologies for the accidental bumps, just remembered I could be using the edit button for all this. Sorry)

  • Empharand on 25/03/2023 - 18:58
    +1

    Lol @ your title before Twix changed it. Twix, you ruined it, change it back.

    Also nice Home Lab, now show us your 'Meth Lab'

    • Zorlin on 25/03/2023 - 18:59

      I mean it was cheeky, I actually appreciate Twix's edit though as it was what I was actually trying to get at.

      I can't really afford to run a meth lab, the extra power use would tip my partner's tolerance for my hobbies just past the point of reasonableness, legality and ethics aside ;)

  • bdl on 25/03/2023 - 19:06

    "A $150 fan from Bunnings provides enterprise grade cooling" :)

    • Zorlin on 25/03/2023 - 19:07

      VERY tongue in cheek ;) but it's seriously a decent way to vent the room and dump the excess heat even on a hot day. Obviously that's waaay more effective in the winter than the summer!

      Including the fan and UPS and everything all up it's probably closer to 1300W total of power use, and that builds up in a closed room surprisingly quickly, it's like a decent sized space heater.

  • scotty on 25/03/2023 - 20:40

    Wow that's a lot of hardware for a "home lab". What's the average load of your 4x EPYC nodes? I guess you are using all that CPU power & storage for more than just Plex/Jellyfin, right?!

    • Zorlin on 25/03/2023 - 20:44

      Very little, as I am not taking full advantage of the hardware. I would like to take more advantage… there's Filecoin experiments going on, IPFS experiments, there's some Ch*a farming used to offset power costs, which uses otherwise un-utilised storage, there's also tests going on for a work project (big deployment) etc.

      Looking into running Bacalhau (https://github.com/bacalhau-project/bacalhau, loosely affiliated as it's associated with my employer), and some sort of distributed computing to suck up the spare compute. Other than that we run haproxy, various web services, and are planning on migrating Australia's largest Matrix service to the hardware later (not mentioning it by name because I don't want to shill)

      Sometimes I need to push around 100TB or so of data, so having that much hardware is very useful, as is having 70 hard drives worth of IOPs

      Thanks for the interest! :)

      • Kangal on 25/03/2023 - 22:16

        Can it run Crysis?
        (or MSFS, CB2077, Fallout 76, Arkham Knight, DayZ, GTA IV, Doom, Genshin Impact)

        :P

        • Zorlin on 25/03/2023 - 22:44
          +2

          The new CPUs I'm getting can literally run Crysis in software rendering mode with playable frame rates.

          For real.

  • AustriaBargain on 25/03/2023 - 23:44

    You should run your own ChatGPT on there, but make it sassy and moody if it doesn't get constant compliments.

    • Zorlin on 25/03/2023 - 23:44

      lmao I've already actually considered this and will do it once there's easy packages for LLaMa and such :) GPT-3.5 level stuff is apparently about to come out that's relatively accessible.

  • Austere on 28/03/2023 - 21:44

    This is pretty extreme home labbing, I've been doing something surprisingly similar although 5 dell hosts and using xeons. Recently picking up some old RTX A4000s for cheap from miners on ebay

    • Zorlin on 12/04/2023 - 17:09

      Whoa, super cool. Mildly tempted to do the same!

Login or Join to leave a comment
Login Join