[HN Gopher] Turing Pi: Kubernetes Cluster on Your Desk
___________________________________________________________________
Turing Pi: Kubernetes Cluster on Your Desk
Author : jcamou
Score : 56 points
Date : 2020-03-24 19:21 UTC (3 hours ago)
(HTM) web link (turingpi.com)
(TXT) w3m dump (turingpi.com)
| rubyn00bie wrote:
| While Raspberry Pi's are awesome, and the power consumption is
| nothing to scoff at (when considering a cluster), you can
| accomplish this same thing for a lot less, and have quite a lot
| more compute power, by purchasing a used server or even something
| like an AMD 3600X...
|
| A single 3600X will grossly outperform this cluster (and cost
| less) with less headaches (you don't have N physical machines) by
| using KVM to deploy a few virtual machines and using Kubernetes
| to orchestrate and allocate within those VMs. You'll also have a
| lot less latency between nodes running in VMs on the same
| physical host.
|
| Another thing that unfortunately sucks about Raspberry Pis (less
| with Pi 4, but still mostly applies) is really shitty I/O
| performance...
|
| I spent a large amount of time over the past summer and fall
| trying out various ideas to have a "cluster" at home that was
| both practical and useful. While, the PIs were nice, they never
| really amounted to much more than a demo. Latency and I/O become
| real problems for a lot of useful interconnected services and
| applications.
|
| Honestly, if Ryzen 3000 hadn't come out, for cheaper cluster
| builds (~300-400) I still think Pis would be a solid choice
| but... Ryzen 3000 is just so fucking fast with a lot of cores,
| it's truly hard to beat.
|
| Addendum: to touch on used servers, yes your power bill will go
| way up, no joke, but for some applications like large storage
| arrays-- it's hands down the cheapest/easiest route. Search by
| case, not by processor, it sounds weird but the case is likely
| the most valuable part of the old server (like ones with 20+ SAS2
| slots for $500) or PCI-E slots that GPUs can fit into.
| rconti wrote:
| As a project, I like the idea of having the nodes be separate,
| because endless layers of virtual things makes it harder to
| grop what's truly going on, IMO.
|
| Also, gotta factor in electricity hosts.
| hinkley wrote:
| The other part of 'on your desk' is hearing damage.
|
| Server hardware vendors have traditionally not given two shits
| about their servers being north of 90 decibels, and I'm pretty
| sure I've witnessed a few that were pushing 100.
|
| That Raspberry Pi is probably going to absorb more noise than
| it makes.
| bproven wrote:
| you mean using a ryzen with VMs (cluster on one ryzen) I
| assume? bc cost of ryzan as a node would be pretty pricey
| compared to Pi 4s.
| rubyn00bie wrote:
| Yes, a single ryzen node running multiple VMs.
| nsky-world wrote:
| So, you don't see a difference between clustering on
| physically separated nodes and VM?
| moondev wrote:
| Looking at the specs, it seems almost dishonest to promote this
| for kubernetes.
|
| > The nodes interconnected with the onboard 1 Gbps switch.
| However, each node is limited with 100 Mbps USB speed.
|
| Not only that but Compute Module 3+ are limited to 1GB RAM, is it
| really expected someone could run a realistic workload? How
| stable is the control plane node with such limited resources?
|
| It seems like picking up 3 raspi4s (4GB RAM each) and powering
| via PoE would be a must better result.
| Floegipoky wrote:
| I built a cluster of 6 pi 3b+ duct taped to a USB hub for power
| and a switch and router a few years ago for the explicit
| purpose of experimenting with clustering technologies,
| including kubernetes. I wasn't running it for long periods of
| time but it was surprisingly stable. But to your point, 1GB RAM
| doesn't get you very far with many of the popular distributed
| systems these days.
| detaro wrote:
| If you insist on using Pis (IMHO you're quickly at the point
| where a (potentially used) NUC or small office PC is the better
| choice), why would you go for extra expense and effort for PoE?
| Just connect a 5V PSU to the power pins? I guess it gives you
| individual power switching without any DIY, but other than
| that?
| afterwalk wrote:
| I randomly came across a similar offering and thought about
| buying for fun: https://www.mininodes.com/product/5-node-
| raspberry-pi-3-com-...
|
| Any idea how the specs compare?
| mbreese wrote:
| This is still based on the compute modules, so you're going
| to have the same tradeoffs.
|
| These are interesting ideas, but until the compute modules
| start having more on board RAM, you'd be much better off
| working with a few RPi 4's. They will be faster (1Gbps
| ethernet, more RAM), and cheaper since you only need a
| gigabit switch to connect them together, not a custom carrier
| board.
|
| _If_ the compute modules start to get more powerful, then
| having to deal with only one power supply and ethernet uplink
| would be nice. It 's a very appealing idea. But, you're going
| to almost always have a better experience with a bunch of
| standard RPis.
| wedn3sday wrote:
| I agree, using the USB bus for inter node communication seems
| like a poor design choice. Anyone got any insight on why they
| wouldnt use the much faster ethernet connection?
| cconstantine wrote:
| My understanding is that the ethernet module on the raspberry
| pi is a usb device, not a pci device.
| oakwhiz wrote:
| The SoC used in the Pi 3 does not have an Ethernet MAC. It's
| always provided via USB. The CM3 does not have a USB Ethernet
| device onboard, it's provided by the carrier board.
| kylek wrote:
| I'd hope people wouldn't be using a setup like this for any
| other reason than learning k8s or tinkering/fun.
|
| edit-before-actually-posting (sorry I'm a bad person for typing
| a reply before clicking the link): Wait, they're selling these
| things? Ok, then I'm stumped. What would you do with 8 rpi's
| that you couldn't do with one?
| aivarsk wrote:
| If you're looking for a cheaper alternative then
| https://clusterhat.com/ is worth taking a look. I have one
| sitting on my desk (4 Pi Zero nodes and Raspberry Pi 2
| controller).
| kube-system wrote:
| This is neat, but I'm really more interested to hear about
| potential use cases. I'm guessing this is mostly useful for ARM
| workloads? Maybe some situations with low power requirements?
|
| Personally, for my multi-node test clusters, I just run VMs on
| cheap x86 hardware.
| hexman wrote:
| some ideas https://blog.alexellis.io/
| closeparen wrote:
| There's something amusingly cyclical about a blade server
| architecture for Kubernetes. The tech comes out of a whole
| movement towards combining commodity machines using clever
| software instead of buying specialist hardware, but then adds the
| specialist hardware back in.
|
| Some deeper integration between Kubernetes and the hardware
| (acceleration/offload ASICs maybe), branding of k8s + this
| hardware as a unified product, and this would literally just be a
| mainframe. Which is not a terrible idea! Maybe Kubernetes is the
| mainframe operating system of the future.
| inetknght wrote:
| Think of it as a way of using mixed hardware: mainframe
| hardware in conjunction with commodity hardware. It's extremely
| useful in that context.
| tlrobinson wrote:
| Note the $189 price tag doesn't include the Raspberry Pi compute
| modules, which are about $30-40 each.
|
| It's a neat form factor but you could just buy some regular
| Raspberry Pis and an Ethernet switch.
| apple4ever wrote:
| I thought it was $189 for the whole kit and caboodle, but as
| you said its not.
|
| $189 is a little expensive for what you get.
| sgt wrote:
| That's without eMMC though. Having a bunch if normal Pi's
| running SD cards would end up in tears at some point.
___________________________________________________________________
(page generated 2020-03-24 23:00 UTC)