[HN Gopher] LXD containers on macOS at near-native speeds
___________________________________________________________________
LXD containers on macOS at near-native speeds
Author : pickleMeTimbers
Score : 228 points
Date : 2022-11-27 12:56 UTC (10 hours ago)
(HTM) web link (beringresearch.github.io)
(TXT) w3m dump (beringresearch.github.io)
| mistrial9 wrote:
| this is so .. exciting! .. however please recall that you the
| user are now using hardware that is remotely run in most cases by
| the OS vendor (and who-knows-what-else), with opaque code
| executing at multiple layers.
| remram wrote:
| Remotely?
| jbverschoor wrote:
| Too bad almost all containers rely on bridged networking and
| different ports. Why not just bind to 127.0.x.y, where x is the
| project number, and y is the machine number. That way, you can
| just use default ports
| [deleted]
| Eleison23 wrote:
| Because the loopback address doesn't work like that.
| 127.0.0.0/8 is defined as all loopback, which means the OS
| shouldn't differentiate between them, so any VMs binding on
| 127.* are going to interfere with the OS binding there.
| jbverschoor wrote:
| Any other (local) range would be fine. 127.* does seem to
| have some strange behaviors
| jsjohnst wrote:
| This is cool and a worthwhile thing, but how is this different
| than the many (b/x)hyve clones and others based on QEMU that use
| MacOS's virtualization framework to run a minimal Linux for
| containers? What's the differentiator that makes this better
| (hopefully?) from what's come before?
| nerdponx wrote:
| It's appealing to me because I can use LXD on both Mac and
| Linux now. One less discrepancy between those systems.
|
| That said, there now seems to be a huge number of options for
| running various combinations of VMs and containers on Mac and
| Lenix now. I just use whatever seems popular and is easy to set
| up, but I have no sense of the pros and cons among them. I
| would love a coherent document on this topic.
| barefeg wrote:
| I like the progress that is being made for running containerized
| workloads on macOS. In my case I like some of the benefits of
| running the workload on a remote machine; such as no fan noise,
| less heat, less power consumption (especially on laptops).
| However the downsides can be also quite annoying, such as file
| sync times or IDE input lag.
|
| My current setup is to have both data and workload run on a
| remote machine and I connect to it via SSH. I can either run
| neovim inside or use the remote development plugin from VSCode.
| But as mentioned, the input lag can be very annoying. I'm
| wondering if there's another setup where I can still retain some
| of the upsides of running the workloads remotely and still having
| a decent user experience (reduced lag)
| iamveen wrote:
| Mutagen is what you're looking for!
|
| - Local and remote copy of files - Two way sync
|
| I've been using this for years, editing locally with Sublime,
| while running everything on a remote Linux vm. Because it's two
| way sync, anything created/modified on the server is also
| sync'd back to your local filesystem. I end up syncing most of
| the remote $HOME so I can easily fire up a new VM, sync, and
| go. It really is fantastic.
|
| https://mutagen.io/documentation/synchronization
| robertlagrant wrote:
| If you're on Kubernetes remotely, Telepresence [0] might be
| worth a look.
|
| [0] https://www.telepresence.io
| miohtama wrote:
| This is good news.
|
| I come from the development background and the number one use
| case of containers on macOS is development enviroments, as on
| Windows too. For this use case, file system IO has always been
| bottleneck, not CPU. I do not know if there is some silver bullet
| in the horizon that could make this faster.
| hannibalhorn wrote:
| I've been really impressed with what I can accomplish with
| lima-vm, beyond just "replace docker desktop". The latest pre-
| release supports Virtualization.framework, including virtiofs.
| [deleted]
| emptysongglass wrote:
| Check out Finch, a new OSS project out of AWS that packs OSS
| container tools like Lima and nerdctl together:
| https://aws.amazon.com/blogs/opensource/introducing-finch-
| an...
| mark_l_watson wrote:
| I am also happy with Lima on my M1 Macs. I wanted a Intel
| Linux container to run picolisp and it was easy to set up and
| works nicely.
| gpanders wrote:
| Last I checked (which was not that long ago), Lima was stuck
| using 9p because QEMU for Darwin didn't yet support virtiofs.
| Has this changed? Or maybe I'm just misremembering the
| details...
|
| EDIT: Looks like this was indeed just added in the last
| couple of weeks https://github.com/lima-
| vm/lima/commit/c18ae239b69a47db77436...
| JacobSeated wrote:
| virtiofs is supported by newer Docker builds. It used to be
| very buggy with permissions, but the latest builds should have
| solved that.
|
| I am personally slowly moving back to Docker after having used
| limactl for a long time. It's just a pain in the ass.
|
| A Linux laptop is better for all intents and purposes, except
| the shitty battery life...
| est wrote:
| > Prerequisites Install QEMU
|
| So in other words, qemu runs at near-native speeds?
| zweifuss wrote:
| Not right now. Can be an order of magnitude slower compared to
| native. Depends a lot on the program and current task. Also,
| the QEMU defaults are not optimal.
| jbverschoor wrote:
| Well yeah bc it uses
| https://developer.apple.com/documentation/virtualization/run...
| st3fan wrote:
| Yes, on recent MacOS it is mostly just an abstraction on top of
| the native Virtualization Framework.
| ricardobeat wrote:
| Not if you're running x86 emulation on an ARM host. It's
| terribly slow.
|
| EDIT: This is from experience. When I started using a M1 mac,
| our docker builds on x86 images took up to 5x or 10x longer
| vs running on an arm64 image. Had slightly faster, but
| similar results running my own docker host on Lima, or
| podman, or building without docker inside a UTM host which
| also uses qemu.
| runeks wrote:
| No, it's decent performance. Minimum 25% of native, and
| even more using Rosetta:
| https://news.ycombinator.com/item?id=32149080
| ricardobeat wrote:
| For real world use cases like making cross-platform
| builds, 1/4 of native speed is pretty far from decent.
| jbverschoor wrote:
| I don't think that's true.
|
| https://developer.apple.com/documentation/virtualization/ru
| n...
| [deleted]
| [deleted]
| odo1242 wrote:
| Well, it can be. You can still emulate additional hardware
| with QEMU, and QEMU still has other backends (which it will
| use if you run a x86-64 OS on ARM macs).
| garaetjjte wrote:
| It's not abstraction, QEMU uses Hypervisor framework (which
| is equivalent to KVM). Virtualization framework is Apple own
| hypervisor (which is equivalent to QEMU).
| Veliladon wrote:
| QEMU can use the MacOS virtualization framework layer to run a
| VM built for the same arch without going through any
| translation.
|
| From there you can register Rosetta as a runtime and use it to
| run x86-64 Linux apps on ARM64 if you want. (https://developer.
| apple.com/documentation/virtualization/run...)
| qeternity wrote:
| Wait, what? QEMU can pass x86 instructions directly to
| Rosetta without emulating an entire host OS?
| maattdd wrote:
| It does emulate the OS (it's the point of running stuff in
| qemu ) but doesn't translate the instructions
| jsjohnst wrote:
| > It does emulate the OS (it's the point of running stuff
| in qemu)
|
| The point of running stuff in qemu is to virtualize /
| emulate the hardware. You still need a "normal" OS inside
| qemu.
| dontlaugh wrote:
| No, QEMU runs an ARM VM. Inside that VM, Rosetta for Linux
| can be used to run x86 executables, just like on macOS.
| a-dub wrote:
| so arm macbooks are still problematic for developers who
| use containers and x86 servers...
|
| it's still surprising to me that they did this as i
| always got the sense that developer word of mouth helped
| drive a lot of macbook adoption in the 2000s.
| marcellus23 wrote:
| Aside from the basis of your comment being not really
| true (as mrunkel commented), every thread about Apple
| Silicon Macs is full of people praising the performance
| and battery life. This is more positive word-of-mouth
| than Apple's had in years.
| mrunkel wrote:
| In what way?
|
| I use ARM containers locally to dev and test locally,
| check in the code and the build server creates an x86
| image for deployment to the server.
|
| Worst case you have to build the ARM container yourself
| locally and tag it with the "remote" name and docker will
| just use it instead of pulling the remote version.
| a-dub wrote:
| you have to add support for your infrastructure to
| support building of everything for two architectures.
| sometimes problems are architecture dependent. if you
| have to use x86 containers for some reason, they will be
| dog slow because you'll be running the kernel under
| emulation and in userland rosetta won't help.
| cpuguy83 wrote:
| Rosetta absolutely does help. You register rosetta (which
| can be mounted into the VM using macOS's virtualization
| framework) with binfmt_misc and your x86 binaries inside
| the VM will run through Rosetta.
| nerdponx wrote:
| My whole team got M1s, so we just use ARM images on AWS.
| Haven't had any problems.
| codedokode wrote:
| I don't understand why one needs to emulate a virtual machine
| to run a container. Why do you need a kernel? Why emulate
| physical devices, interrupts and everything else? Why cannot
| one run a container as a protected mode application?
| marwis wrote:
| You don't and that's the idea behind gVisor.
| anthk wrote:
| Containers are just fancy Linux chroots. So you need a
| Linux OS.
| sneak wrote:
| Because containers are at their heart a shorthand term for
| a bunch of Linux kernel features that do not exist on other
| OSes.
| pmontra wrote:
| Adding to other answers, not only containers of the docker
| and LXC/D types are Linux things that need a Linux kernel
| but often the software inside the container must be
| deployed and run on a Linux host. Often the build is done
| on a Linux server in a CI system and not on developers
| machines but developers do have to run that software too,
| kind of by definition.
| ryanmccullagh wrote:
| Does QEMU emulate the CPU on macOS?
| gizmo wrote:
| The CPU architecture of the VM and Host have to match,
| otherwise you get software emulation which is very very slow.
| It's good enough for occasional usage, but nowhere near good
| enough for daily development purposes.
| zweifuss wrote:
| Depends. QEMU can use Apple's Hypervisor virtualization
| framework to run ARM64 operating systems on Apple Silicon
| without emulation. It seems to be possible to then use Rosetta
| 2 to run x64-code on those operating systems.
| eurasiantiger wrote:
| Can confirm this works, but some applications will be slow.
| Seen a magnitude of difference with x86 JVM containers
| between Rosetta 2 and native ARM64.
| sirwhinesalot wrote:
| Rosetta 2 works great for AoT stuff but for JIT you'll
| always end up with at least a 2x slowdown since it needs to
| always JIT the JVMs JIT output. That said, it working _at
| all_ is pretty amazing.
| dchuk wrote:
| So is there some canonical guide to running a docker compose
| style app on Mac m1 machines that has good filesystem
| performance? It seems like there's many ways to approach the
| topic now so it's hard to tell which one is "winning".
|
| I'd love to containerize all of my local development efforts
| (scripts and rails apps) but the slow ass filesystem always
| ruined it in the past.
| irusensei wrote:
| Canonical has a product of their own: https://multipass.run/
|
| They all use the underlying OS hypervisor so I'm not sure you
| can get perceivable performance out of one solution over the
| other.
| todotask wrote:
| Sometime it can be buggy on macOS and quite slow to boot with
| Ubuntu VM, wish there is a cloud VM image.
| hda111 wrote:
| > wish there is a cloud VM image
|
| What do you mean? Multipass only runs Ubuntu server cloud
| images.
| pbowyer wrote:
| Have you tried enabling Virtiofs in Docker Desktop? On my M1
| Mac that sorted the performance for me for large PHP apps, and
| I haven't experienced the permissions problems that some people
| have (there is a test build that appears to fix those out now).
| charrondev wrote:
| I have an M1 Pro MacBook and I turned on the 2 experimental
| settings and that sorted my performance issues.
|
| I think the new virtualization framework just came out of
| beta though so now it's just VirtuoFS that needs to be
| enabled.
| pickleMeTimbers wrote:
| We're using Multipass + Bravetools
| (https://github.com/bravetools/bravetools) / Docker to build
| system/application containers.
|
| I believe multipass uses SSHFS
| (https://discourse.ubuntu.com/t/how-to-improve-mounts-
| perform...) to mount filesystems between the host and the VM.
| Performance has been excellent.
| hamandcheese wrote:
| Because docker for Mac has always kind of sucked, I've moved on
| to making nix-based development environments and I've been very
| pleased so far.
|
| Lately I've been trying https://devenv.sh/ and it works great!
|
| I haven't tried it for ruby, though I have used vanilla nix
| shell for ruby projects before and it worked quite well after I
| over-rode GEM_PATH and GEM_HOME to the correct values.
| nerdponx wrote:
| I've been using Colima lately and it's been a great
| experience. The only thing that didn't quite work was pushing
| an image to AWS ECR, but pulling images, building images
| (with BuildKit), and running containers all worked fine, and
| faster than Docker.
| hamandcheese wrote:
| I've tried Colima too but it has always felt sluggish
| compared to local dev. I don't think any form of cross-host
| filesystem sharing could ever be fast enough.
| MuffinFlavored wrote:
| isn't Buildkit Docker?
| MuffinFlavored wrote:
| https://github.com/moby/buildkit it looks like it is not,
| kind of?
| krab wrote:
| Moby is Docker.
| nerdponx wrote:
| Whatever it is, Colima seems to have faster image builds
| and faster container startup, which makes my life better.
| MuffinFlavored wrote:
| help me understand
|
| nix file helps you build OCI image and then you run it not
| locally/somewhere else?
|
| can that be compared apples to apples to docker where you can
| build with Dockerfile syntax, and then run locally the image?
| bpye wrote:
| I've been using Nix for a few years and really like the
| direction of devenv. I much prefer the Nix model over
| containers everywhere.
| bsnnkv wrote:
| I've recently switched to using nix-shell and overmind with a
| Procfile to officially retired docker-compose from all of my
| projects. Such a breath of fresh air.
|
| Like Mitchell Hashimoto (Hashicorp), I no longer do any
| development on macOS directly; I have a Linux VM where all my
| projects live and have their environments managed with a
| shell.nix file. Highly highly recommend it.
| unboxingelf wrote:
| Would you elaborate on the setup? I recall seeing a comment
| or article where Mitchell mentioned this setup previously.
| I've been using Make + Docker for development on macOS for a
| long time now and perhaps it's time for a change. I've looked
| at nix once or twice over the years but it never really
| clicked.
| nerdponx wrote:
| What do you use to run the VM? Do you try to minimize
| overhead at all?
| riffic wrote:
| would be nice to have macOS / Darwin kernel-native containers
| though.
| amelius wrote:
| I experimented with LXD containers on Linux recently, but I found
| the technology it builds on (cgroups) too hard to wrap my head
| around, and tutorials leave me in the dark.
|
| E.g. here is page 2 of one tutorial:
| https://access.redhat.com/documentation/en-us/red_hat_enterp...
|
| All these rules made no sense to me, and while I suppose they
| become clear at some point, I like my tutorials to be clear from
| the start.
| yrro wrote:
| That's documentation for RHEL 6 which is now 12 years old. It
| pertains to cgroups v1 which was horribly complex. It was
| replaced by cgroups v2 which is much simpler; there's now a
| single hierarchy of cgroups.
|
| https://access.redhat.com/documentation/en-us/red_hat_enterp...
|
| That said I don't remember having to learn all that much about
| cgroups when I last looked at lxc...
| amelius wrote:
| Thanks!
|
| Well, I started looking into it because there were all these
| mounted filesystems that I couldn't unmount, even as root. I
| guess I will have to look into namespaces also.
| yrro wrote:
| Yeah, it's pretty much an implementation detail - the thing
| running your containers likely exposes knobs like 'memory
| limit' and 'cpu limit' and under the hood it takes care of
| manipulating the files within the mounted cgroup
| filesystem.
|
| You'll be pleased that at least with cgroups v2 there's
| only a single /sys/fs/cgroup instead of half a dozen
| filesystems mounted underneath that path. :)
| mekster wrote:
| lxd is so much easier than alternatives like systemd-nspawn.
|
| 3 minutes search reveals an easy to follow tutorial.
|
| https://ubuntu.com/server/docs/containers-lxd
| amelius wrote:
| Thanks but I wasn't talking about lxd, but about the
| underlying technology.
| Izkata wrote:
| > underlying technology
|
| I'm guessing bocker would be more useful for that:
| https://news.ycombinator.com/item?id=33218094
| [deleted]
| kaku1019 wrote:
| Does QEMU emulate the CPU on macOS?
| 0x457 wrote:
| No, Rosetta for Linux is used for that.
| loloquwowndueo wrote:
| Keep in mind lxd can manage two types of "containers" these days
| - the traditional cgroup-based kind that runs as its own set of
| processes on top of the hosts kernel with isolation, and
| traditional qemu-backed virtual machines. The user experience is
| homogeneous but the backing engines are different as noted here.
| pxc wrote:
| At first, I thought this was based on a syscall compatibility
| layer like Solaris' Linux Zones or WSL1 (RIP), or the Linux
| support in FreeBSD and NetBSD.
|
| If you've ever tried to spin up a whole bunch of Docker
| containers in WSL2 and watched `vmmem` memory and CPU usage
| explode, you know that 'near-native speed' in VMs comes with lots
| of asterisks.
|
| Does macOS have usable native _macOS_ containers yet?
| alexklarjr wrote:
| IDK what lightweight about virtualisation that takes at least 30%
| of cpu performance for context switching. And much more on io.
| st3fan wrote:
| This is part of the bigger Macpine project, which to me is much
| more interesting than LXD:
| https://github.com/beringresearch/macpine
|
| """ The goal of this project is to enable MacOS users to:
|
| Easily spin up and manage lightweight Alpine Linux environments.
| Use tiny VMs to take advantage of containerisation technologies,
| including LXD and Docker. Build and test software on x86_64 and
| aarch64 systems """
| asmor wrote:
| It feels like lima and colima are much more mature. Being
| unable to remap the SSH port (or other ports) alone makes this
| a non-starter.
| VWWHFSfQ wrote:
| I ended up just getting a System 76 Meerkat computer for
| doing anything locally with LXC/Docker. Linux emulation in
| Windows or macOS is unbearably bad if you really need to do
| serious work.
___________________________________________________________________
(page generated 2022-11-27 23:00 UTC)