[HN Gopher] Show HN: Hocus - self-hosted alternative to GitHub C...
___________________________________________________________________
Show HN: Hocus - self-hosted alternative to GitHub Codespaces using
Firecracker
Author : hugodutka
Score : 177 points
Date : 2023-04-03 17:00 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| itissid wrote:
| This seems to be at the same level of how Meta manages their dev
| environments. You pick the environment right from VS code and it
| spins up an environment.
|
| I think
|
| > Add multi-node support
|
| will be important for non trivial project development where the 2
| distinct environments might be needed to talk to each other like
| say for prototyping or "sharing configs"..
| hugodutka wrote:
| By multi-node support we mean the ability of Hocus to manage
| multiple machines, so you can run dev envs on more than one
| server and handle more concurrent users. These environments
| would not communicate with one another. From the end user's
| perspective using a multi-node deployment vs a single-node
| deployment would not differ. If you needed 2 distinct
| environments I would suggest using virtualization within a
| single dev env.
| politelemon wrote:
| I see Docker listed as a requirement, and I am familiar with
| Docker. The repo mentions using Firecracker, a microVM that I've
| only ever heard of. What is the relation, if any, between these
| two? Is this a wrong question: if this uses Firecracker why is
| Docker involved?
| guywithabowtie wrote:
| Firecracker is the VM powering AWS Lambda, it is maintained by
| Amazon. It is pretty decent and popular in certain areas. I
| believe Fly.io is using them behind them too.
| guywithabowtie wrote:
| It has special optimizations for reducing cold start.
| hugodutka wrote:
| We use Firecracker to provider greater isolation and more
| capabilities in dev envs. For example, one thing you often want
| in your dev env is to run Docker containers, but to do that you
| need some permissions that usually only a root user has. If we
| used Docker to virtualize dev envs then we would have several
| options to enable that:
|
| - make dev envs privileged containers, effectively giving any
| random user root control over the machine where Hocus is hosted
|
| - mount the Docker agent from host into the container, but that
| also allows the user to gain root control over the host machine
|
| - use Linux user namespaces, and give you a root user with
| limited capabilities inside your dev env. That's actually what
| Gitpod - another Github Codespaces alternative - does. That
| root can't do everything though. It can't run Kubernetes,
| LocalStack, or the Elastic stack, because it can't modify some
| system limits.
|
| A dev env backed by Firecracker is a full-fledged virtual
| machine that has none of the aforementioned drawbacks. It has
| some performance tradeoffs though, but we are working to
| minimize them as much as possible.
| politelemon wrote:
| OK thanks for that. I am tinkering with it and noticed that
| it makes use of KVM and just found out that WSL2 has KVM
| support so I'm going to give it a try there and see what
| breaks, ha!
|
| https://serverfault.com/questions/1043441/how-to-run-kvm-
| nes...
| cube2222 wrote:
| Docker is used for running the control plane.
| hugodutka wrote:
| That's right! Docker is used to simplify deployment, so other
| components, like the agent, are also containerized.
| cpa wrote:
| I believe this kind of service is the way forward when it comes
| to developer tooling. But paying $39/mo/user for having the right
| to deal with the deployment myself for a tool that is still alpha
| is pretty high. I understand the (tremendous) value prop, but
| that's the kind of price I'd be expecting from a SaaS version,
| not from a self-hosted app. Although not giving a discount for
| self-hosting and a price per user is the "modern" way to do open
| source (see gitlab or mattermost), it still feels a bit wrong.
|
| Anyway, excuse my old man rant, awesome work and good luck making
| a business out of it! I'll give it a try.
| hugodutka wrote:
| Thanks! The pricing is there mostly to set expectations for the
| future. The current release works (for the most part) for
| individual users and it would not support a team well. Hocus is
| free for up to 3 users, so we expect people who deploy the
| alpha version not to pay a thing.
| benatkin wrote:
| For me it's a different issue. To me, Developer Tools are very
| personal, and I've come to view using non-free IDEs as a waste
| of time in the long-run. If I'm going to set it up for myself
| individually, I want the whole thing to be open source*.
|
| * It has the dreaded "View License" on GitHub
| itay wrote:
| Pretty nice, I have two questions:
|
| 1. Comparing to something like GitPod (which lets you run on your
| own instances as well), where do you think Hocus shines?
|
| 2. Given you're leveraging Firecracker for isolation, and
| Firecracker doesn't support GPUs, I assume that adding GPU-
| enabled machines isn't on your near-term roadmap?
| hugodutka wrote:
| Thanks!
|
| 1. GitPod doesn't officially support self-hosting anymore -
| https://news.ycombinator.com/item?id=33907897. When they did,
| it was extremely hard to set up. In fact we tried to do it at
| my previous company several times and failed. Hocus is designed
| with self-hosting in mind, and we want to make deployment and
| management straightforward. And from an end user's perspective
| there are 2 main advantages:
|
| - Hocus dev envs are VMs instead of containers, so you can run
| anything you want inside. For example Kubernetes or nested KVM.
|
| - Workspaces start in <5 seconds, while Gitpod can take a
| minute or longer.
|
| 2. Actually we are exploring moving to Cloud Hypervisor or
| QEMU, so we may support GPUs sooner than later. If you have a
| specific use case in mind, feel free to contact me - contact
| info is in my profile. I'd be happy to hear why you need them.
| guywithabowtie wrote:
| What would be the future plan here? Any hosted commercial
| offerings? I am asking to understand whether one could use and
| depend upon it for few years.
| hugodutka wrote:
| A hosted offering is not on our radar right now. Currently we
| want to monetize by selling seat-based licenses for larger
| teams.
| ezekg wrote:
| I run a software licensing/distribution API
| (https://keygen.sh) that may be able to help you once your
| self-hosted licensing requirements expand over the next
| couple years. Reach out if you ever need anything or just
| want to talk licensing best practices.
|
| We're going to be going source-available under ELv2 soon as
| well. Good to see more companies lately using the Elastic
| license.
| capableweb wrote:
| It seems the future plans (short and long-term) are already
| outlined in the README itself. - Ease of
| deployment and management are central goals of Hocus, with
| minimal assumptions made about the underlying OS and hardware.
| - Hocus should provide a native development experience with
| performance on par with bare-metal development. - Hocus
| should be scalable to accommodate large teams, with support for
| thousands of users and heavyweight repositories. We're not
| there yet. Roadmap - Add basic single
| node support - Optimize single node storage usage,
| performance, and reliability - Add permissioning for
| teams, admins, and regular users - Add multi-node
| support - Add support for more IDEs, particularly
| JetBrains
| notamy wrote:
| This looks excellent! I have to try spinning it up in my homelab
| to see how well it works for my needs.
| yewenjie wrote:
| How does this compare to https://coder.com?
| hugodutka wrote:
| Coder lets you describe a set of Terraform templates for your
| development environments. Your team members then manually spin
| up new VMs on e.g. AWS from these templates.
|
| With Hocus you specify your dev env right alongside your code
| in your Git repository with a hocus.yml file
| (https://hocus.dev/docs/hocus-yml). Hocus is deployed on a set
| of servers you manage (only 1 server during alpha). It has a
| built-in CI system that watches your repository for new commits
| and continuously builds the environment - we call this process
| a prebuild. You can specify that your dependencies should be
| installed and your code should be compiled during a prebuild.
| You can also describe tasks that should run when you open a dev
| env - for example automatically starting the app you are
| working on. Once a prebuild is ready you can spin up a new dev
| env - which we call a workspace - and connect to it
| automatically with your IDE. We also have a VSCode extension
| that automatically opens terminals connected to those tasks
| from hocus.yml. All this combined allows each team member to
| connect to a fresh, ready-to-code dev env anytime they want,
| whether they are reviewing a PR or writing code themselves.
| emptysongglass wrote:
| I really wish you had used one of the existing standards in
| place like .devcontainer or a Devfile. Frustrating to see a
| solution emerge that uses its own declarative devenv
| standard. Portability is also an important feature and I wish
| the industry would converge on one or two.
| donmcronald wrote:
| I didn't look at it thoroughly, but it looks really cool and
| I'll probably give it a proper try at some point. Judging by
| your comment only...
|
| > watches your repository for new commits and continuously
| builds the environment
|
| That feels wrong to me. For example, what happens if I want
| to check out a 1 year old commit and edit it? Will the dev
| environment that was used at the time be available or will it
| be rebuilt? If it gets rebuilt, there's no way I get any
| guarantees it's going to work.
|
| IMHO most of the industry is getting it wrong when it comes
| to dev environments and build systems. I've always called it
| butterfly effect builds. You make a commit and it triggers a
| ton of non-repeatable builds. There's not much value in doing
| that for every commit IMO. What happens when something like
| an APT mirror fails and the whole system grinds to a halt?
| What happens when a runtime or dependency gets updated and
| breaks at exactly the worst time during the week?
|
| I'd rather see something on a (calendar) schedule with the
| potential to trigger a manual build if needed. For example,
| skim this [1] article. That's always made sense to me and, in
| the context of a dev environment, why not build the base dev
| environment once a week (or day) and only layer in the
| resources / code on-demand? IE: Build the things I _don 't_
| control on a calendar schedule and only build on-demand for
| changes that are part of my project (using the base env).
|
| The way I've always envisioned it is to have a stable env
| build once per month or per week. It's fast enough to keep
| everything rolling in terms of being current, yet slow enough
| that I'm not spending time wondering if a problem I'm
| debugging is caused by a change in my dev environment. I try
| to do the same thing as dedicated build containers for
| projects, but it's hard because everything is tunnel visioned
| on per-commit re-builds these days.
|
| 1. https://phauer.com/2019/no-fat-jar-in-docker-image/
| hugodutka wrote:
| > what happens if I want to check out a 1 year old commit
| and edit it?
|
| Your dev environment is version controlled too because it
| lives in your repository. You would run a prebuild for that
| old commit and get a dev env from that time. It would only
| break if your old dev env can no longer be built, for
| example if it depended on some external resource which is
| no longer available on the internet.
|
| > I'd rather see something on a (calendar) schedule with
| the potential to trigger a manual build if needed.
|
| That's actually on our todo list, but it didn't make it
| into the 0.1 release. This feature is especially useful for
| monorepos, where rebuilding everything on every commit is
| too much.
| donmcronald wrote:
| > It would only break if your old dev env can no longer
| be built, for example if it depended on some external
| resource which is no longer available on the internet.
|
| That's what I was getting at. Version controlling a
| config doesn't make a build deterministic when it depends
| on transient resources that aren't in version control.
|
| There's nothing repeatable about a container style build.
| They can change between runs that are 5 seconds apart. I
| see a lot of Docker builds where the author claims
| they're repeatable because they're version controlled and
| then they start with 'apt-get update && apt-get upgrade'.
|
| At least you don't claim yours repeatable, but I'd much
| rather build my dev environment on the first Monday of
| every month, check it in to an artifact repository, and
| use that for a full month. The re-build would be similar
| to (using Docker as an example): FROM
| mygroup/myproject:2023.04 RUN git checkout ...
|
| That way when I go back to a commit that's a year old I
| re-use the _exact_ environment that was in use at the
| time, If I want an updated dev environment I create a
| branch and update the version of the dev environment.
|
| From the perspective of being the developer, I'd rather
| specify a release cadence and have the matching dev
| environment used. Ex: monthly, weekly, daily, ci. IMO
| container like systems that are used as part of the dev /
| build chain should be promoted and reused, not comitted
| and rebuilt. The only person using a CI based build
| environment should be the person responsible for
| maintaining it. Everyone else should be on a slower
| cadence.
|
| It's seems so weird to me to come to HN and see people
| livid about Windows auto-updates and then having no
| problem with their dev and build tools (potentially)
| change 50 times a day.
|
| To put it another way, would you auto-update VS Code,
| your runtime, and your dependencies every time you commit
| to Git? If not, why build the auto-magical dev
| environment so it works like that? Maybe I'm
| misunderstanding what it does. Admittedly, I didn't try
| it out yet.
| enedil wrote:
| This is actually something I wish we had at my previous company.
| Setting up the environment was a pain, and very few people knew
| how to run all the tests (integration tests really). I am willing
| to give Hocus a go.
| 0cf8612b2e1e wrote:
| Very intrigued. With the real VM isolation, I wonder if I could
| use this as a way of dipping my toe into the Nix pool.
|
| Have you given any thought to offering pre-baked development
| templates? For example, here is a standard Django build with
| poetry + js utilities + Postgres drivers + jq, etc.
| hugodutka wrote:
| That's something we would love to do in the future. For now
| extending the base development template so that it has the most
| popular programming languages already installed is on top of
| our todo list.
| cube2222 wrote:
| Ah, great to see this here!
|
| Disclaimer: Know the authors, who are extremely bright engineers.
|
| Based on my discussions with one of them, Hugo, re the
| architecture a couple weeks ago, I really think you should take a
| look at this if you want to do auto-provisioned dev environments
| in a self-hosted fashion, they know what they're doing.
___________________________________________________________________
(page generated 2023-04-03 23:00 UTC)