[HN Gopher] Windmill: Open-source developer platform to turn scr...
___________________________________________________________________
Windmill: Open-source developer platform to turn scripts into
workflows and UIs
Author : InitEnabler
Score : 157 points
Date : 2023-05-12 18:20 UTC (4 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| nico wrote:
| Would love a Jupyter or chat style interface for producing the
| scripts, from within the platform
|
| Amazing concept, thank you
| rubenfiszel wrote:
| We have someone from our team working on exactly this. We will
| release that feature in the following months.
| sisve wrote:
| I actually got codeium to work on windmill today trough their
| chrome extension. the extension lets you whitelist sites to
| work on. They do not offer a chat interface yet for the chrome
| extensions but they do have it on other integrations they
| provide. So I would not be surprised if they got it on the
| chrome extension also, at least if it turns out to be popular.
| rubenfiszel wrote:
| Hello, founder here. Thank you for submitting Windmill. We're
| really close from hitting v2 (adding multiplayer with yjs is our
| last milestone) and would have waited a bit before submitting but
| here we go.
|
| We are fully open-source (AGPLv3) with just one ee plugin for
| syncing the cache of the workers at large scale (more on that
| below). You can deploy it on a small ec2 instance with our
| docker-compose or on very large clusters with our helm charts:
| https://docs.windmill.dev/docs/advanced/self_host
|
| We are an onion of 3 layers, which are usually separated
| verticals and the closest services we replace in each vertical
| are: Lambda + n8n/temporal + Retool:
|
| - Workers implemented in Rust that can run any scripts from
| source in python/typescript(deno)/go/bash. They are extremely
| efficient, with most typescript scripts being able to run in 40ms
| e2e. This can be seen as a self-hosted lambda except it works
| very differently. Lambdas spawn microvms using firecracker and
| launch http servers within those micro-vms. We run the scripts
| bare (with isolation using nsjail on our multi-tenant instance)
| but cache very aggressively the dependencies. For instance, in
| python we analyze the dependencies using the AST:
| https://github.com/windmill-labs/windmill/blob/main/backend/...,
| infer the dependencies from there, have them go through pip-
| compile to generate a lockfile (and cache the result), then
| create a pathset similar to how virtualenv behave with the
| version of each dependency locked. For typescript, we leverage
| the very well made immutable deno cache. The cache works well but
| if you scale your cluster to more than 10 workers, the likelihood
| of having your worker see dependencies for the first time
| increase, which is why we reimplemented a pip scheme backed by s3
| that is extremely fast (we do not compress, just tar individual
| wheels and s3 is very fast on local networks). For all the
| supported languages, we parse the signatures of the main function
| to infer the corresponding json schema of the payload to trigger
| them, and from that jsonschema we can then generate automatically
| the UI and infer the input shape required in the flows for that
| particular step.
|
| - A workflow engine re-implemented from scratch which can be seen
| as a distributed FSM and whose state is stored in postgresql and
| every transition done through postgres transactions. Every worker
| is picking a job and progressing the fsm at completion. The full
| spec is at: https://docs.windmill.dev/docs/openflow. We support
| branches, for loop, suspend/sleep, approval steps, sharing the
| same folder between steps and everything you would expect from a
| temporal like engine (retries, error handlers).
|
| - A dashboard/app builder that is very similar to Retool but much
| faster because it's implemented in Svelte and which can also run
| python scripts directly leveraging the 1st layer. We also support
| full react apps: https://github.com/windmill-labs/windmill-react-
| template
|
| We have a CLI that allows you to sync from github directly:
| https://github.com/windmill-labs/windmill-sync-example or your
| local filesystem. Our scripts are absolutely normal scripts so
| there is no lock-in, and they are executable locally
| https://docs.windmill.dev/docs/advanced/local_development so you
| do not have to use our webeditor if you do not want to but our
| editor supports pyright and deno lsp through our dedicated
| backend communicating to monaco using websockets.
|
| We also support worker groups to run some jobs on hardware
| accelerated machines (GPU) and a lot of other features (oauth
| syncing of the resources, schedules, granular permissioning,
| groups and folders).
|
| Every script/flow deployed has a dedicated webhook endpoint to
| run synchronously and asynchronously. Scripts are never
| overwritten, they keep a lineage of hash for each version and
| have per-hash dedicated webhook to ensure their behavior never
| change.
|
| We also have a hub for sharing re-usable scripts
| https://hub.windmill.dev to be used in flows to have some of the
| same convenience you can find in Pipedream or Zapier.
|
| Our closest alternative would be Airplane.dev, but they're not
| open-source, not self-hostable air-gapped, less performant, less-
| featured and much more expensive. You can self-host us fully
| free.
|
| We are used in production at scale in clusters of 50 nodes+ and
| thanks to the reliability of Rust and Postgresql have had no
| downtime to report.
|
| I'm a solo founder, YC S22. The MVP got some attention at the
| very beginning: https://news.ycombinator.com/item?id=31272793 but
| was very bare. Now we're ready to bring this to the next level
| and enable all engineers to spend more time building and less
| time reinventing the wheel thanks to it :)
| EMIRELADERO wrote:
| Your product sounds extremely great! I'm hooked!
|
| One nitpick: In my humble opinon, you should switch from
| Discord chat to Zulip.
|
| It's completely open-source (Apache 2, self-hostable, free full
| cloud plan for open-source projects) and has an excellent
| threads-first (called "topics" model): https://zulip.com/why-
| zulip/
|
| It can run fully in the browser, but it has desktop and mobile
| clients as well.
|
| It also supports public channels which AFAIK discord doesn't.
| So you can link to specific messages inside a GitHub issue and
| anyone who doesn't even have an account can see the relevant
| messages and context.
|
| Interestingly it's what the Rust language team (and community)
| uses for comms: https://rust-lang.zulipchat.com
|
| (I'm not affiliated in any way with Zulip btw, I just love the
| product)
| rubenfiszel wrote:
| I'm very tempted and agree that Zulip sounds much better than
| Discord. The only thing I'm a little bit worried about is
| that I have noticed that most people hate having multiple
| chat clients open and that discord seem to be the de factor
| standards for open source projects. I apologize to be
| contributing to the network effect and will strongly consider
| switching to Zulip.
| EMIRELADERO wrote:
| That's a great response, thank you!
|
| As a sidenote: is the windmill hub open-source too? Does it
| make use of the base product to operate? I think it might
| be a good candidate for showing what your product is
| capable of, as an actual example of use in production
| rubenfiszel wrote:
| Windmill Hub is not open-source, not because we want to
| keep it private but because I wrote it a bit quickly
| using svelte kit and it's not ready for the world to see
| yet. It does not use windmill to operate but every
| actions trigger a windmill webhook and alert the discord.
| CSSer wrote:
| Why did you choose AGPLv3?
| rubenfiszel wrote:
| I love open-source and wanted it to be open-source but given
| that's our whole product and not just a part of a bigger
| ecosystem, and given that we intend to commercialize mostly
| the large enterprise audience, we needed to have a reason for
| the legal departments of those enterprises to push to buy our
| commercial license. So otherwise said, we do not want to
| employ dark patterns and want it to be as easy as possible to
| self-host but also need to be sustainable as a company. Most
| features that you would expect to be premium plugins are
| baked-in and open-source (sso, audit logs, smart
| autocompletion on the webeditor, etc etc)
| jchw wrote:
| I am a fan of this concept, I've thought a lot about it actually.
| Having a proper open source option is awesome and there is no
| doubt I'll be trying this on some of my home/hobby infra to see
| what it has to offer. If nothing else, I really think we need
| more exploration into the space of adding some basic engineering
| and ops on top of things like this.
|
| Not just for servers and infrastructure, but does it bother
| anyone else that for example, the Linux desktop has robust APIs
| for managing powerful service daemons like systemd, but
| relatively few applications that use them and provide useful
| UIs/control panels/dashboards? Somehow for many of the tools we
| DO have, the Linux desktop is stuck in the age of using popen and
| regex against command output even though we really ought to be
| able to do better. It's an odd blindspot in a lot of open source
| work, maybe it is deceptively difficult!
|
| That's mostly nothing to do with this project though, but the two
| problems are intertwined in my head as problems where a little
| bit of engineering could really go a long way.
| robertlagrant wrote:
| Without looking in detail, I've had some experience with internal
| business process/app development, and I always thought Temporal
| would be a great technology to build a platform on. Great choice.
| thawab wrote:
| Hi, windmill is not using temporal, it's an alternative to it.
| robertlagrant wrote:
| Oh! I misread the readme. Ta!
| ttt3ts wrote:
| Was hoping it allowed my runtime. I have a bunch of scripts in
| NodeJS, Python 2, C#, etc. I don't care to rewrite them and/or
| write scripts tied to a weird sandbox runtime.
|
| Any support for docker based workers?
| rubenfiszel wrote:
| We could but decided against it because we have no way to run
| those efficiently given our architecture and prefer to support
| a few languages and ensure we can run them blazing fast than
| all languages but with no guarantees of performance. In
| windmill, you have the guarantee to have no cold-start thanks
| to leveraging the great design of the deno runtime. For python,
| we had to do a LOT of engineering to make it work well and did
| not have the bandwidth to do the same for all languages.
| Ultimately, we could decide that some steps could be slow and
| it doesn't matter but it wasn't requested enough to prioritize
| it.
|
| To be clear, the workers can run in docker, but do not run
| docker jobs.
| ttt3ts wrote:
| Thanks for the clarification.
|
| Looks like you're a new y combinator startup. Kool. I would
| suggest you find away to allow existing code to be reused.
| Deno market share is very small and experienced/larger teams
| are not going to want to rewrite and learn that stack. They
| would have to depend on you, a small new company. Too much
| risk.
| simtel20 wrote:
| It depends on how efficient you need to be. There is the
| option to just use "shell" scripts where you run the
| command. I haven't done much with that, but it's there.
| pjmlp wrote:
| Go is now a script language?
| revskill wrote:
| Yes, just create a script.sh and append #!/usr/bin/go then
| you're ready to go.
| pjmlp wrote:
| If that is the measure for scripting, then any language goes.
| rubenfiszel wrote:
| Our users requested it, so we build it. We use scripts where we
| could just say "code" or "serverless functions".
| iamjackg wrote:
| I get what they mean -- it has definitely gained a foothold in
| the tool/utility world thanks to easy cross-compilation and
| single-binary output. Given their target, I definitely
| understand why they would put it in that group.
| slig wrote:
| Yesterday I submitted a one-click install to Caprover, never been
| so easy to self host. It's really fast on Oracle's free 4vCPU,
| 24GB RAM machine.
| rubenfiszel wrote:
| Thanks a lot for that contribution :)
| GGO wrote:
| If I use self-hosted version with my closed source software (by
| calling my app's APIs), will I have to disclose my App's source
| code due to AGPLv3?
| rubenfiszel wrote:
| No, just don't modify the software itself without making your
| changes open-sources and you will be good to go. Having scripts
| on the platform is not modifying the software.
| GGO wrote:
| Thanks Ruben
| mikepurvis wrote:
| Thank you for this: https://github.com/windmill-labs/windmill#cli
|
| My first questions with any new tool of this kind is immediately:
|
| - Can I run/validate branch versions of the scripts non-
| destructively, for example as part of pre-merge checks in an IaC
| CD workflow?
|
| - Can I run my scripts locally without a lot of drama, while
| developing?
|
| If the answer to either of these questions is no, then the tool
| is the moral equivalent of directly editing PHP files on the prod
| machine, unless you're committed to deploying an entire second
| instance of the tool for staging/dev purposes.
| rubenfiszel wrote:
| In opposite order
|
| 2) yes,
| https://docs.windmill.dev/docs/advanced/local_development
|
| 1) since you can run them locally, you can also run tests as
| part of your CI. You can also deploy a full windmill as part of
| your CI, deploy to that local instance, run some e2e tests by
| calling the webhooks, and then only deploy once those work.
|
| Note that we have a full drafting system baked in into the
| webeditor as well:
| https://docs.windmill.dev/docs/core_concepts/draft_and_deplo...
| and have support for folders with different permissions and you
| can create workspaces so you may implement dev/staging/prod in
| any fashion you like (different folder, workspace, instances)
| replwoacause wrote:
| Does this work with PowerShell scripts?
| sisve wrote:
| Currently the only support shell is Bash
| rubenfiszel wrote:
| We will add powershell support soon (within the next month),
| it's very easy to add given our current architecture and has
| been requested a few times.
| replwoacause wrote:
| Great, can't wait to try it out. I'll be checking back!
| ropeladder wrote:
| I'm confused as to how this compares to tools in the workflow
| manager space such as Airflow/Perfect/Dragster. The words
| describing the software are almost the same but it seems to come
| from alternate universe of tools I'm not familiar with. Could
| someone fill me in?
| pas wrote:
| this is targeting the "low-code" audience, like https://n8n.io
| rubenfiszel wrote:
| I like to think that we target hybrid teams made of low-
| code/semi-technical people AND engineers that expect to have
| the same characteristics seen in temporal/airflow.
| Abstraction without compromise, or zero-cost abstractions as
| the rust community like to say
| rubenfiszel wrote:
| It's similar to airflow/prefect/dragster in that it can run
| workflows, and those workflows can do ETL and ops and can run
| arbitrary code.
|
| It's different in those fashions. Windmill can run python but
| also typescript, go and bash. Airflow/Prefect etc uses code
| annotation which mean the code is actually specific to the
| platform. We actually have the steps be directly scripts/code
| with a main function, and then have a low-code builder to
| compose them into powerful workflows. That low-code builder
| produces a yaml like that one: https://github.com/windmill-
| labs/windmill-sync-example/blob/... which you could
| theoretically write by hand.
|
| The low-code builder is intuitive and similar to n8n except it
| has the feature-set of the above framework. It's easy to test
| the individual steps or to test the whole flow. The workflow
| engine is very efficient and running 50 steps will essentially
| have about 1s overhead total, and the steps are executed bare
| so it will take as much time as if you were running them
| without windmill on your node directly. They can be hardware
| accelerated easily in that fashion. We also support some
| features borrowed from temporal like the ability to suspend at
| no-cost flows until they are resumed by an external event such
| as an approval from slack.
| majkinetor wrote:
| Couple of questions:
|
| 1. Can I see list of all executions, their parameters and
| complete stdout, with handy URL I can refer too?
|
| 2. Can I get notifications (email etc) when script fails or
| certain conditions are met (something appears in the output)?
|
| 3. Can you set execution timeout?
| rubenfiszel wrote:
| 1. yes (that's exactly what the run page of each execution is)
|
| 2. yes (our flows have optional error handlers)
|
| 3. yes (TIMEOUT is an env variable of the workers)
| marcodiego wrote:
| Can it be combined with Gitlab and set us free from Jira?
| rubenfiszel wrote:
| Yes, we expose webhooks for all scripts and flows so you can
| integrate it for anything that can send webhooks, including
| Gitlab, and our workflows can be used as CI/CD.
| thawab wrote:
| The best part about windmill is the team behind it, join the
| discord server to see how fast they fix issues and add features.
| The blog is also top notch: https://www.windmill.dev/blog
| slig wrote:
| Ruben is a machine, I've reported things and he fixed within
| minutes. Their discord server is top notch, 10/10.
| simtel20 wrote:
| Agreed. I hope that they are rewarded greatly for his and the
| team's fanatical attention to everyone who brings up
| suggestions and bugs.
| 1000thVisitor wrote:
| How does this compare to Ansible, and AWX?
|
| https://github.com/ansible/awx
| SPascareli13 wrote:
| How does it compare to something like apache airflow? I like the
| native Go integration, and that it is specifically mentioned that
| the overhead is minimal to start a job. Airflow can take quite a
| varying amount of time to schedule your job.
| rubenfiszel wrote:
| Passing information between steps is a pain in Airflow
| (https://docs.astronomer.io/learn/airflow-passing-data-
| betwee...) whereas it's quite trivial in windmill, the
| performances/latency is better, and it supports typescript.
| Overall, the biggest improvement is the support for
| typescript/go/bash and the low-code builder that will allow you
| to build extremely complex flows that would be hard to
| visualize with just code annotations but with exactly the same
| featureset and more (retries, error handlers, suspend/sleep).
| We also make it really easy to test individual steps and the
| whole flow since web apps allow you to build advanced UX fit
| for big graphs. But Airflow is a great framework that we took a
| lot of inspiration from.
___________________________________________________________________
(page generated 2023-05-12 23:00 UTC)